Medical image processing method based on deep learning
By optimizing the Transformer model through multimodal adaptive preprocessing and adaptive asymmetric attention mechanism, the asymmetric interference problem in the diagnosis of unilateral stroke lesions in the existing technology is solved, and higher diagnostic accuracy and sensitivity are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANYANG OPEN UNIVERSITY
- Filing Date
- 2026-01-22
- Publication Date
- 2026-04-17
AI Technical Summary
In existing technologies, the symmetric masking attention mechanism suffers severe interference when dealing with unilaterally distributed stroke lesions due to its inherent asymmetry, thus affecting diagnostic accuracy.
A deep learning-based medical image processing method is adopted, which uses multimodal adaptive preprocessing, deep learning registration network and adaptive asymmetric attention mechanism optimized Transformer model, combined with temporal attention mechanism, to perform medical image registration and segmentation, and generate the final segmentation prediction result.
It effectively avoids false-positive lesions in the healthy brain region, improves the sensitivity and clinical value of segmentation results, can identify abnormal lesions in dynamic processes, and improves the accuracy and sensitivity of diagnosis.
Smart Images
Figure CN121883473A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical image processing technology, and in particular to a deep learning-based medical image processing method. Background Technology
[0002] Deep learning, with its powerful feature extraction, pattern recognition, and end-to-end learning capabilities, can automatically learn multi-level abstract representations from massive medical image data, ranging from low-level pixel features to high-level semantic information. By constructing deep learning models such as convolutional neural networks, generative adversarial networks, or Transformers, it can achieve high-precision segmentation, lesion detection, structural reconstruction, functional analysis, and multi-modal data fusion of multimodal medical images such as brain CT, MRI, and PET. This improves the objectivity, accuracy, and efficiency of clinical diagnosis, reduces the risk of missed or misdiagnosed diagnoses, and supports early disease screening, efficacy evaluation, and personalized treatment plan development. Its significance lies not only in significantly improving the automation level of medical image analysis and clinical decision support capabilities, providing key technical support for the precise diagnosis and treatment of complex diseases such as tumors, neurodegenerative diseases, and vascular diseases, but also in promoting the leapfrog development of medical image data from visualization to quantification and predictability, facilitating the efficient utilization of medical resources and the improvement of the hierarchical medical system, and ultimately helping to achieve the transformation from experience-based medicine to precision medicine. It has profound clinical application value and scientific research innovation significance.
[0003] Existing symmetric masking attention mechanisms force models to focus on symmetrical regions of the left and right hemispheres, which can cause severe interference when processing unilateral stroke lesions due to their inherent asymmetry. Therefore, a deep learning-based medical image processing method is proposed. Summary of the Invention
[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a deep learning-based medical image processing method.
[0005] To achieve the above objectives, the present invention adopts the following technical solution: A deep learning-based medical image processing method includes the following steps: S1: Acquire medical images including brain CT, MRI and PET, and perform multimodal adaptive preprocessing to obtain preprocessed multimodal image data; S2: The image to be registered in the preprocessed multimodal image data is registered to the standard space through a deep learning registration network to obtain the registered multimodal image data; S3: The registered multimodal image data is input into the Transformer model optimized based on the adaptive asymmetric attention mechanism for training and inference. The model outputs an initial semantic segmentation heatmap. S4: Post-process the initial semantic segmentation heatmap to derive the final segmentation prediction result.
[0006] The above further includes: Furthermore, the multimodal adaptive preprocessing in S1 involves cropping the pixel values of CT and MRI image data to a preset reasonable range and then performing Z-score regularization.
[0007] Furthermore, the specific steps of adaptive preprocessing for PET image data are as follows: The raw pixel values of PET image data are converted into standardized intake values based on body weight or body surface area for quantitative standardization. The standard ingestion value is subjected to a nonlinear logarithmic function transformation to compress its high dynamic range and enhance the contrast in the low value region. The transformed image data is then subjected to Z-score regularization. The regularized low-resolution PET modal images are input into a deep learning super-resolution model pre-trained on a brain image dataset for resolution upscaling, generating high-resolution PET image data that matches the spatial resolution of MRI images.
[0008] Furthermore, when the PET image is a dynamic scan image, after adaptive preprocessing, high-resolution PET image data from multiple consecutive time points are stacked in the time dimension to construct a four-dimensional tensor data containing time series information. This four-dimensional tensor data will be used as input for subsequent model training to utilize the dynamic characteristics of tracer metabolism in the lesion area over time.
[0009] Furthermore, the deep learning registration network in step S2 adopts an end-to-end architecture based on unsupervised learning (a network based on VoxelMorph or CycleMorph architecture), and the registration process includes: The deep learning registration network takes the image to be registered and the selected standard template image as input, and learns and predicts a dense, nonlinear voxel-level deformation field from the space of the image to be registered to the space of the standard template image through a convolutional neural network. The image to be registered is spatially transformed using the voxel-level deformation field, so that the image to be registered is aligned with the standard template image in terms of anatomical structure, and the registered multimodal image data is output.
[0010] Furthermore, the Transformer model in step S3 integrates an adaptive asymmetric attention mechanism, the computation process of which includes: Extract prior feature maps reflecting brain anatomy from anatomical images that have been registered to standard space using convolution operations; The prior feature map is concatenated and fused with the output features of the previous level of the Transformer decoding layer in the channel dimension. The concatenated fused features are passed through a learnable convolutional layer with a Sigmoid activation function to generate an adaptive weight map with a value range between zero and one, which dynamically reflects the symmetry weights at each position. The adaptive weight map is used to perform linear weighted fusion on the probability distribution map and its horizontal mirror flip map obtained by transforming the output features of the previous level, thereby generating an attention mask that adaptively adjusts the strength of the symmetry constraint according to the image content. The adaptive mask M_adapt is then applied to the attention calculation of the Transformer decoding layer.
[0011] Furthermore, when the input image data contains a dynamic PET four-dimensional tensor, the Transformer model integrates an independent temporal attention mechanism module on the basis of the spatial attention mechanism. The temporal attention mechanism module is responsible for performing sequence modeling on the four-dimensional tensor data in the temporal dimension, calculating the correlation weights between features at different time points to capture the temporal change patterns of tracer metabolism, and fusing the temporal information with the features extracted by the spatial attention mechanism to jointly output a feature map containing spatiotemporal context information.
[0012] Furthermore, the loss function for model training in step S3 consists of a weighted sum of three loss terms. The first part is the cross-entropy loss, which measures the accuracy of pixel-level classification. The second part is the Dice loss, which optimizes the overlap of segmented regions. The third part is a specially designed modal consistency loss, which aims to constrain and encourage the model to maintain the greatest possible consistency in the spatial distribution of the segmentation results predicted by the model for the same set of input data from different imaging modalities, thereby improving the robustness and reliability of the model in multimodal scenarios.
[0013] Furthermore, the post-processing in step S4 specifically includes: The initial semantic segmentation heatmap output by the Transformer model is subjected to pixel-by-pixel category decision, and the category with the highest score in its channel dimension is selected as its predicted label for each pixel, thereby generating a two-dimensional or three-dimensional discrete segmentation mask. The discrete segmentation mask is subjected to morphological dilation and erosion operations in sequence to smooth the boundaries and fill small holes. Then, an algorithm based on breadth-first search or connected component labeling is used to identify and separate independent lesion instances that are not connected in space. When the PET image is a dynamic scan image, a graph cut algorithm based on optical flow or temporal coherence constraints is applied in the time dimension to ensure that the segmentation results of the same lesion instance at different time points have good temporal consistency and stability.
[0014] The present invention has the following beneficial effects: 1. In this invention, an adaptive asymmetric attention mechanism is adopted. By introducing anatomical prior features and generating a learnable asymmetric weight map, the model can dynamically and adaptively determine when, where, and to what extent to rely on symmetry information. For healthy or symmetrical tissues, the model can maintain a high symmetry weight to utilize prior knowledge. For identified unilateral lesion areas, the model will automatically reduce the attention weight of its symmetrical areas, thereby effectively avoiding the generation of symmetrical false positive lesions in the healthy brain region.
[0015] 2. In this invention, the nonlinear registration network based on deep learning can better handle the complex nonlinear deformations caused by different imaging principles between images of different modalities. At the same time, for dynamic PET data that includes a time dimension, by introducing a time attention mechanism, the model can not only analyze spatial features, but also capture the dynamic changes in tracer metabolism. This makes the segmentation results not only based on the signal intensity at a single time point, but also consider its time dynamic characteristics, thereby enabling the identification of lesions that are not obvious in static images but show abnormalities in dynamic processes, thus improving the sensitivity and clinical value of the method. Attached Figure Description
[0016] Figure 1 This is a flowchart illustrating the steps of a deep learning-based medical image processing method proposed in this invention. Detailed Implementation
[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] Please see Figure 1 As shown, this invention is a medical image processing method based on deep learning, comprising the following steps: S1: Acquire medical images including brain CT, MRI and PET, and perform multimodal adaptive preprocessing to obtain preprocessed multimodal image data; S2: The image to be registered in the preprocessed multimodal image data is registered to the standard space through a deep learning registration network to obtain the registered multimodal image data; S3: The registered multimodal image data is input into the Transformer model optimized based on the adaptive asymmetric attention mechanism for training and inference. The model outputs an initial semantic segmentation heatmap. S4: Post-process the initial semantic segmentation heatmap to derive the final segmentation prediction result.
[0019] In one embodiment, the multimodal adaptive preprocessing in S1 involves cropping the pixel values of CT and MRI image data to a preset reasonable range and then performing Z-score regularization.
[0020] In one embodiment, the specific steps of adaptive preprocessing for PET image data are as follows: The raw pixel values of PET image data are converted into standardized intake values based on body weight or body surface area for quantitative standardization. The standard ingestion value is subjected to a nonlinear logarithmic function transformation to compress its high dynamic range and enhance the contrast in the low value region. The transformed image data is then subjected to Z-score regularization. The regularized low-resolution PET modal images are input into a deep learning super-resolution model pre-trained on a brain image dataset for resolution upscaling, generating high-resolution PET image data that matches the spatial resolution of MRI images.
[0021] In one embodiment, when the PET image is a dynamic scan image, after adaptive preprocessing, high-resolution PET image data from multiple consecutive time points are stacked in the time dimension to construct a four-dimensional tensor data containing time series information. This four-dimensional tensor data will be used as input for subsequent model training to utilize the dynamic characteristics of tracer metabolism in the lesion area over time.
[0022] In one embodiment, the deep learning registration network in step S2 adopts an end-to-end architecture based on unsupervised learning (a network based on VoxelMorph or CycleMorph architecture), and the registration process includes: The deep learning registration network takes the image to be registered and the selected standard template image as input, and learns and predicts a dense, nonlinear voxel-level deformation field from the space of the image to be registered to the space of the standard template image through a convolutional neural network. The image to be registered is spatially transformed using the voxel-level deformation field, so that the image to be registered is aligned with the standard template image in terms of anatomical structure, and the registered multimodal image data is output.
[0023] In one embodiment, the Transformer model in step S3 integrates an adaptive asymmetric attention mechanism, the computation process of which includes: Extract prior feature maps reflecting brain anatomy from anatomical images that have been registered to standard space using convolution operations; The prior feature map is concatenated and fused with the output features of the previous level of the Transformer decoding layer in the channel dimension. The concatenated fused features are passed through a learnable convolutional layer with a Sigmoid activation function to generate an adaptive weight map with a value range between zero and one, which dynamically reflects the symmetry weights at each position. The adaptive weighted map is used to linearly weight and fuse the probability distribution map obtained from the previous stage's output feature transformation and its horizontally mirrored flipped map, thereby generating an attention mask that adaptively adjusts the strength of the symmetry constraint according to the image content. This adaptive mask is then used to... It is applied to the attention calculation of the Transformer decoding layer in this layer.
[0024] In one embodiment, when the input image data contains a dynamic PET four-dimensional tensor, the Transformer model integrates an independent temporal attention mechanism module on the basis of the spatial attention mechanism. The temporal attention mechanism module is responsible for performing sequence modeling of the four-dimensional tensor data in the temporal dimension, calculating the correlation weights between features at different time points to capture the temporal change patterns of tracer metabolism, and fusing the temporal information with the features extracted by the spatial attention mechanism to jointly output a feature map containing spatiotemporal context information.
[0025] In one embodiment, the loss function for model training in step S3 consists of a weighted sum of three loss terms: the first part is the cross-entropy loss, which measures the accuracy of pixel-level classification; the second part is the Dice loss, which optimizes the overlap of segmented regions; and the third part is a specially designed modal consistency loss. The modal consistency loss aims to constrain and encourage the model to maintain the greatest possible consistency in the spatial distribution of the segmentation results predicted by the model for the same set of input data from different imaging modalities, thereby improving the robustness and reliability of the model in multimodal scenarios.
[0026] In one embodiment, the post-processing procedure in step S4 specifically includes: The initial semantic segmentation heatmap output by the Transformer model is subjected to pixel-by-pixel category decision, and the category with the highest score in its channel dimension is selected as its predicted label for each pixel, thereby generating a two-dimensional or three-dimensional discrete segmentation mask. The discrete segmentation mask is subjected to morphological dilation and erosion operations in sequence to smooth the boundaries and fill small holes. Then, an algorithm based on breadth-first search or connected component labeling is used to identify and separate independent lesion instances that are not connected in space. When the PET image is a dynamic scan image, a graph cut algorithm based on optical flow or temporal coherence constraints is applied in the time dimension to ensure that the segmentation results of the same lesion instance at different time points have good temporal consistency and stability.
[0027] In this embodiment: First, brain imaging data of the same patient was obtained from the hospital's imaging archive system, including: A cranial CT image used for rapid assessment of hemorrhage and calcification.
[0028] A single T1-weighted MRI image provides high-resolution information about anatomical structures.
[0029] A fluorodeoxyglucose (FDG)-PET image reflects the metabolic activity of brain tissue.
[0030] After acquiring these images, multimodal adaptive preprocessing is performed: For CT and MRI images: The pixel values (Henry units) of the CT images are cropped to a reasonable range of [-100, 400] to remove irrelevant extreme noise. Similarly, the MRI images are cropped based on their signal intensity. Then, Z-score regularization is performed on the cropped CT and MRI images respectively to make their pixel value distribution mean 0 and standard deviation 1. For PET images: Due to their wide dynamic range and low spatial resolution, the processing steps are as follows: Dynamic range adaptation: First, the raw PET count data is converted into standard uptake values to correct for differences in patient weight and injection dose. Then, the SUV value is logarithmically transformed (log(1+SUV)) to compress a wide range of values (e.g., 0-20) into an interval more suitable for deep learning model processing while preserving relative differences. Resolution enhancement: The regularized low-resolution PET image is input into a super-resolution generative adversarial network that has been pre-trained on a large number of brain image pairs. The resolution generative adversarial network learns the mapping from low-resolution to high-resolution and outputs a high-resolution PET image with enhanced spatial details that matches the resolution of the MRI image. Data integration: All preprocessed images (CT, MRI, high-resolution PET) were resampled to the same spatial size (256x256 pixels) to form a spatially aligned multimodal image dataset; Spatial registration was performed using a deep learning registration network based on the VoxelMorph architecture: Preprocessed MRI images were used as fixed images (i.e., standard spatial references), while CT images and high-resolution PET images were used as images to be registered. The MRI images and the PET images to be registered were input in pairs into the trained registration network, which analyzed the features of the two images and directly output a nonlinear deformation field. This deformation field defined the displacement vector required to map each point on the PET image to the MRI image space. This nonlinear deformation field is used to spatially transform the original PET image to be registered (using bilinear interpolation), generating a registered PET image that is precisely aligned anatomically with the MRI image. This process is repeated for the CT images, ultimately yielding a set of registered multimodal image data that is perfectly aligned in standard (MRI) space. The registered multimodal image data (CT, MRI, PET) are stitched together along the channel dimension and used as input to the adaptive asymmetric attention Transformer model. Model architecture: Transformer encoder-decoder model that integrates an adaptive asymmetric attention mechanism.
[0031] Workflow of the adaptive asymmetric attention mechanism (taking a certain layer of decoder as an example): The model receives the output feature map from the previous layer. ; Meanwhile, from the registered MRI images (as anatomical priors), an anatomical prior feature map A is extracted through a lightweight convolutional network. This feature map contains anatomical boundaries and structural information.
[0032] feature map The data is concatenated with the anatomical prior feature map A and then processed through a learnable 1x1 convolutional layer. The purpose of this convolutional layer is to learn and generate an asymmetric weight map. Asymmetric weighted graph Each pixel value is between 0 and 1, indicating whether the original feature (value close to 1) or its symmetrical feature (value close to 0) should be given more attention at that location. Subsequently, calculation The probability distribution after softmax activation is then horizontally mirrored. Using asymmetric weighted graphs The original probability distribution and its mirror image are weighted and fused to generate an adaptive mask. Specifically: = ⊙Original probability distribution + (1- The mirrored probability distribution is used as a mask to guide the attention calculation of the current layer. This mechanism enables the model to dynamically adjust its symmetry prior. For healthy or symmetrical brain regions, it may assign a higher symmetry weight, while for regions known to have unilateral lesions (through anatomical priors and image feature hints), it will automatically reduce the symmetry weight, thereby avoiding false positive segmentation results in the healthy hemisphere. Model training and loss function: The model is trained using a training dataset with pixel-level annotations. A combined loss function is used. =α∙ +β∙ +γ∙ ; (Cross-entropy loss) and (Dice loss) is responsible for optimizing segmentation accuracy; (Modal consistency loss) is key. It calculates the Dice similarity between two results when the model makes predictions using only MRI and only PET modalities, respectively. This encourages the model to learn consistent features across modalities and improves robustness. The model parameters are continuously updated by minimizing the comprehensive loss using the Adam optimizer, and the final output is the predicted initial semantic segmentation heatmap, where each pixel contains the probability of belonging to the background, ischemic region, or hemorrhage region. The initial semantic segmentation heatmap output by the model is post-processed to generate the final segmentation results that can be used clinically: Pixel-level classification: For each pixel in the heatmap, the category with the highest probability score among all categories (background, ischemia, hemorrhage) is selected as the predicted label for that pixel. This step transforms the probability map into a discrete segmentation mask of shape (H, W). For example, when it is necessary to differentiate between multiple independent lesion instances in clinical practice, the following steps are taken: A series of morphological dilation and erosion operations were performed on the segmented binary lesion areas to fill small pores and smooth the boundaries; A breadth-first search algorithm is used for connected component analysis. The image is traversed, and spatially connected pixels are marked as the same connected region, thereby separating different lesion instances and assigning a unique label to each instance. Results Export: The final segmentation prediction results, which can be semantic segmentation masks or instance segmentation maps, can be exported in standard medical image formats (such as DICOM) or visualization files for doctors to view and analyze in the diagnostic system.
[0033] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A deep learning-based medical image processing method, characterized in that, Includes the following steps: S1: Acquire medical images including brain CT, MRI and PET, and perform multimodal adaptive preprocessing to obtain preprocessed multimodal image data; S2: The image to be registered in the preprocessed multimodal image data is registered to the standard space through a deep learning registration network to obtain the registered multimodal image data; S3: Input the registered multimodal image data into the Transformer model optimized based on the adaptive asymmetric attention mechanism for training and inference. The Transformer model outputs an initial semantic segmentation heatmap. S4: Post-process the initial semantic segmentation heatmap to derive the final segmentation prediction result.
2. The medical image processing method based on deep learning according to claim 1, characterized in that, The multimodal adaptive preprocessing in S1 involves cropping the pixel values of CT and MRI image data to a preset reasonable range and then performing Z-score regularization.
3. The medical image processing method based on deep learning according to claim 2, characterized in that, The specific steps of adaptive preprocessing for PET image data are as follows: Convert the raw pixel values of PET image data into standard intake values based on body weight or body surface area; The standard ingestion value is subjected to a nonlinear logarithmic function transformation to compress its high dynamic range and enhance the contrast in the low value region. The transformed image data is then subjected to Z-score regularization. The regularized low-resolution PET modal images are input into a deep learning super-resolution model pre-trained on a brain image dataset for resolution upscaling, generating high-resolution PET image data that matches the spatial resolution of MRI images.
4. The medical image processing method based on deep learning according to claim 3, characterized in that, When the PET image is a dynamic scan image, after adaptive preprocessing, high-resolution PET image data from multiple consecutive time points are stacked in the time dimension to construct a four-dimensional tensor data containing time series information.
5. The medical image processing method based on deep learning according to claim 1, characterized in that, The deep learning registration network in step S2 adopts an end-to-end architecture based on unsupervised learning, and the registration process includes: The deep learning registration network takes the image to be registered and the selected standard template image as input, and learns and predicts a voxel-level deformation field from the space of the image to be registered to the standard template image through a convolutional neural network. The image to be registered is spatially transformed using the voxel-level deformation field, so that the image to be registered is aligned with the standard template image in terms of anatomical structure, and the registered multimodal image data is output.
6. The medical image processing method based on deep learning according to claim 1, characterized in that, The Transformer model in step S3 integrates an adaptive asymmetric attention mechanism, the computation process of which includes: Extract prior feature maps reflecting brain anatomy from anatomical images that have been registered to standard space using convolution operations; The prior feature map is concatenated and fused with the output features of the previous level of the Transformer decoding layer in the channel dimension. The concatenated fused features are passed through a learnable convolutional layer with a Sigmoid activation function to generate an adaptive weight map with a value range between zero and one, which dynamically reflects the symmetry weights at each position. The adaptive weight map is used to linearly weight and fuse the probability distribution map obtained from the previous level output feature transformation and its horizontal mirror flip map, thereby generating an attention mask that adaptively adjusts the strength of the symmetry constraint according to the image content.
7. The medical image processing method based on deep learning according to claim 4, characterized in that, When the input image data contains a dynamic PET four-dimensional tensor, the Transformer model integrates an independent temporal attention mechanism module on the basis of the spatial attention mechanism. The temporal attention mechanism module is responsible for performing sequence modeling on the four-dimensional tensor data in the time dimension, calculating the correlation weight between features at different time points, and fusing the temporal information with the features extracted by the spatial attention mechanism to jointly output a feature map containing spatiotemporal context information.
8. The medical image processing method based on deep learning according to claim 1, characterized in that, In step S3, the loss function for model training consists of a weighted sum of three loss terms: the first part is the cross-entropy loss, which measures the accuracy of pixel-level classification; the second part is the Dice loss, which optimizes the overlap of segmented regions; and the third part is a specially designed modality consistency loss.
9. The medical image processing method based on deep learning according to claim 1, characterized in that, The post-processing in step S4 specifically includes: The initial semantic segmentation heatmap output by the Transformer model is subjected to pixel-by-pixel category decision, and the category with the highest score in its channel dimension is selected as its predicted label for each pixel, thereby generating a two-dimensional or three-dimensional discrete segmentation mask. Morphological dilation and erosion operations are sequentially performed on the discrete segmentation mask, and then an algorithm based on breadth-first search or connected component labeling is used to identify and separate independent lesion instances that are not connected in space. When the PET image is a dynamically scanned image, a graph cut algorithm based on optical flow or temporal coherence constraints is applied in the time dimension.