A deep learning-based modality-adaptive medical image segmentation method

Abstract

The application provides a modal adaptive medical image segmentation method based on deep learning, and belongs to the technical field of medical image processing.The application realizes pixel-level segmentation prediction under different medical imaging modalities by constructing a deep learning network comprising multi-scale feature extraction, modal adaptive feature modeling and segmentation decoding.In the training stage, the feature extraction module extracts local structure and semantic features of the medical image by using a deep neural network, and then effectively models and aligns the cross-modal differences by using a modal adaptive feature modeling mechanism.The segmentation decoding module generates a segmentation result consistent with the size of the input image by using a space reconstruction and query-driven feature fusion method.The application does not need to manually adjust parameters for a specific imaging modality, has good cross-modal adaptability and segmentation precision, and is suitable for automatic segmentation application scenarios of multi-modal medical images.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and in particular relates to a modality-adaptive medical image segmentation method based on deep learning. Background Technology

[0002] Medical image segmentation is one of the core technologies in the field of medical image processing and analysis. Its purpose is to automatically and accurately extract organs, tissues, or lesion regions from medical images, providing crucial information for disease diagnosis, treatment planning, intraoperative navigation, and efficacy evaluation. With the rapid development of medical imaging technology, the types of medical imaging modalities widely used in clinical practice are constantly increasing, including but not limited to computed tomography (CT), magnetic resonance imaging (MRI), ultrasound imaging, and various variants. Different imaging modalities exhibit significant differences in imaging mechanisms, grayscale distribution, spatial resolution, noise characteristics, and tissue contrast, resulting in the same anatomical structure presenting drastically different appearances in different modalities.

[0003] Furthermore, the targets in medical image segmentation are highly diverse, including both structurally intact, large-scale organ regions and tiny lesions or nodules embedded in complex backgrounds. These targets differ significantly in morphology, scale, boundary continuity, and semantic consistency, further increasing the complexity of the segmentation task. Therefore, medical image segmentation methods not only need strong feature representation capabilities but should also be able to adaptively model segmentation based on the imaging modality and structural characteristics of the input image.

[0004] Traditional medical image segmentation methods typically rely on manually designed features and rules, such as thresholding, region growing, level sets, and statistical model-based methods. These methods often depend on human experience, are sensitive to changes in imaging conditions and modal differences, and struggle to maintain stable performance in multimodal medical imaging scenarios. With the development of deep learning, medical image segmentation methods based on convolutional neural networks and Transformer architectures have gradually become mainstream. These methods generally employ a "feature extraction-decoding and reconstruction" paradigm, extracting high-level semantic features through a backbone network and generating corresponding segmentation masks using a decoding structure, achieving better segmentation results under specific datasets and single-modal conditions.

[0005] However, existing deep learning-based medical image segmentation methods still have significant shortcomings in terms of modality adaptability. Most methods rely on data distributions specific to a particular modality or task during training, and their segmentation performance often deteriorates significantly when the model is directly applied to other imaging modalities or anatomical sites. To improve the model's adaptability, some studies have introduced segmentation strategies based on human prompts or interactive information, guiding the model to focus on the target region by providing prior information. While these methods improve segmentation results to some extent, they rely on human intervention and cannot meet the demands of fully automated, high-efficiency segmentation in clinical applications.

[0006] On the other hand, although existing network structures are constantly evolving in feature extraction, models often struggle to fully retain key details across different modalities during multi-scale feature interaction and reconstruction. This can lead to problems such as insufficient semantic consistency, blurred boundaries, and unstable representation of features within the same category. In particular, in cross-modal scenarios, the distribution of high-frequency information differs significantly between different modalities, making it difficult for models to form consistent and discriminative feature representations for the same semantic target, thus affecting the final segmentation accuracy.

[0007] Furthermore, medical image data commonly suffers from high annotation costs, limited sample sizes, and imbalanced class distribution in practical applications. Small target structures account for a relatively low proportion of images, making models susceptible to interference from background categories during training, resulting in insufficient feature discrimination capabilities. In recent years, research on improving adaptive supervision strategies to enhance feature discriminative power has gradually increased; however, effectively applying such strategies to improve the model's adaptability to different imaging modalities still faces certain challenges.

[0008] In summary, existing medical image segmentation techniques still struggle to simultaneously achieve segmentation accuracy, stability, and full automation when facing multiple imaging modalities, multiple target scales, and complex clinical scenarios. This is especially true when there are significant modal differences, leading to insufficient model generalization ability. Therefore, it is necessary to propose a deep learning-based modality-adaptive medical image segmentation method. This method would enable the model to adaptively adjust the segmentation modeling process based on the imaging characteristics of the input medical image, thereby achieving more stable and reliable segmentation results across different medical imaging modalities and application scenarios. Summary of the Invention

[0009] The purpose of this invention is to provide a deep learning-based modality-adaptive medical image segmentation method to overcome the problems of insufficient adaptability of existing medical image segmentation techniques under different imaging modalities, strong reliance on human priors, and low segmentation stability in complex scenes, so as to achieve automatic and accurate segmentation of target structures under various medical imaging modalities.

[0010] To achieve the above objectives, this invention provides a deep learning-based modality-adaptive medical image segmentation method, comprising the following steps:

[0011] Step 1: Acquisition of medical image data.

[0012] At least one medical image is acquired as input. The medical image originates from medical imaging equipment or a medical image dataset, and includes, but is not limited to, computed tomography (CT) images, magnetic resonance imaging (MRI) images, or other two-dimensional medical images. The segmentation target corresponding to the medical image can be an organ structure, a tissue region, or a lesion region.

[0013] Step 2: Medical image preprocessing.

[0014] The input medical images are preprocessed, including size adjustment, intensity normalization, and necessary spatial transformation, to eliminate differences in numerical distribution caused by different imaging modalities and acquisition conditions, and to provide a unified data input format for subsequent network modeling.

[0015] Step 3: Construct a modality-adaptive medical image segmentation network.

[0016] Construct an end-to-end medical image segmentation neural network, wherein the neural network includes at least the following modules:

[0017] Feature extraction module: The feature extraction network Swin-Transformer and multi-scale feature fusion convolutional neural network are used to extract multi-scale features from the input medical image to obtain feature representations containing spatial and semantic information; the feature extraction network is used to uniformly represent image features under different imaging modalities to reduce the impact of modal differences on feature expression.

[0018] Modality-adaptive feature modeling module: By introducing learnable query features, this module interacts with the multi-scale features output by the feature extraction module to model the model adaptively based on the imaging characteristics of the current input medical image, thereby enhancing the model's adaptability to different imaging modalities and structural differences. This module is used to autonomously adjust the feature response mode based on the statistical characteristics and structural features of the input medical image without explicitly introducing modality labels.

[0019] Segmentation and decoding module: Based on the output of the modality adaptive feature modeling module, it generates the corresponding segmentation feature representation, interacts with the learnable query, and performs joint modeling of query features and spatial features through pixel-level feature fusion operation to obtain the final segmentation result.

[0020] Step 4: Train the neural network.

[0021] The neural network is trained using labeled medical images, specifically including:

[0022] The preprocessed medical image is input into the neural network, and sequentially passes through the feature extraction, modality adaptive feature modeling, and segmentation decoding modules to obtain the segmentation prediction result of the medical image. The backpropagation algorithm is used to optimize the network parameters, so that the neural network gradually learns the segmentation feature representation under different imaging modalities.

[0023] Step 5: Repeat step 4 until the training error of the neural network converges to below a preset threshold or the number of training rounds reaches a preset upper limit; save the trained network parameters as a pre-trained model file.

[0024] Step 6: Load the pre-trained model and perform the same preprocessing operation as in Step 2 on the new medical image; input the preprocessed medical image into the neural network and output the corresponding medical image segmentation result.

[0025] Beneficial effects

[0026] Compared with existing technologies, the present invention has the following advantages: The present invention enables the model to automatically adjust the feature representation according to the imaging characteristics of the input medical image through a modal adaptive feature modeling mechanism, thereby improving the stability and generalization ability of cross-modal segmentation; The present invention can achieve fully automatic medical image segmentation without the need for manual prompts or additional fine-tuning for different modalities; The present invention can obtain more consistent and reliable segmentation results under complex structures and diverse target conditions, and is applicable to a variety of medical imaging scenarios. Attached Figure Description

[0027] Figure 1 Flowchart of a Deep Learning-Based Modality Adaptive Medical Image Segmentation Method

[0028] Figure 2 A schematic diagram of a deep learning-based modality adaptive medical image segmentation method

[0029] Figure 3 A deep learning-based modality-adaptive medical image segmentation method: Modality-adaptive feature modeling diagram. Detailed Implementation

[0030] To more clearly illustrate the deep learning-based modality adaptive medical image segmentation method of the present invention, the implementation of the present invention will be described in detail below with reference to specific embodiments. These embodiments are specific implementation methods and operation procedures given without departing from the technical solution of the present invention, but the scope of protection of the present invention is not limited to the following embodiments.

[0031] This example proposes a deep learning-based modality-adaptive medical image segmentation method, which includes, but is not limited to, the following functional modules: medical image acquisition module, image preprocessing module, feature extraction module, modality-adaptive feature modeling module, and segmentation decoding module.

[0032] Step 1: Acquire at least one two-dimensional medical image as input image. The medical image may be derived from data acquired by medical imaging equipment or from a pre-constructed medical image dataset. The medical image includes, but is not limited to, computed tomography (CT) images, magnetic resonance imaging (MRI) images, and their different sequence formats. The medical image contains organ structures, tissue regions, or lesion targets to be segmented.

[0033] Step 2: Perform preprocessing on the input medical images to reduce the impact of different imaging modalities and acquisition conditions on the segmentation results.

[0034] Specifically, the preprocessing process includes the following steps: First, the medical image is resized or resampled to meet the input requirements of the neural network; second, the pixel intensity of the image is normalized to map the image grayscale value to a preset numerical range to unify the numerical distribution range of different modal images; during the training phase, random spatial transformation or intensity perturbation operations can be further introduced to enhance the model's adaptability to different imaging conditions.

[0035] Step 3: Input the medical image processed in Step 2 into the deep learning model. The neural network mainly includes a feature extraction module, a modality adaptive feature modeling module, and a segmentation and decoding module, as detailed below:

[0036] Feature extraction module: The feature extraction module is used to extract multi-scale semantic features from the input medical image.

[0037] The feature extraction module adopts a hierarchical visual backbone network based on Swing Transformer and combines it with a multi-scale deformable attention mechanism to achieve efficient and robust multi-scale feature modeling.

[0038] Specifically, the input image is first divided into non-overlapping patches, which are then mapped to high-dimensional feature representations via linear embedding, serving as network input. The backbone network employs alternating stacked window multi-head self-attention and shifted window multi-head self-attention structures. This achieves continuous spatial attention modeling within windows and between adjacent windows while maintaining manageable computational complexity, thus balancing local detail awareness with global context modeling capabilities.

[0039] As the network depth increases, the feature map resolution is gradually reduced and the channel dimension is increased through the Patch Merging operation to construct a multi-level feature representation. Finally, four feature maps of different scales are output to represent semantic information at different spatial resolutions.

[0040] Building upon this foundation, a multi-scale deformable attention module is introduced to model low- and medium-resolution features. This module achieves efficient information exchange across scale features through adaptive weighting of finite sampling locations in the feature space, enhancing the ability to express small-scale structural and local semantic differences while maintaining manageable computational complexity. The multi-scale features reconstructed by the deformable attention module are progressively upsampled and fused with high-resolution features to obtain the output of the feature extraction layer. ,in This represents the feature map at the l-th scale. This results in a feature representation that combines global semantic consistency with local detail resolution, providing stable feature support for subsequent pixel-level segmentation prediction tasks.

[0041] Modal Adaptive Feature Modeling Module: The modal adaptive feature modeling module is used to adaptively model features under different imaging modalities.

[0042] This module introduces learnable query features and interacts with the image features output by the feature extraction module to perform calculations. It dynamically adjusts the feature response weights based on the imaging characteristics of the current input medical image, thereby enhancing the model's adaptability to different imaging modalities and structural differences. For example... Figure 3 As shown, learnable query vectors Each time, a feature map is extracted by the feature extraction module. Information exchange is achieved through three components: Cross attention, Galerkin attention, and Feed Forward Network (FFN). Here, N represents the total number of categories in the image dataset to be segmented, and E represents the number of feature channels. The specific calculation process is shown below. This represents the initial query characteristics input. The output of the first Cross attention interaction can be represented as follows:

[0043]

[0044] in, This represents the hierarchical index of the module. , , .in , These represent the spatial resolution of the feature map at that layer. (Function) , , All are linear transformations.

[0045] Furthermore, the update form of the query features in the Galerki attention part is as follows:

[0046]

[0047] in, and , , , Representation layer normalization operation, function , , Both are linear transformations. It is a feedforward neural network. This is the activation function.

[0048] In output It integrates multi-scale image features Therefore, the model can autonomously model global context information and long-range dependencies in different modalities without human prompting, thereby improving the consistency and discriminativeness of feature representation.

[0049] Segmentation Decoding Module: The segmentation decoding module is used to map the features output by the modality adaptive feature modeling module to the final medical image segmentation result.

[0050] Specifically, such as Figure 2 As shown, the segmentation and decoding module first performs a learnable four-neighborhood interpolation space reconstruction operation on the input features, mapping multi-scale features pixel by pixel to a feature map with the same spatial size as the original medical image, in order to recover the high-frequency information lost in the original image during feature extraction. Then, the reconstructed feature map is multiplied pixel by pixel with the query features output by the modality adaptive feature modeling module. Through pixel-level feature fusion, the query features and spatial features are jointly modeled to guide the generation of pixel-level semantic responses, resulting in a segmentation probability map. Where H and W are the resolution (height and width) of the original image, respectively. Finally, the final medical image segmentation result is obtained by taking the index corresponding to the maximum class probability at each pixel location.

[0051] Step 4: During the training phase, the neural network is trained using medical images with segmentation annotations. The specific process is as follows:

[0052] First, the preprocessed medical image from step two is input into a neural network, where feature representations are obtained through a feature extraction module and a modality adaptive feature modeling module. Then, the features are input into a segmentation decoding module, which outputs the corresponding medical image segmentation prediction result. Next, the error between the segmentation prediction result and the actual segmentation annotation is calculated using a segmentation loss function, where the segmentation loss formula is as follows:

[0053]

[0054] Among them, the adaptive normalization term Defined as:

[0055]

[0056]

[0057] In the above formula, Indicates the first Sub-regions; For the first Predicted value for each pixel; This is the actual label for that pixel; This is a stabilizing term used to avoid the denominator being zero. yes Numerical value The enhancement factor increases with the number of training rounds.

[0058] Regional weighting coefficient and Defined as:

[0059]

[0060] in, and For hyperparameters; Indicates the first False positive error in each sub-region This indicates the false negative error.

[0061] Through this adaptive weighting method, the model can dynamically adjust the penalty term according to the prediction difficulty of different regions, thereby achieving a better balance between overall segmentation accuracy and boundary detail representation.

[0062] Finally, based on the aforementioned error loss value, the Adam optimization algorithm is used to calculate the gradient of the loss function with respect to the neural network parameters. Using this gradient information, the network parameters are progressively adjusted to minimize the loss function value, thereby gradually improving the model's segmentation accuracy and stability.

[0063] Step 5: Repeat step 4 until the training error of the neural network converges to below the preset threshold or the number of training iterations reaches the preset upper limit; save the trained neural network parameters as a pre-trained model file.

[0064] Step 6: During the inference phase, load the saved pre-trained model parameters and perform the same preprocessing operations as in Step 2 on the new medical image; input the preprocessed medical image into the neural network, and output the corresponding medical image segmentation result end-to-end. This implementation method can achieve fully automated processing from input medical image to segmentation result output, and is suitable for various medical imaging modalities and clinical application scenarios.

[0065] Through the above specific implementation methods, the present invention realizes a deep learning-based modality-adaptive medical image segmentation method, which effectively improves the segmentation stability and generalization ability of the model under different imaging modal conditions, and overcomes the problems of strong dependence on human priors and insufficient modality adaptability in the prior art.

[0066] The above description is only a preferred embodiment of the present invention. Any equivalent transformations or substitutions made within the scope of the technical concept and claims of the present invention should be included within the scope of protection of the present invention.

Claims

1. A modality-adaptive medical image segmentation method based on deep learning, characterized in that, Includes the following steps: Step 1: Acquire medical image data; The medical image data may include medical images from different medical imaging modalities, including one or more of CT, MRI, ultrasound, or RGB images; Step 2: Preprocess the medical images; Input medical images are subjected to size unification, intensity normalization, or interpolation resampling to reduce the differences in spatial resolution and grayscale distribution between different imaging modalities; Step 3: Construct an end-to-end deep learning segmentation network, wherein the deep learning segmentation network includes: Feature extraction module, modality adaptive feature modeling module, and segmentation decoding module; Step 4: Input the preprocessed medical image into the deep learning segmentation network, and let it pass through the feature extraction module, the modality adaptive feature modeling module and the segmentation decoding module in sequence to generate pixel-level segmentation results; Step 5: Based on the difference between the predicted segmentation results and the labeled ground truth, train and optimize the deep learning segmentation network; Step 6: Use the trained network model to perform segmentation reasoning on medical images under unknown modalities or different modal conditions to obtain the final medical image segmentation results.

2. The modality-adaptive medical image segmentation method based on deep learning according to claim 1, characterized in that, In step two, the preprocessing includes: intensity normalization or distribution alignment of medical images of different modalities; spatial resolution adjustment of medical images by interpolation or resampling to reduce scale differences between different modalities; and in the training phase, joint training by introducing multimodal or cross-modal samples to enhance the network's adaptability to modal changes.

3. The modality-adaptive medical image segmentation method based on deep learning according to claim 1, characterized in that, In step three, the feature extraction module performs feature modeling based on the local window attention mechanism and realizes feature interaction between adjacent regions by changing the window position, so as to enhance the stability and expressive power of feature representation; the feature extraction module is used to generate multi-scale feature representations to provide input for modality adaptive feature modeling.

4. The modality-adaptive medical image segmentation method based on deep learning according to claim 1, characterized in that, In step three, the modality adaptive feature modeling module includes: introducing a multi-scale deformable attention mechanism to low- and medium-resolution features; performing feature aggregation by adaptively selecting key feature locations related to different imaging modalities; enhancing the feature's ability to represent changes in tissue structure under different modalities through multi-scale attention fusion and progressive upsampling operations; and laterally connecting the fused features with high-resolution features to form a unified feature representation with modality discriminativeness.

5. The modality-adaptive medical image segmentation method based on deep learning according to claim 4, characterized in that, The deformable attention mechanism learns the dynamic offset parameters of the attention sampling position to adapt to the differences in structural representation and contrast distribution of different medical imaging modalities, thereby improving cross-modal feature alignment capability.

6. The modality-adaptive medical image segmentation method based on deep learning according to claim 1, characterized in that, In step three, the segmentation and decoding module includes: performing a learnable four-neighbor interpolation space reconstruction operation on the features output by the modality adaptive feature modeling module to generate a feature map with the same spatial size as the input medical image; and fusing the reconstructed feature map with the Query features generated by the modality adaptive feature modeling module pixel by pixel to obtain a pixel-level classification probability map.

7. The modality-adaptive medical image segmentation method based on deep learning according to claim 6, characterized in that, The Query feature is a learnable high-dimensional semantic representation used to characterize the discrimination information of different categories under different medical imaging modalities, and achieves modality adaptive segmentation through interaction with image features.

8. The modality-adaptive medical image segmentation method based on deep learning according to claim 1, characterized in that, Step five, network training includes: constructing a loss function to constrain segmentation consistency across different modalities, wherein the segmentation loss formula is as follows: Among them, the adaptive normalization term Defined as: In the above formula, Indicates the first Sub-regions; For the first Predicted value for each pixel; This is the actual label for that pixel; This is a stabilizing term used to avoid the denominator being zero. yes Numerical value The enhancement factor increases with the number of training rounds. Regional weighting coefficient and Defined as: in, and For hyperparameters; Indicates the first False positive error in each sub-region This indicates the false negative error. Based on the aforementioned error loss value, the Adam optimization algorithm is used to calculate the gradient of the loss function with respect to the neural network parameters. Using this gradient information, the network parameters are progressively adjusted to minimize the loss function value, thereby gradually improving the model's segmentation accuracy and stability.

Images