An Unsupervised Cross-Modal CT-MRI Image Segmentation Method and System

By combining adaptive frequency band separation and frequency band attention module, the problem of insufficient accuracy in cross-modal image segmentation of CT and MRI is solved, and efficient cross-modal feature alignment and segmentation performance improvement are achieved.

CN122866657BActive Publication Date: 2026-11-10CHONGQING UNIV OF TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202611356251.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-09-03
Publication Date
2026-11-10
Estimated Expiration
2046-09-03

AI Technical Summary

Technical Problem

Existing single-modal learning algorithms struggle to maintain high accuracy in cross-modal image segmentation of CT and MRI. Traditional frequency domain methods lack adaptive adjustment capabilities and ignore key anatomical structures and edge semantic information in the mid-to-high frequency bands, leading to a decrease in segmentation accuracy.

Method used

An unsupervised cross-modal CT-MRI image segmentation method is adopted. An adaptive frequency band separator decouples the image amplitude spectrum into low-frequency, mid-frequency and high-frequency spectra. A frequency band attention module is used for differential fusion mapping, and the segmentation network is optimized by combining Gram matrix statistical loss to achieve cross-modal feature alignment.

Benefits of technology

It significantly improves the model's generalization performance in unlabeled target domains, overcomes the image distortion and semantic ambiguity of traditional methods, and improves cross-modal segmentation accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122866657B_ABST
    Figure CN122866657B_ABST
Patent Text Reader

Abstract

This invention discloses an unsupervised cross-modal CT-MRI image segmentation method and system, belonging to the field of medical image processing technology. The method acquires source and target domain medical images of different modalities, performs Fourier transforms on both types of images to obtain amplitude and phase spectra, respectively; uses an adaptive frequency band separator to decouple the amplitude spectra into low-frequency, mid-frequency, and high-frequency amplitude spectra, and dynamically adjusts the frequency band boundaries according to the frequency characteristics of the target domain; applies style alignment, attention fusion, and noise suppression to the low-frequency, mid-frequency, and high-frequency amplitude spectra, respectively; reconstructs the fused amplitude spectra of each frequency band and the source domain phase spectrum into a spatially aligned image; and jointly optimizes the segmentation network and frequency domain processing module based on the statistical feature differences between the spatially aligned image and the target domain image and the segmentation loss. This invention can reduce the modal distribution differences between CT and MRI under unlabeled target domain conditions, improving the accuracy and generalization ability of cross-modal image segmentation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to an unsupervised cross-modal CT-MRI image segmentation method and system. Background Technology

[0002] Semantic segmentation of medical images has broad application value in disease screening, clinical diagnosis, treatment planning, and prognostic follow-up. However, due to the specificity of its data, it remains a highly challenging visual task. Unlike natural images, clinical medical images are usually acquired by different physical devices, exhibiting significant multimodal characteristics. Complex clinical diagnoses often require the integration of multimodal images, including CT, MRI, PET, and other imaging data, to provide complementary pathological and anatomical information.

[0003] However, heterogeneous imaging mechanisms inevitably lead to significant distributional differences between different modalities. The contrast, texture, and boundary features of the same organ or lesion vary drastically across different modalities, making it difficult for deep learning models trained on a single source modality to directly generalize to the target modality. Furthermore, high-standard pixel-level annotation of medical images heavily relies on physicians with specialized clinical knowledge, which is not only costly but also difficult to scale, resulting in a severe shortage of high-quality annotated data. As the two most commonly used three-dimensional tomographic imaging techniques in clinical practice, CT and MRI play complementary and crucial roles in disease screening, diagnosis, and treatment follow-up.

[0004] CT images are measured X-ray attenuation coefficient imaging provides excellent contrast for high-density tissues such as bone; while MRI, utilizing hydrogen nuclear magnetic resonance signals, excels at revealing the anatomical structures and pathological details of soft tissues. This fundamental difference in physical principles results in the same anatomical structure exhibiting completely different grayscale distributions, texture features, and boundary morphologies in the two types of images. Deep networks trained on a single modality source domain often overfit to its specific image statistical distribution. When directly transferred to another modality target domain, due to the lack of prior spatial awareness of the target domain, and the fact that the complex spatial high-frequency texture information in MRI images was not captured during CT training, the model is highly prone to making incorrect semantic judgments about unknown spatial frequency bands, leading to a significant decrease in segmentation accuracy.

[0005] Due to the fundamental differences in imaging principles between the source and target modalities, significant semantic mismatches and style differences exist between them. Existing single-modal learning algorithms often ignore the rich unlabeled information contained in the target domain. This leads to a significant degradation in the generalization performance of learned feature representations in classic semi-supervised deep learning methods such as MeanTeacher, FixMatch, and Cross Pseudo Supervision (CPS) when faced with drastic cross-modal domain shifts, making it difficult to maintain high-precision segmentation performance on the target modality. To reduce the distributional differences between modalities, traditional image enhancement methods mainly rely on manual geometric transformations and color space adjustments, lacking the ability to generalize to complex modalities. In recent years, many studies have begun to explore domain adaptation schemes based on image style transfer and feature alignment. Among them, Yang et al. proposed the Fourier Domain Adaptation (FDA) method, initially mainly used to solve cross-domain transfer from synthetic environments to real natural scenes. This method utilizes Fast Fourier Transform (FFT) to decompose an image into amplitude and phase spectra. While preserving the source domain phase spectrum (i.e., the prior anatomical structure), it replaces the low-frequency components of the target domain amplitude spectrum with those of the source domain using an empirical spatial mask. This approach aims to approximate the appearance of the source domain image with that of the target domain while maintaining semantic structure. However, this rigid frequency band mask replacement strategy heavily relies on costly trial-and-error experiments and lacks adaptive adjustment capabilities when facing heterogeneous cross-modal tasks. More importantly, existing frequency domain methods generally focus on coarse-grained alignment of low-frequency styles, neglecting the crucial anatomical structures and marginal semantic information contained in the mid-to-high frequency bands. These semantic gaps often exhibit significant nonlinear spatial differences across different medical modalities. Summary of the Invention

[0006] This invention aims to at least solve the technical problems existing in the prior art, and in particular, it innovatively proposes an unsupervised cross-modal CT-MRI image segmentation method and system.

[0007] To achieve the above-mentioned objectives of this invention, this invention provides an unsupervised cross-modal CT-MRI image segmentation method, comprising the following steps:

[0008] S1, acquire source and target domain medical image datasets; the source domain includes labeled samples and unlabeled samples, and the target domain includes unlabeled samples, which are different modalities in CT and MRI, respectively;

[0009] S2, perform two-dimensional fast Fourier transform on the source mode image and the target mode image respectively, decouple them to obtain their respective amplitude spectrum and phase spectrum, and retain the phase spectrum of the source mode;

[0010] S3 uses an adaptive frequency band separator to separate the source mode amplitude spectrum and the target mode amplitude spectrum into low-frequency amplitude spectrum, mid-frequency amplitude spectrum and high-frequency amplitude spectrum, respectively;

[0011] S4, implement differentiated frequency domain fusion mapping for amplitude spectra of each frequency band: low frequency amplitude spectra are aligned with modal style, mid frequency amplitude spectra are fused through frequency band attention module, and high frequency amplitude spectra suppress noise transmission;

[0012] S5 aggregates the reconstructed low-frequency amplitude spectrum, mid-frequency amplitude spectrum, and high-frequency amplitude spectrum into a synthetic amplitude spectrum, and performs a two-dimensional fast inverse Fourier transform with the source modal phase spectrum to generate a spatial domain aligned image that retains spatial semantic labels.

[0013] S6 constructs a Gram matrix statistical loss based on spatial domain aligned images and target modal images, and combines it with the segmentation model loss to jointly optimize the learnable parameters of the segmentation network and frequency domain mapping.

[0014] In the preferred embodiment of the above technical solution, the source domain medical image in S1 is the source modality image, and the target domain medical image is the target modality image; the source modality image is a CT image, and the target modality image is an MRI image; the labeled samples and unlabeled samples in the source domain undergo isomorphic processing during the frequency domain mapping stage, and the spatial semantic labels corresponding to the labeled samples are retained after frequency domain mapping.

[0015] In the preferred embodiment of the above technical solution, the adaptive frequency band separator in S3 employs a soft mask filter based on Gaussian bandpass filtering to generate smooth transition frequency edges; the adaptive frequency band separator generates symmetrical discrete frequency coordinates according to the spatial dimension of the input image, and constructs a normalized frequency band radial grid based on the distance of each discrete frequency coordinate relative to the zero-frequency center; the center position and bandwidth of each frequency band are learnable parameters, and are dynamically adjusted according to the frequency characteristics of the target modal image to separate the source modal amplitude spectrum and the target modal amplitude spectrum into low-frequency amplitude spectrum, mid-frequency amplitude spectrum and high-frequency amplitude spectrum, respectively.

[0016] In a preferred embodiment of the above technical solution, the low-frequency amplitude spectrum in S4 is used to characterize the global contrast and intensity distribution of the image; by covering the target modality low-frequency amplitude spectrum with the corresponding region of the source modality low-frequency amplitude spectrum, modal style alignment from the source modality image to the target modality image is achieved; the source modality high-frequency amplitude spectrum and the target modality high-frequency amplitude spectrum are weighted and constrained, and the introduction ratio of the target modality high-frequency amplitude spectrum is limited, so as to suppress the transmission of target modality noise and imaging artifacts while preserving the anatomical edge details of the source modality.

[0017] In a preferred embodiment of the above technical solution, the frequency band attention module in S4 receives the source mode intermediate frequency amplitude spectrum and the target mode intermediate frequency amplitude spectrum, extracts the cross-modal correlation features between the source mode intermediate frequency amplitude spectrum and the target mode intermediate frequency amplitude spectrum, and generates a frequency attention mask; and dynamically adjusts the fusion ratio of the source mode intermediate frequency amplitude spectrum and the target mode intermediate frequency amplitude spectrum according to the frequency attention mask to obtain the fused intermediate frequency amplitude spectrum.

[0018] In a preferred embodiment of the above technical solution, the frequency band attention module in S4 further includes: a fully convolutional architecture, comprising four cascaded two-dimensional convolutional layers and corresponding activation functions; the first three two-dimensional convolutional layers are used to extract cross-modal correlation features between the source modal mid-frequency amplitude spectrum and the target modal mid-frequency amplitude spectrum, and the last two-dimensional convolutional layer is used to map the cross-modal correlation features into a frequency attention mask with the same size as the input mid-frequency amplitude spectrum.

[0019] In the preferred embodiment of the above technical solution, the reconstructed low-frequency amplitude spectrum, mid-frequency amplitude spectrum, and high-frequency amplitude spectrum described in S5 are linearly combined to obtain a complete synthetic amplitude spectrum; the synthetic amplitude spectrum is combined with the source mode phase spectrum, which is preserved as a spatial invariant, and transformed to the spatial domain through a two-dimensional fast inverse Fourier transform to generate a spatial domain aligned image; the spatial topology of the spatial domain aligned image remains unchanged relative to the source mode image, and the spatial semantic labels are completely preserved.

[0020] In a preferred embodiment of the above technical solution, the joint optimization in S6 includes: extracting shallow features of the spatial domain aligned image and the target modality image using a pre-trained feature extraction network with fixed parameters, and constructing the Gram matrix statistical loss based on the differences in second-order statistical features represented by the Gram matrix of the shallow features; combining the Gram matrix statistical loss with the segmentation model loss into an overall loss, and jointly updating the weight parameters of the segmentation network and the learnable parameters of the adaptive frequency band separator and the frequency band attention module during the end-to-end single-stage training process.

[0021] The present invention also discloses a computer system, comprising:

[0022] processor;

[0023] Memory used to store instructions executed by the processor;

[0024] The processor is configured to implement the method described above when executing the instructions.

[0025] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0026] By utilizing the Fourier amplitude spectrum decoupling mechanism, heterogeneous modal features are finely divided into multiple independent frequency bands, achieving accurate alignment of multi-scale semantic styles. By dynamically adjusting the fusion ratio of anatomical boundaries and tissue textures, image distortion and semantic ambiguity caused by traditional hard masking are effectively overcome. The generalization performance of the model in the unlabeled target domain is significantly improved with extremely low computational overhead. The cross-modal segmentation performance is significantly improved, fully verifying the effectiveness of this invention in cross-domain transfer tasks in CT and MRI.

[0027] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0028] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0029] Figure 1 This is a topological diagram of the overall architecture of the present invention;

[0030] Figure 2 This is a diagram of the frequency band attention module of the present invention;

[0031] Figure 3 This is a schematic diagram of the workflow of the adaptive band splitter of the present invention;

[0032] Figure 4 This is a comparative visualization of the generalization from CT modality to MRI modality on the MMWHS dataset (10% labeled data).

[0033] Figure 5 This is a comparative visualization of the generalization from CT modality to MRI modality on the MMWHS dataset (5% labeled data).

[0034] Figure 6 This is a comparative visualization of the generalization from MRI modality to CT modality on the MMWHS dataset (10% labeled data).

[0035] Figure 7 This is a comparative visualization of the generalization from MRI modality to CT modality on the MMWHS dataset (5% labeled data).

[0036] Figure 8 This is a comparative visualization of the generalization from CT modality to MRI modality on the AMOS dataset (10% labeled data).

[0037] Figure 9 This is a comparative visualization of the generalization from MRI modality to CT modality on the AMOS dataset of this invention (10% labeled data, showing the first slice);

[0038] Figure 10 This is a comparative visualization of the generalization from MRI modality to CT modality on the AMOS dataset of this invention (10% labeled data, showing the second slice);

[0039] Figure 11 This is a comparative visualization of the generalization from MRI modality to CT modality on the AMOS dataset (5% labeled data). Detailed Implementation

[0040] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0041] To achieve the above objectives, this invention provides an unsupervised multi-modal CT-MRI image segmentation method and system (Multi-Band Fine-Grained Alignment, MBFGA). This method is constructed as a fully differentiable parameterized transformation architecture, deeply coupled with the main segmentation network. During training iterations, a band splitter is first used to decouple the image amplitude spectrum into low-frequency, mid-frequency, and high-frequency amplitude spectra. Then, driven by feedback from the image segmentation task, explicit dynamic differential constraints are applied to different spectral bands. The method includes the following steps:

[0042] S1, obtain the source and target domain medical image datasets.

[0043] Suppose the dataset consists of two parts: a source domain and a target domain. The source domain dataset is defined as follows: This includes a set of labeled samples. Compared with unlabeled sample sets The target domain dataset is denoted as It consists entirely of unlabeled samples, used to provide statistical style features of the target modality, in which The source domain contains labeled data. The corresponding spatial semantic labels are used. Under the MBFGA strategy, input samples are divided only according to their modal space, and labeled and unlabeled data in the source domain undergo isomorphic processing during the frequency domain mapping stage.

[0044] S2 performs two-dimensional fast Fourier transform on the source modal image and the target modal image respectively, decouples them to obtain their respective amplitude spectrum and phase spectrum, and retains the phase spectrum of the source modal.

[0045] like Figure 1As shown, the MBFGA strategy decouples the source mode and target mode images into their respective amplitude and phase spectra using a two-dimensional fast Fourier transform, denoted as follows: as well as Subsequently, an adaptive band splitter was used to separate the amplitude spectra of the two into three feature representations with independent frequency scales, namely the source domain amplitude. Target domain amplitude ,in , and These represent low frequency, mid frequency, and high frequency, respectively. To address the characteristic distribution of different frequency bands, this invention employs a differentiated frequency domain fusion mapping scheme. After frequency division modulation, each frequency band is re-aggregated into a complete synthetic amplitude spectrum, which is then compared with the source mode phase spectrum, which is preserved as a spatial invariant. Perform a two-dimensional fast inverse Fourier transform to finally generate a spatially aligned image. ,in Represents spatially aligned images. This indicates a spatially aligned labeled image, where the labels correspond to spatial semantic labels. , This represents a spatially aligned, unlabeled image. This indicates alignment of the source domain to the target domain. Since this frequency domain mapping does not alter the spatial topology of the image, spatial semantic labels... The transformed spatially aligned labeled image pairs are fully preserved. Label-free images aligned with spatial domain The aforementioned Figure 1 The Chinese and English annotations and parameter labels are as follows: Mix indicates proportional mixing, Encoder indicates encoding, Teacher indicates the teacher model, Student indicates the student model, Weak Aug indicates weak augmentation, Strong Aug indicates strong augmentation, f(x), f(x1), and f(x2) represent encoders and decoders with different network structures, respectively. e ) represents the network encoder, f(x) a1 f(x) a2 Z1 and Z2 represent different prediction results, respectively, representing the decoders of different networks. , Representing different pseudo-label data, denoted by Gram matrix statistical loss function, MeanTeacher, FixMatch, CPS (Cross PseudoSupervision), and CCT (Cross-Consistency Training). All images reconstructed via the MBFGA strategy will be used as standard input for supervised training of the segmentation network.

[0046] S3 uses an adaptive frequency band separator to separate the source mode amplitude spectrum and the target mode amplitude spectrum into low-frequency amplitude spectrum, mid-frequency amplitude spectrum and high-frequency amplitude spectrum, respectively.

[0047] To address the distortion in transition regions caused by hard truncation in traditional 2D Fast Fourier Transform (FFT), this step employs a soft mask filter based on Gaussian bandpass filtering. This generates smooth frequency edges to preserve natural image texture. The band splitter automatically fits the frequency band data to determine the dynamic boundaries of each band; details of its internal data processing flow can be found in [link to documentation]. Figure 3 .

[0048] Let the input image be The spatial dimension is The dimensions of each dimension are denoted as vectors. The spatial dimension The value is 2. Based on the properties of the two-dimensional Fast Fourier Transform, in each dimension... The system generates symmetrical discrete frequency coordinates and obtains the distance relative to the zero-frequency center by taking the absolute value. ,in Let be any coordinate axis index in the multidimensional discrete frequency space, and let the corresponding multidimensional coordinate point be denoted as a vector. And satisfy Unnormalized frequency band radial grid in multidimensional space Determined by the Euclidean norm of the frequency amplitude spectrum in all dimensions:

[0049] (1)

[0050] Normalizing the grid by dividing it by the global maximum value yields the mapping interval. Static frequency band radial grid :

[0051] (2)

[0052] in For frequency band radial grid The maximum value. The generated mesh. It is only related to image size and does not participate in backpropagation during training.

[0053] To address the frequency drift phenomenon present in the amplitude spectra of different samples, the center position vector of each frequency band is determined. These parameters are parameterized as learnable variables and iteratively optimized during network training. For the frequency band radial grid... Any coordinate point in the array, the first Gaussian frequency domain response of each frequency band The calculation is as follows:

[0054] (3)

[0055] in This represents a mapping function guided by the target frequency characteristics, used to dynamically adjust the frequency band center; its specific form is as follows: In the formula Let be the learnable center location of the i-th frequency band. This represents the dynamic offset explicitly driven by the target domain frequency band characteristics, i.e., the target modal amplitude spectrum. The frequency band feature is the correction amount generated by the input to the frequency band center, which is used to characterize the drift direction and amplitude of the target domain frequency distribution relative to the initial frequency band center; This represents the natural exponential function. By explicitly incorporating the target domain frequency band features into the calculation of the frequency band center, each frequency band center no longer depends solely on a fixed, learnable initial value, but rather adaptively shifts with the frequency characteristics of the target sample, thereby achieving dynamic guidance of the frequency band center by the target frequency features.

[0056] This is the learnable bandwidth parameter for this frequency band, used to control the smoothness of the Gaussian distribution. The generated frequency domain response... It can adaptively capture the frequency characteristics of the target domain. Finally, by multiplying the frequency domain response by the amplitude spectra of the source and target domains respectively, the decoupled amplitude spectra of each frequency band can be obtained, denoted as sets. , .

[0057] S4, based on the decoupling of amplitude spectra in each frequency band, implements differentiated frequency domain fusion mapping for amplitude spectra in each frequency band: low frequency amplitude spectra are aligned with modal style, mid frequency amplitude spectra are fused through frequency band attention module, and high frequency amplitude spectra suppress noise transmission.

[0058] The MBFGA strategy employs an adaptive frequency band separator to divide the image amplitude spectrum into low-frequency, mid-frequency, and high-frequency amplitude spectra, so as to apply differentiated constraints to different spectral bands.

[0059] Low-frequency amplitude spectrum encoding of the global contrast and intensity distribution of an image. Style transfer is achieved by encoding the low-frequency amplitude spectrum of the target domain. This is achieved by covering the corresponding region of the source domain. The reconstructed low-frequency amplitude spectrum is represented as follows:

[0060] (4)

[0061] in The low-frequency amplitude spectrum represents the spatially aligned image. In the decoupled multi-band system, the low-frequency and high-frequency amplitude spectra constrain macroscopic modal styles and fine edges, respectively, while the mid-frequency amplitude spectrum contains dense cross-modal transition information and complex organizational topology. To achieve adaptive fusion of spectra in heterogeneous systems, this invention further introduces a Frequency Attention Block (FAB), which uses a convolutional neural network to fit a pixel-level adaptive weight matrix online. By accurately measuring the cross-domain spectral mapping, it optimizes energy transitions between modalities and eliminates spectral discontinuities.

[0062] Figure 2 The frequency band attention module is demonstrated to employ a fully convolutional architecture, consisting of four cascaded two-dimensional convolutional layers and corresponding activation functions. Figure 2 The symbols and their annotations are as follows: Conv represents a 2D convolutional layer, BatchNorm represents a batch normalization layer, ReLU represents a modified linear unit activation function, Sigmoid represents a sigmoid activation function, Channel represents the number of channels, © represents channel concatenation, ⊕ represents element-wise addition, and ⊙ represents the Hadamard product. The first three convolutional layers are used to extract cross-modal correlation features of the mid-frequency amplitude spectrum between the source and target domains. The last convolutional layer maps the high-dimensional features to a frequency attention mask with the same size as the input amplitude spectrum. This is used to determine the dynamic mixing ratio of the mid-frequency amplitude spectra in the source and target domains. The adaptive synthesis mechanism in the mid-frequency domain is as follows:

[0063] (5)

[0064] in, This represents the original mid-frequency amplitude spectrum of the source domain. This represents the original mid-frequency amplitude spectrum of the target domain. This represents the mid-frequency amplitude spectrum of a spatially aligned image, where 1 represents a uniformly one matrix of the same dimension. It represents the Hadamardi (or Hadama) stack.

[0065] Since the high-frequency amplitude spectrum contains local edges and imaging noise, and the noise distribution of different modal devices is heterogeneous, in order to suppress the propagation of high-frequency random noise from the target domain to the generated source domain image, a low-weighting coefficient is used in the high-frequency region. The low-weight coefficients are constrained by a linear combination. By limiting the gain of the high-frequency components in the target domain to Internally, it can effectively block the introduction of target domain noise and imaging artifacts, while utilizing high weighting coefficients. The inherent anatomical edge details of the source domain are preserved to the greatest extent possible. The high-frequency domain synthesis calculation is as follows:

[0066] (6)

[0067] in Represents the high-frequency amplitude spectrum of the spatially aligned image. , These are the original high-frequency amplitude spectra of the source and target domains, respectively. These are pre-defined constant hyperparameters.

[0068] S5 aggregates the reconstructed low-frequency amplitude spectrum, mid-frequency amplitude spectrum, and high-frequency amplitude spectrum into a synthetic amplitude spectrum, and performs a two-dimensional fast inverse Fourier transform with the source modal phase spectrum to generate a spatial domain aligned image that retains spatial semantic labels.

[0069] The reconstructed low-frequency, mid-frequency, and high-frequency amplitude spectra are linearly combined to reconstruct the full-band synthetic amplitude spectrum. :

[0070] (7)

[0071] Furthermore, the synthesized amplitude spectrum Phase spectrum with source region By combining these methods, the frequency domain features are transformed to the spatial domain using a two-dimensional fast inverse Fourier transform, thereby generating an aligned image. The generated spatially aligned image It is then used as input to labeled and unlabeled data streams for network training to drive dynamic updates of the target domain feature distribution.

[0072] S6 constructs a Gram matrix statistical loss based on spatial domain aligned images and target modal images, and combines it with the segmentation model loss to jointly optimize the learnable parameters of the segmentation network and frequency domain mapping.

[0073] To optimize the band boundary decisions in the adaptive band splitter and guide the band attention module to effectively fuse the intermediate frequency amplitude signal, the spatial domain aligns the image. The target domain image needs to be further approximated in the feature space. A pre-trained VGG network with fixed parameters is introduced as a feature extractor.

[0074] Let the image Shallow features are extracted using a VGG network, and second-order statistical features are captured by calculating the Gram matrix of the feature map. Gram matrix of layers calculate:

[0075] (8)

[0076] in This indicates the VGG network's... Layer feature mapping, Indicates the input image, superscript The transpose sign is used. Based on this, the algorithm achieves explicit alignment of feature distributions by minimizing the mean square error between the two Gram matrices. The statistical loss function is:

[0077] (9)

[0078] in The shallow features of the VGG network, which represents the Frobenius norm, primarily encode fine-grained representations such as local texture and details, effectively avoiding redundant high-dimensional semantic interference introduced by deep networks.

[0079] Based on the statistical loss of the Gram matrix mentioned above, the overall loss function of the system is further constructed, and the learnable parameters of the segmentation network and the frequency domain mapping are jointly optimized.

[0080] Let the weight parameter set of the segmentation network be... All learnable parameter sets in the adaptive band splitter and band attention module are collectively referred to as The overall loss function of the system Defined as:

[0081] (10)

[0082] in The segmentation model has a partial loss due to the labeled data. To split the semi-supervised part of the model loss, both losses are weighted by cross-entropy loss and Dice loss. For the source domain space real label, The constant hyperparameter for balancing the alignment intensity of the spatial feature distribution is gradually increased with the number of iterations based on Gaussian heating: , This represents the current iteration number. is the maximum number of iterations, and e is the natural constant.

[0083] This invention employs an end-to-end single-stage training mode. The system calculates the partial derivatives of the joint loss with respect to two independent parameter sets and uses the backpropagation algorithm to simultaneously update the gradients of both sets within the same physical control flow. The update mechanism is as follows:

[0084] (11)

[0085] (12)

[0086] in Let be the system's learning rate. The parameter set can be learned through co-optimization using bidirectional gradient flow. Each optimization will lead to an improvement in the segmentation performance of the main network, and the change in the feature extraction topology within the segmentation network will also have a real-time feedback effect on the frequency division boundary in the frequency domain.

[0087] Due to the weight parameter set of the main segmentation network With learnable parameter set Significant heterogeneity exists between the data manifold and the alignment target in terms of magnitude. The system introduces a time-varying hyperparameter scheduling mechanism to address this. Dynamic adjustment is implemented, with the learning rate adaptively adjusted using a polynomial decay strategy as the iteration progresses. The calculation is as follows:

[0088] (13)

[0089] in The preset initial learning rate, This refers to the hyperparameters used to control the decay rate.

[0090] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0091] 1. Dataset

[0092] The MMWHS dataset consists of 20 unpaired CT and 20 MRI datasets from different cardiac sites. This invention selects four cardiac structures for segmentation: the ascending aorta (AA), left atrial chamber (LAC), left ventricular chamber (LVC), and left ventricular myocardium (MYO). This invention performs adaptive tasks in two modal directions: from CT to MRI and from MRI to CT, with 80% used for training and 20% for testing.

[0093] The AMOS dataset is a dataset containing 300 CT images and 60 MRI images across multiple centers, modalities, and temporal phases. Each label includes 15 abdominal organs, and the CT-MRI data in the dataset are not paired. The training set consists of 200 CT images and 40 MRI images, while the test set consists of 100 CT images and 20 MRI images. Since only modal differences are considered, this invention selects 33 CT images and 33 MRI images (66 images total) from the training set and 20 CT images and MRI images (40 images total) from the test set. Data of the same modality are all from the same brand and model of instrument to ensure that semantic features within the same modality are not biased due to differences in instrument scanning.

[0094] 2. Experimental Environment

[0095] The segmentation network used in the experiment adopted a U-Net network as the backbone network, where U-Net represents a U-shaped segmentation network. The hardware configuration and software selection used in all subsequent experiments are shown in Table 1.

[0096] Table 1 Experimental Environment

[0097]

[0098] 3. Experimental parameters

[0099] To ensure the fairness of the comparative experiments and to adhere to general experimental benchmarks in the field of image segmentation, all experiments in this invention use uniform training parameter settings, including the maximum number of iterations. Optimizer SGD, initial learning rate Hyperparameters controlling the decay rate optimizer decay weights All models were finally inferred and evaluated on the target modality test set. Detailed experimental setup is shown in Table 2.

[0100] Table 2 Experimental Parameter Settings

[0101]

[0102] 4. Evaluation Indicators

[0103] To quantitatively validate the segmentation performance of the MBFGA strategy, the experiment introduced two metric systems with complementary constraint effects: the region overlap index and the boundary spatial distance index. The region overlap index, encompassing the Dice coefficient and the intersection-over-union ratio (IoU), assesses the spatial volume overlap between the predicted segmented anatomical topology and the ground truth label at the voxel manifold level. The boundary spatial distance index, including the 95% Hausdorff distance (HD95) and the average symmetric surface distance (ASD), refines the spatial deviation between the predicted contour and the ground truth boundary at the geometric level, capturing the alignment effect of the anatomical edge morphology.

[0104] 5. Data Preprocessing

[0105] The MMWHS (Multi-Modality Whole Heart Segmentation) and AMOS (Abdominal Multi-Organ Segmentation) datasets are 3D datasets. Due to the low efficiency of 3D Fourier transform and device performance bottlenecks, model training consumes a significant amount of time. Therefore, we pre-slice the two datasets using their Z-axis, dividing the 3D image data into 2D image data. The number of slices for each 3D dataset is determined by the Z-axis. The MMWHS dataset uses preprocessed data released by SIFA V2. For the AMOS dataset, since the slice sizes differ between modalities, the 3D image slices need to be standardized to 512×512 pixels. To avoid structural distortion caused by image stretching, the long side of the image is scaled to 512 pixels and placed in the center, with the remaining portion filled with black pixels. The label slicing logic is consistent with the image data. Specific dataset slicing information is shown in Table 3.

[0106] Table 3 Dataset Slicing Information

[0107]

[0108] 6. Performance Comparison

[0109] This invention integrates the MBFGA strategy into the training process of some classic semi-supervised training networks, including four common training networks: mean teacher, FixMatch, CCT (Cross-Consistency Training), and CPS. The results are compared with the networks themselves that are not integrated to verify the effectiveness of this strategy.

[0110] 6.1 MMWHS Dataset

[0111] This invention extensively evaluates the MBFGA strategy on the MMWHS dataset to verify its effectiveness and universality under different baseline models and annotation ratios. Specific parameters and settings are shown in Table 2.

[0112] Tables 4 and 5 show the performance comparison of integrating the MBFGA strategy during the migration from the CT modality to the MRI modality. In the experimental setup, the MBFGA strategy was integrated into the training process of the classic base model, while the control group did not include it.

[0113] Table 4 shows the experimental results in scenarios with 10% labeled data and 90% unlabeled data. As can be seen from the table, with the exception of one method, integrating the MBFGA strategy improved the segmentation performance metrics of the models to varying degrees. Furthermore, comparison with fully supervised methods reveals that some baseline models incorporating the MBFGA strategy have performance that approaches or even surpasses the baseline performance of fully supervised training. Table 5 further compares the performance differences between the model and classic baseline models under the low-resource condition of only 5% labeled data. The results show that even with the challenge of extremely scarce labeled data, this invention still delivers significant performance gains; the improvement achieved by some baseline models with 5% labeled data even exceeds their performance with 10% labeled data.

[0114] Visualization of the MMWHS dataset from CT to MRI modalities, for example Figure 4 and Figure 5 As shown, compared to the true ground truth (GT), the model trained with the MBFGA policy achieves more accurate prediction results. It can also be observed that integrating the MBFGA policy reduces the number of class prediction errors.

[0115] Table 4. CT to MRI cross-modal performance on the MMWHS dataset (10% annotation)

[0116]

[0117] Table 5. CT to MRI cross-modal performance on the MMWHS dataset (5% annotation)

[0118]

[0119] Tables 6 and 7 show the performance comparison of the MBFGA strategy in feature alignment from MRI to CT modality. Continuing with the settings in Tables 4 and 5, the experimental group integrated the MBFGA strategy into the classic baseline model. The data show that the models achieved significant performance improvements at different labeling ratios, and this improvement remained effective under fully supervised baseline conditions. Combined with the experimental results from the CT to MRI direction mentioned above, this demonstrates that the MBFGA strategy exhibits consistent effectiveness across two opposite modality transfer paths.

[0120] Figure 6 and Figure 7 The visualizations show the comparison between MRI generalization to CT modality under 10% and 5% labeling. The model trained with the integrated MBFGA strategy can make more accurate and complete predictions than the model without integration, and can also avoid prediction errors or missing predictions caused by differences in features of different categories.

[0121] Table 6. MRI to CT cross-modal performance on the MMWHS dataset (10% annotation)

[0122]

[0123] Table 7. MRI to CT cross-modal performance on the MMWHS dataset (5% annotation)

[0124]

[0125] Experimental data and visualizations on the MMWHS dataset for two generalization directions show that the segmentation model with the MBFGA strategy has a relatively stable performance improvement and produces more accurate predictions, reducing the segmentation errors caused by feature differences. It also produces positive performance improvements at different annotation ratios, which proves the effectiveness of the MBFGA strategy in semi-supervised medical image segmentation tasks.

[0126] 6.2 AMOS Dataset

[0127] This invention also evaluated the MBFGA strategy on the AMOS dataset. All experiments were conducted with the same parameter settings, and the specific parameter items and settings are shown in Table 2.

[0128] Tables 8 and 9 show the performance comparison after integrating the MBFGA strategy during the transfer from the CT modality to the MRI modality. Consistent with the experiments on the MMWHS dataset, the MBFGA strategy was integrated into the training process of the classic base model, while the control group did not integrate it. As shown in Table 8, with 10% labeled data, all baseline methods, except one, achieved varying degrees of improvement after integration. Specifically, the Dice coefficient of the CCT method improved by 3.04%, and the Dice coefficient of the MeanTeacher method also increased by nearly 3%. Furthermore, a comparison of fully supervised methods was also included in the AMOS dataset. The results show that the MBFGA strategy still has a positive impact on the results, and compared with the semi-supervised methods integrating this strategy, its performance is close to that of fully supervised methods. Table 9 shows the performance comparison with only 5% labeled data, and the results show a significant improvement even in this case.

[0129] Figure 8 The visualization compares the generalization from CT modality to MRI modality on the AMOS dataset. Figure 9This section presents a visual comparison of generalization from MRI modality to CT modality on the AMOS dataset. It can be seen that, compared to GT labels, the segmentation model trained using the MBFGA strategy achieves more accurate prediction results. Furthermore, compared to methods not introduced, it can predict as many organs as possible present in the current slice in multi-organ slices, with relatively accurate location and size.

[0130] Table 8. CT to MRI cross-modal performance on the AMOS dataset (10% annotation)

[0131]

[0132] Table 9. CT to MRI cross-modal performance on the AMOS dataset (5% annotation)

[0133]

[0134] To further verify the universality of the MBFGA strategy in the transfer direction, Tables 10 and 11 present the experimental results of cross-modal transfer from MRI to CT on the AMOS dataset. As shown in Table 10, with 10% labeled data, all baseline models improved their performance metrics after integration, except for one. Consistent with previous conclusions, the MBFGA strategy maintains a positive gain effect under fully supervised benchmarks. The results in Table 11 further demonstrate that when the labeled data is reduced to a smaller 5% labeled scenario, the segmentation model integrating the MBFGA strategy does not experience performance degradation and still provides stable performance improvements.

[0135] Figure 10 and Figure 11 The visualization compares the generalization from MRI modality to CT modality on the AMOS dataset. It shows that after integrating the MBFGA strategy, the prediction of background as organs or the misprediction of organ categories is improved, and the predicted organ location and size are more accurate than in the experiment without integration.

[0136] Table 10. MRI to CT cross-modal performance on the AMOS dataset (10% annotation)

[0137]

[0138] Table 11. MRI to CT cross-modal performance on the AMOS dataset (5% annotation)

[0139]

[0140] Experimental results on the AMOS dataset for both generalization directions also show that the integration of the MBFGA strategy has a positive improvement on the segmentation performance of the final model, and this remains true even at different scales. This indicates that the strategy proposed in this invention can positively improve the final segmentation performance of the segmentation network on different datasets and in different generalization directions, and has good general applicability.

[0141] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. An unsupervised cross-modal CT-MRI image segmentation method, characterized in that, Includes the following steps: S1, acquire source and target domain medical image datasets; the source domain includes labeled samples and unlabeled samples, and the target domain includes unlabeled samples, which are different modalities in CT and MRI, respectively; S2, perform two-dimensional fast Fourier transform on the source mode image and the target mode image respectively, decouple them to obtain their respective amplitude spectrum and phase spectrum, and retain the phase spectrum of the source mode; S3 uses an adaptive frequency band separator to separate the source mode amplitude spectrum and the target mode amplitude spectrum into low-frequency amplitude spectrum, mid-frequency amplitude spectrum and high-frequency amplitude spectrum, respectively; S4, implement differentiated frequency domain fusion mapping for amplitude spectra of each frequency band: low frequency amplitude spectra are aligned with modal style, mid frequency amplitude spectra are fused through frequency band attention module, and high frequency amplitude spectra suppress noise transmission; S5 aggregates the reconstructed low-frequency amplitude spectrum, mid-frequency amplitude spectrum, and high-frequency amplitude spectrum into a synthetic amplitude spectrum, and performs a two-dimensional fast inverse Fourier transform with the source modal phase spectrum to generate a spatial domain aligned image that retains spatial semantic labels. S6 constructs a Gram matrix statistical loss based on spatial domain aligned images and target modal images, and combines it with the segmentation model loss to jointly optimize the learnable parameters of the segmentation network and frequency domain mapping.

2. The method according to claim 1, characterized in that, In S1, the source domain medical image is the source modality image, and the target domain medical image is the target modality image; the source modality image is a CT image, and the target modality image is an MRI image; the labeled and unlabeled samples in the source domain undergo isomorphic processing during the frequency domain mapping stage, and the spatial semantic labels corresponding to the labeled samples are retained after frequency domain mapping.

3. The method according to claim 1, characterized in that, The adaptive frequency band separator in S3 employs a soft mask filter based on Gaussian bandpass filtering to generate smooth frequency edges. The adaptive frequency band separator generates symmetrical discrete frequency coordinates based on the spatial dimension of the input image and constructs a normalized frequency band radial grid based on the distance of each discrete frequency coordinate relative to the zero-frequency center. The center position and bandwidth of each frequency band are learnable parameters and are dynamically adjusted according to the frequency characteristics of the target modal image to separate the source modal amplitude spectrum and the target modal amplitude spectrum into low-frequency amplitude spectrum, mid-frequency amplitude spectrum, and high-frequency amplitude spectrum, respectively.

4. The method according to claim 1, characterized in that, The low-frequency amplitude spectrum mentioned in S4 is used to characterize the global contrast and intensity distribution of the image; by covering the target modality low-frequency amplitude spectrum with the corresponding region of the source modality low-frequency amplitude spectrum, modal style alignment from the source modality image to the target modality image is achieved; the source modality high-frequency amplitude spectrum and the target modality high-frequency amplitude spectrum are weighted and constrained, and the introduction ratio of the target modality high-frequency amplitude spectrum is limited, so as to suppress the transmission of target modality noise and imaging artifacts while preserving the anatomical edge details of the source modality.

5. The method according to claim 1, characterized in that, The frequency band attention module in S4 receives the source mode intermediate frequency amplitude spectrum and the target mode intermediate frequency amplitude spectrum, extracts the cross-modal correlation features between the source mode intermediate frequency amplitude spectrum and the target mode intermediate frequency amplitude spectrum, and generates a frequency attention mask; The fusion ratio of the source mode intermediate frequency amplitude spectrum and the target mode intermediate frequency amplitude spectrum is dynamically adjusted according to the frequency attention mask to obtain the fused intermediate frequency amplitude spectrum.

6. The method according to claim 4, characterized in that, The frequency band attention module described in S4 further includes: a fully convolutional architecture, comprising four cascaded two-dimensional convolutional layers and corresponding activation functions; the first three two-dimensional convolutional layers are used to extract cross-modal correlation features between the source modal mid-frequency amplitude spectrum and the target modal mid-frequency amplitude spectrum, and the last two-dimensional convolutional layer is used to map the cross-modal correlation features into a frequency attention mask with the same size as the input mid-frequency amplitude spectrum.

7. The method according to claim 1, characterized in that, The reconstructed low-frequency amplitude spectrum, mid-frequency amplitude spectrum, and high-frequency amplitude spectrum described in S5 are linearly combined to obtain a complete synthetic amplitude spectrum. The synthetic amplitude spectrum is then combined with the source mode phase spectrum, which is preserved as a spatial invariant, and transformed to the spatial domain through a two-dimensional fast inverse Fourier transform to generate a spatial domain aligned image. The spatial topology of the spatial domain aligned image remains unchanged relative to the source mode image, and the spatial semantic labels are fully preserved.

8. The method according to claim 1, characterized in that, The joint optimization described in S6 includes: using a pre-trained feature extraction network with fixed parameters to extract shallow features from the spatial domain aligned image and the target modality image respectively, and constructing the Gram matrix statistical loss based on the differences in second-order statistical features represented by the Gram matrix of the shallow features; combining the Gram matrix statistical loss with the segmentation model loss into an overall loss, and jointly updating the weight parameters of the segmentation network and the learnable parameters of the adaptive frequency band separator and the frequency band attention module during the end-to-end single-stage training process.

9. A computer system, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 8 when executing the executable instructions.

Citation Information

Patent Citations

  • Medical image segmentation method based on semi-supervised domain self-adaption and cross-domain collaborative learning

    CN119785033A

  • Dual-stage patch-level frequency domain consistency semi-supervised medical image segmentation method

    CN121505258A