Cross-modality brain image synthesis method and system based on structural feature distribution constraint

Abstract

The present application relates to the technical field of medical image generation, and discloses a cross-modal brain image synthesis method and system based on structure feature distribution constraint; in the training stage of the method, structure features of a target domain synthetic brain image and a real target domain brain image are extracted respectively, mapped to a shared space grid, a generated structure feature distribution and a target structure feature distribution are constructed, and a structure feature distribution distance between the two distributions is calculated; and a position weight is constructed according to the local change intensity of the target structure feature map, the weight is applied to the two groups of distributions respectively and is normalized again, and a structure weighted feature distribution distance is calculated; the two distances are combined into a structure feature distribution constraint loss, and are jointly optimized with a generated network original training target; in the synthesis stage, a source domain brain image to be synthesized is input into the generated network after training, and a target domain synthetic brain image is output.

Description

Technical Field

[0001] This invention relates to the field of medical image generation technology, specifically to a method and system for cross-modal brain image synthesis based on structural feature distribution constraints. Background Technology

[0002] Imaging assessment of brain diseases typically requires the integration of multiple medical imaging information. Different modalities can reflect lesion morphology, tissue changes, and anatomical structures from different perspectives; therefore, multimodal information helps improve the reliability of disease analysis and clinical diagnosis. However, in real clinical settings, some modalities may be missing due to factors such as acquisition costs, equipment conditions, or patient status. Cross-modal medical image synthesis, by utilizing acquired modalities to generate missing target modalities, has become an important research direction for alleviating the problem of modality loss. The development of deep generative models has driven research on cross-modal medical image synthesis. Generative adversarial networks and diffusion models have been widely used in tasks such as CT-to-MRI, MR-to-CT, and MRI cross-sequence synthesis. In these applications, the generated results not only need to have a reasonable visual appearance but also need to retain information relevant to subsequent analysis, such as lesion neighborhood, tissue boundaries, and local structural patterns.

[0003] Most existing supervised learning objectives are based on correspondence, guiding model updates by constraining differences between corresponding locations, regions, or features. Pixel-by-pixel reconstruction loss directly compares intensity differences at the same spatial location, while perceptual loss compares differences between corresponding features in a deep feature space. Although these objectives function at different levels, they typically guide the model to learn target modal information by reducing local correspondence differences.

[0004] However, structural information cannot always be fully described by feature differences at a single location. Lesion neighborhoods, tissue boundaries, and local texture patterns in brain imaging are often formed by multiple adjacent or related locations. What is important is the distribution, clustering, and relative arrangement of these structural features in the image space, rather than simply whether structural features at a particular location are consistent.

[0005] Existing research has introduced segmentation masks, anatomical continuity constraints, and anatomically-aware supervision strategies to enhance the role of structural information in the training process. Other methods utilize edge losses, long- and short-range dependency modeling, or frequency information to strengthen boundary and local texture representations. However, these constraints primarily provide structural guidance at the region-wide or contour level, and their constraints on texture variations and detailed features within regions remain limited. Summary of the Invention

[0006] This invention aims to provide a method and system for cross-modal brain image synthesis based on structural feature distribution constraints. The method maps structural features from both the generation and target ends to the same spatial grid, using distribution distance to characterize the overall spatial relationship of structural features. Simultaneously, it constructs positional weights based on the intensity of local structural changes in the real target domain brain image, applying these weights to both the generated and target structural feature distributions and renormalizing them. This ensures that locations with significant structural changes contribute more to the distribution comparison. Thus, without limiting the specific generation backbone, it provides supervision for cross-modal brain image synthesis that combines overall distribution with key local structures.

[0007] The technical solution of this invention is as follows: A cross-modal brain image synthesis method based on structural feature distribution constraints includes the following operations: S1. Obtain source domain brain images and real target domain brain images. Input the source domain brain images into the generator network to obtain the target domain synthetic brain images. Extract the structural features of the target domain synthetic brain images and the real target domain brain images respectively to obtain the generated structural feature map and the target structural feature map. S2. Map the generated structural feature map and the target structural feature map to a shared space grid to construct a unified representation of the structural feature distribution, obtaining the generated structural feature distribution and the target structural feature distribution. On the shared space grid, calculate the structural feature distribution distance between the generated structural feature distribution and the target structural feature distribution. Construct position weights based on the intensity of local structural changes in the target structural feature map, and apply these position weights to the generated structural feature distribution and the target structural feature distribution respectively. After normalization, obtain the structure-weighted generated distribution and the structure-weighted target distribution. Calculate the structure-weighted feature distribution distance based on the structure-weighted generated distribution and the structure-weighted target distribution. Combine the structural feature distribution distance and the structure-weighted feature distribution distance in a weighted manner to obtain the structural feature distribution constraint loss. S3. The structural feature distribution constraint loss is used as an additional supervision signal and combined with the original training objective of the generator network to form a joint training objective. The generator network is a generative adversarial network or a diffusion model. During the training phase, the generator or denoising network parameters are updated according to the generator network until the preset number of training rounds is reached or the preset stopping condition is met. After training is completed, the source domain brain image to be synthesized is input into the trained generator network, and the target domain synthesized brain image is output.

[0008] A cross-modal brain image synthesis system based on structural feature distribution constraints, used to implement the aforementioned cross-modal brain image synthesis method based on structural feature distribution constraints, includes: The brain image acquisition and synthesis module is used to acquire source domain brain images and real target domain brain images, input the source domain brain images into the generation network, and obtain the target domain synthesized brain images. The structural feature extraction module is used to extract structural features from the synthetic brain image of the target domain and the real brain image of the target domain, and obtain the generated structural feature map and the target structural feature map. The structural feature distribution constraint loss generation module maps the generated and target structural feature maps to a shared space grid, constructs a unified structural feature distribution representation, and obtains the generated and target structural feature distributions. On the shared space grid, it calculates the structural feature distribution distance between the generated and target structural feature distributions. It constructs position weights based on the intensity of local structural changes in the target structural feature map, applies these position weights to both the generated and target structural feature distributions, and normalizes them to obtain a structure-weighted generated distribution and a structure-weighted target distribution. Based on these two distributions, it calculates the structure-weighted feature distribution distance. Finally, it weights and combines the structural feature distribution distances to obtain the structural feature distribution constraint loss. The model training and target domain synthetic brain image generation module is used to combine the structural feature distribution constraint loss as an additional supervision signal with the original training objective of the generator network to form a joint training objective; the generator network is a generative adversarial network or a diffusion model; during the training phase, the generator or denoising network parameters are updated according to the generator network until a preset number of training rounds is reached or a preset stopping condition is met; after training is completed, the source domain brain image to be synthesized is input into the trained generator network, and the target domain synthetic brain image is output.

[0009] A cross-modal brain image synthesis device based on structural feature distribution constraints includes a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the above-mentioned cross-modal brain image synthesis method based on structural feature distribution constraints.

[0010] A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the above-described method for cross-modal brain image synthesis based on structural feature distribution constraints.

[0011] The beneficial effects of this invention are as follows: The present invention provides a cross-modal brain image synthesis method based on structural feature distribution constraints. First, it extracts structural features from both the synthesized brain image and the real target domain brain image, mapping them to a shared spatial grid to construct a unified structural feature distribution representation. This extends structural supervision from traditional pixel-by-pixel or local correspondence comparisons to a comparison of the overall distribution of structural features in the image domain. This addresses the shortcomings of existing pixel-by-pixel loss methods in characterizing the spatial organization of structural features and the insufficient supervision of internal texture details by region-level constraints such as segmentation masks. Then, it calculates the structural feature distribution distance on the shared spatial grid and constructs positional weights based on the local variation intensity of the target structural feature map. These positional weights are applied to both the generated structural feature distribution and the target structural feature map. The feature distribution is re-normalized to ensure that locations with significant structural changes contribute more in the distribution comparison, resulting in a structure-weighted generative distribution and a structure-weighted target distribution. Based on the two sets of weighted distributions, the distance between the structure-weighted feature distributions is calculated, and a structural feature distribution constraint loss is constructed, which helps improve boundary continuity and local detail recovery. Then, the structural feature distribution constraint loss is used as an additional supervision signal and combined with the original training objective of the generative adversarial network or diffusion model to form a joint training objective. During the training phase, the generator or denoising network parameters are updated according to the generative network until a preset number of training rounds is reached or a preset stopping condition is met. After training, the source domain brain image to be synthesized is input into the trained generative network, and the target domain synthesized brain image is output.

[0012] The present invention provides a cross-modal brain image synthesis method based on structural feature distribution constraints. This method constructs structural feature distribution constraints for structural supervision in cross-modal brain image synthesis. Unlike traditional structural supervision, which mainly relies on local correspondences, structural feature distribution constraints describe structural supervision as a comparison problem between structural feature distributions. By constructing structural feature distributions on a shared spatial grid and establishing supervision signals using structural feature distribution distances and weighted structural feature distribution distances, the structural feature distribution constraints constrain the spatial distribution relationships of structural feature maps at the distribution level. This method is decoupled from the generator structure and can be integrated as an independent structural constraint into different image generation frameworks.

[0013] This invention provides a cross-modal brain image synthesis method based on structural feature distribution constraints. On four datasets (ISD, YHD, ISLES, and RAD), pairwise comparisons of five GAN backbones, loss component ablation, comparisons of structural feature extraction methods, diffusion framework validation, external center validation, bidirectional cross-center evaluation, CT to T1-weighted MRI synthesis, random partitioning and patient-level robustness analysis, and ISLES fixed segmenter prediction consistency analysis were conducted. Experimental results show that structural feature distribution constraints can be used as additional structural supervision to integrate different generated backbones, improving the quality of medical image synthesis. Attached Figure Description

[0014] The solutions and advantages of this application will become clear to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0015] In the attached diagram: Figure 1 In this example, the representative synthesis results of five GAN backbones on the ISD dataset before and after adding SFDC are shown. Figure 2 In this example, the loss component represents the results of difficult samples during ablation; Figure 3 In the examples, the BBDM and Fast-DDPM diffusion frameworks show representative results before and after the addition of SFDC; Figure 4 In the example, the test slice-level index distributions of the BBDM and Fast-DDPM diffusion frameworks before and after the addition of SFDC; Figure 5 In this example, the five GAN backbones on the YHD dataset are representative results before and after being added to SFDC; Figure 6 In this example, the distribution of patient-level metrics before and after adding SFDC to the five GAN backbones on the YHD dataset; Figure 7 In this example, the five GAN backbones on the RAD dataset are representative results before and after being added to SFDC; Figure 8 In this example, the distribution of patient-level metrics before and after adding SFDC to the five GAN backbones on the RAD dataset is shown. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the exemplary embodiments of this application clearer, the technical solutions in the exemplary embodiments of this application are described clearly and completely below. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0017] This embodiment provides a cross-modal brain image synthesis method based on structural feature distribution constraints. It uses paired source domain brain images and real target domain brain images as training data. The overall process includes structural feature extraction, shared spatial distribution construction, structural feature distribution constraint calculation, and joint training of the generative model. Specifically, it includes the following steps: S1. Obtain source domain brain images and real target domain brain images. Input the source domain brain images into the generator network to obtain the target domain synthetic brain images. Extract the structural features of the target domain synthetic brain images and the real target domain brain images respectively to obtain the generated structural feature map and the target structural feature map. S2. Map the generated structural feature map and the target structural feature map to a shared space grid to construct a unified representation of the structural feature distribution, obtaining the generated structural feature distribution and the target structural feature distribution. On the shared space grid, calculate the structural feature distribution distance between the generated structural feature distribution and the target structural feature distribution. Construct position weights based on the intensity of local structural changes in the target structural feature map, and apply these position weights to the generated structural feature distribution and the target structural feature distribution respectively. After normalization, obtain the structure-weighted generated distribution and the structure-weighted target distribution. Calculate the structure-weighted feature distribution distance based on the structure-weighted generated distribution and the structure-weighted target distribution. Combine the structural feature distribution distance and the structure-weighted feature distribution distance in a weighted manner to obtain the structural feature distribution constraint loss. S3. The structural feature distribution constraint loss is used as an additional supervision signal and combined with the original training objective of the generator network to form a joint training objective. The generator network is a generative adversarial network or a diffusion model. During the training phase, the generator or denoising network parameters are updated according to the generator network until the preset number of training rounds is reached or the preset stopping condition is met. After training is completed, the source domain brain image to be synthesized is input into the trained generator network, and the target domain synthesized brain image is output. The specific steps are detailed below.

[0018] S1. Obtain source domain brain images and real target domain brain images. Input the source domain brain images into the generator network to obtain the target domain synthetic brain images. Extract the structural features of the target domain synthetic brain images and the real target domain brain images respectively to obtain the generated structural feature map and the target structural feature map.

[0019] To obtain a unified representation for subsequent structural comparisons, structural features need to be extracted from the input image. The details of the steps for extracting structural features are as follows.

[0020] Step 1: Calculate the gradient response of the brain image (synthesized brain image of the target domain or real brain image of the target domain) in the horizontal and vertical directions respectively. The calculation formula is as follows: , , These represent the gradient responses in the horizontal and vertical directions, respectively. , These are Sobel convolution kernels in the horizontal and vertical directions, respectively. This represents a two-dimensional convolution operation. This is a brain imaging image.

[0021] Brain imaging, namely target domain synthetic brain images and real target domain brain images, includes any two modal combinations of CT images, DWI images and T1-weighted MRI images.

[0022] Step 2: Based on the gradient responses in the horizontal and vertical directions, construct the gradient magnitude using the following formula: , For gradient magnitude, It is a numerically stable term.

[0023] Step 3: To mitigate the impact of overall gradient magnitude differences between different images, first calculate the average gradient magnitude at all spatial locations in the gradient magnitude map, denoted as... Then, adaptive mean normalization is performed according to the following formula: , for The adaptive normalized gradient magnitude at the location, for gradient magnitude at that point Indicates the first A spatial location, This represents the average gradient magnitude across all spatial locations in the image. For numerically stable terms, after normalization, the structural feature map is obtained through a thresholded Sigmoid mapping, and the calculation formula is as follows: , The slope of the sigmoid function. To control the activation threshold, This is a structural feature map; the higher the value, the more obvious the local structural changes at the corresponding location.

[0024] The preferred generative networks are generative adversarial networks or diffusion models.

[0025] S2. Map the generated structural feature map and the target structural feature map to a shared space grid to construct a unified representation of the structural feature distribution, obtaining the generated structural feature distribution and the target structural feature distribution; calculate the structural feature distribution distance between the two distributions on the shared space grid; construct position weights based on the intensity of local structural changes in the target structural feature map, apply the position weights to the generated structural feature distribution and the target structural feature distribution respectively, and normalize them to obtain the structural weighted generated distribution and the structural weighted target distribution; calculate the structural weighted feature distribution distance based on the structural weighted generated distribution and the structural weighted target distribution; obtain the structural feature distribution constraint loss based on the structural feature distribution distance and the structural weighted feature distance.

[0026] First, in order to calculate the differences between the structural feature distributions, the generated structural feature map and the target structural feature map are mapped to a shared space grid to construct a unified representation of the structural feature distribution, thus obtaining the generated structural feature distribution and the target structural feature distribution.

[0027] The operation to construct a unified structural feature distribution representation is as follows: scaling the generated structural feature map and the target structural feature map to an auxiliary resolution. H × W , H, WRepresenting height and width respectively, a discrete space is constructed on the auxiliary mesh. , For the first i Each grid location N=H×W ;exist At this point, the feature values ​​of the generated structural feature map and the target structural feature map are denoted as follows: , , , The generated structural feature map and the target structural feature map are located at the grid positions, respectively. eigenvalues ​​at that location , The generated structural feature map and the target structural feature map are respectively. The generated structural feature map and the target structural feature map are normalized into discrete structural feature distributions, respectively, to obtain the generated structural feature distribution and the target structural feature distribution. Thus, the structural feature map is transformed from a local feature value representation to a distribution representation in a unified space, providing input for subsequent structural feature distribution constraint calculation.

[0028] Then, on the shared space grid, the structural feature distribution distance between the generated structural feature distribution and the target structural feature distribution is calculated to compare the differences between the generated structural feature distribution and the target structural feature distribution on the shared space grid.

[0029] The structural feature distribution distance is calculated using the following formula: , , Distance of structural feature distribution , These represent the generated structural feature distribution and the target structural feature distribution at the grid positions, respectively. The distribution weights at that location, , These represent the generated structural feature distribution and the target structural feature distribution at the grid positions, respectively. The distribution weights at that location, For grid position and The normalized Euclidean distance between them It is a two-dimensional Euclidean distance. The diagonal length of the auxiliary grid is used to normalize the distance to a uniform scale to ensure that the distances are comparable under different auxiliary resolutions. This normalization only scales the Euclidean distance proportionally and does not change the relative distance relationship between spatial locations. Simultaneously considering the spatial differences between the two sets of distributions and the degree of spatial dispersion within each distribution, this method is used to measure the differences in the distribution of structural features on a shared spatial grid.

[0030] Generate structural feature distribution, target structural feature distribution at grid location Distribution weights at the location , It is calculated using the following formula: , , , The generated structural feature map and the target structural feature map are located at the grid positions, respectively. The eigenvalue at that location.

[0031] Next, positional weights are constructed based on the intensity of local structural changes in the target structural feature map. These positional weights are then applied to both the generated structural feature distribution and the target structural feature distribution, and normalized to obtain a structure-weighted generated distribution and a structure-weighted target distribution. This step increases the relative weight of locations with significant structural changes in subsequent distribution comparisons. The specific process is as follows.

[0032] Step 1: Obtain the normalized local structural change intensity of the target structural feature map. The calculation formula is as follows: , For grid position The intensity of structural feature changes after mean adaptive normalization. Represents the Laplacian operator. For the target structure feature map, The absolute Laplacian response of the target structure feature map in discrete space The average value across all positions. The target structure feature map is used to provide a relatively stable source of positional weights, avoiding additional fluctuations in weights as the generated results change during the training phase.

[0033] Step 2: Construct position weights based on the intensity of structural feature changes. The calculation formula is: , To control the weighting intensity, constant term 1 retains the basic contribution of each location in the shared space grid to both distributions. Additional weighting is provided based on the intensity of changes in local structural features. When At that time, the original weight is retained at that position, and as... As the value increases, the contribution of that position also increases accordingly.

[0034] Step 3: Apply the positional weights to the generated structural feature distribution and the target structural feature distribution respectively, and renormalize the two sets of weighted results respectively: , , These represent the structure-weighted generation distribution and the structure-weighted target at the grid location, respectively. The distribution weights, , These represent the generated structural feature distribution and the target structural feature distribution at the grid positions, respectively. The distribution weights, , Grid positions , The position weights are calculated from the target structure feature map, but they are applied to both the generated distribution and the target distribution at the same grid location.

[0035] Subsequently, the structurally weighted feature distribution distance is calculated based on the structurally weighted generated distribution and the structurally weighted target distribution, using the following formula: , The distance is the structural weighted feature distribution distance. This distance is calculated by adjusting the generated distribution and the target distribution using the position weights generated from the target structural feature map, and comparing the two sets of renormalized structural weighted distributions. This ensures that locations with significant local structural changes receive a higher contribution while maintaining symmetrical comparison. When the original generated distribution and the target distribution are the same, the two sets of weighted distributions are the same, and the structural weighted feature distribution distance is zero.

[0036] Finally, based on the structural feature distribution distance and the structural weighted feature distribution distance, a structural feature distribution constraint loss is constructed. , The weighting coefficients are the weighted feature distribution distances of the structure.

[0037] S3. The structural feature distribution constraint loss is used as an additional supervision signal and combined with the original training objective of the generator network to form a joint training objective. The generator network is a generative adversarial network or a diffusion model. During the training phase, the generator or denoising network parameters are updated according to the generator network until the preset number of training rounds is reached or the preset stopping condition is met. After training is completed, the source domain brain image to be synthesized is input into the trained generator network, and the target domain synthesized brain image is output.

[0038] Generative Adversarial Networks (GANs) consist of a generator and a discriminator. The input image is... s The target image is x The generator output is G(s) Real-world paired samples (s,x) and generating paired samples (s,G(s)) The input is used by a discriminator to determine whether it is true or false. The generator's adversarial loss is also considered. , For the output of the discriminator network, To obtain the expectation of all real samples s; the discriminator feature matching loss is used to compare the differences between real paired samples and generated paired samples in the feature space of the discriminator's intermediate layer. , Indicates the discriminator's first... z Intermediate features of the layer This indicates the number of elements in the feature map of this layer. T Indicates the number of layers involved in feature matching. For all real paired samples (s,x) Take the expected value.

[0039] For generative adversarial networks, the structural feature distribution constraint loss is used as an additional supervision signal, and the joint objective loss function of the generator is: , and represents the weights of the feature matching loss and the structural feature distribution constraint loss, respectively. The structural feature distribution constraint is added as an additional structural distribution constraint to the original adversarial discrimination and discriminator feature matching objectives, without replacing the original GAN ​​loss or changing the discriminator structure or discriminator loss.

[0040] For the diffusion model, the structural feature distribution constraint uniformly applies to the final recovered clean image estimate, rather than directly to the intermediate diffusion variables or the prediction target. To uniformly represent the access locations in different diffusion backbones, the clean image estimate used to calculate the structural feature distribution constraint loss is denoted as... For image spatial diffusion models, The latent variables are recovered from network prediction results; for the latent space diffusion model, the target latent variables are first recovered, and then mapped to the image domain by the decoder to obtain the corresponding latent variables. Therefore, regardless of the specific form of diffusion parameterization, the structural feature distribution constraints are calculated in the image space.

[0041] For the diffusion framework, let's denote... x t Indicates the state of the diffusion process. o t This represents the prediction target defined by the corresponding diffusion backbone during training. This represents the network output. The predicted target forms differ among different diffusion backbones, but their original training loss remains the same. ,in, This represents the original definition loss corresponding to the diffusion backbone. The clean image estimate used for structural feature distribution constraints is denoted as... For image spatial diffusion models, The latent variables are directly recovered from the network prediction results; for the latent space diffusion model, the target latent variables are first recovered, and then mapped back to the image space by the decoder to obtain the corresponding latent variables. .

[0042] After incorporating structural feature distribution constraints, the objective loss function of the diffusion model... , Represents the target image. The loss weights are constrained by the structural feature distribution. The structural feature distribution constraint is gradually added as a warm-up weight during training. It does not change the original prediction target of the diffusion backbone, but only introduces an additional structural feature distribution constraint between the recovered image and the target image.

[0043] The generator or denoising network parameters are updated according to the adopted generative framework, which is a generative adversarial network or a diffusion model, until a preset number of training rounds is reached or a preset stopping condition is met. After training is completed, the source domain brain image to be synthesized is input into the trained generative network, and the target domain synthesized brain image is output.

[0044] To verify the technical effectiveness of the method in this embodiment, the following experiment was conducted, and the experimental setup and results are as follows.

[0045] Datasets. Four datasets were used to evaluate the structural feature distribution constraint proposed in this embodiment. The structural feature distribution constraint is named Structural Feature Distribution Constraint (SFDC) for short. The datasets include two anonymized private clinical CT-to-DWI datasets, ISD and YHD, and two public datasets, ISLES and SynthRAD2023 (RAD). ISD was used for comparing primary effects, loss component ablation, and structural feature extraction methods; YHD was used for external center validation and bidirectional cross-center evaluation with ISD; RAD was used for CT-to-T1 weighted MRI synthesis; and ISLES was used for CT-to-DWI synthesis and predictive consistency evaluation of the fixed U-Net segmentation model.

[0046] Table 1. Datasets, sources, number of patients, modalities, and tasks used in the experiment .

[0047] Data preprocessing. CT images were processed using fixed brain windows with a window level of 50 HU and a window width of 100 HU. Intensity was truncated to [0,100] HU and then linearly mapped to [0,255]. DWI images were mapped to [0,255] after robust max-min normalization. T1-weighted MRI images were mapped to [0,255] after z-score normalization. ISLES lesion masks underwent only spatial processing consistent with the corresponding images. Except for ISLES, each dataset was first divided into training and test sets at the case level in an 8:2 ratio, and then two-dimensional slices were generated. For ISLES, CT, DWI, and lesion masks were simultaneously divided by slice, and test slices were not used for model training.

[0048] Implementation details. The GAN experiments used a paired conditional image translation backbone, denoted as Base cGAN, to represent the main reference backbone. The generator employed an encoder-residual bottleneck-decoder structure, while the discriminator used a three-scale PatchGAN. The input resolution was 256×256, the optimizer was Adam, and the learning rate was 1×10⁻⁶. -4 , SFDC uses a Sobel-Sigmoid structure feature map. , Base cGAN and BBDM use a 128×128 auxiliary mesh, while Fast-DDPM uses a 64×64 auxiliary mesh. Default structure weighting coefficients. and structural change weights All values ​​were set to 1.0. The SFDC weight of the Base cGAN linearly decreased from 1.5 to 0.5 over the first 200 epochs. The SFDC weights of BBDM and Fast-DDPM were both 0.1; the warm-up weight of BBDM increased to 0.5 over 100 epochs, while that of Fast-DDPM... The value was set to 0.5, and the warm-up lasted for 50,000 iterations. The maximum number of training epochs for GAN experiments on ISD, YHD, and RAD were 800, 400, and 300 epochs, respectively, while the ISLES and randomized partitioning experiments had a maximum of 600 epochs.

[0049] Evaluation metrics. Image synthesis quality was evaluated using PSNR, MAE, SSIM, and FSIM, with higher PSNR, SSIM, and FSIM being better, and lower MAE being better. ISLES prediction consistency experiments used Dice, Precision, Recall, and HD95 to compare the consistency between the synthesized DWI and the predicted masks obtained after inputting the same fixed U-Net segmentation model to the real DWI.

[0050] The main results on the ISD dataset are shown in Table 2. Pairwise results for the five GAN backbones before and after adding SFDC are presented. Each backbone showed improvement in at least one metric, with MultiCycleGAN, UKAN, and Base cGAN showing improvement in all four metrics; ARGAN showed improvement in PSNR and MAE, while HisGAN showed improvement in PSNR, MAE, and SSIM. After adding SFDC to Base cGAN, PSNR increased from 28.9436 to 30.5657, MAE decreased from 7.8124 to 7.4473, SSIM increased from 0.8051 to 0.8123, and FSIM increased from 0.8954 to 0.8987. All five GAN models showed reduced reconstruction error or increased structural similarity after introducing SFDC, indicating that SFDC can be integrated into different GAN models and achieves improvement on most evaluation metrics.

[0051] Table 2. Pairwise quantitative results of five GAN backbones before and after adding SFDC on the ISD dataset. .

[0052] Figure 1 A representative case was selected from each of the five GAN backbones to observe changes in local structure and error before and after the addition of SFDC. Each row sequentially displays the absolute error map and error change map for input CT, GT DWI, Original, +SFDC, and two other settings, with magnified ROI maps of the same region below GT, Original, and +SFDC. In the error change map, blue indicates a decrease in absolute error after adding SFDC, and red indicates an increase in absolute error. Paired results from the five groups show that after adding SFDC, the high-error regions in most representative cases shrank, and the tissue boundaries and local textures within the ROI became closer to GT. The changes were more pronounced in MultiCycleGAN and Base cGAN, while ARGAN, HisGAN, and UKAN also showed varying degrees of local improvement. The visualization results are consistent with the quantitative trends in Table 2, indicating that SFDC has a positive effect on structural recovery for different backbones.

[0053] Loss composition ablation. Table 3 compares Baseline, location-by-location edge L1 constraint, unweighted SFDD, and full SFDC. Full SFDC outperforms the other settings in all four metrics; unweighted SFDD improves PSNR and SSIM, while full SFDC further reduces MAE and improves SSIM and FSIM, indicating that structural change weighting can further improve the synthesis results over the basic distribution distance.

[0054] Table 3. Loss composition and ablation results on the ISD dataset .

[0055] Figure 2 A more challenging case is presented to supplement the visualization comparison between different loss configurations. Compared to the baseline and Unweighted SFDD, the results generated by Edge-L1 and SFDC are closer to GTDWI in terms of overall profile, with the absolute error map of SFDC showing smaller local high error regions. SFDC outperforms Edge-L1 and Unweighted SFDD in PSNR, MAE, SSIM, and FSIM, indicating that weighting by structural change intensity has a positive effect on the overall improvement of integrity constraints.

[0056] Comparison of structural feature extraction methods. Table 4 compares four structural feature extraction methods—CNN, Gabor, LocalVar, and the default Sobel-Sigmoid—under the same other settings. All structural feature extraction methods can be integrated into the same distribution comparison framework, but their performance differs; the default SFDC method achieves the highest PSNR, SSIM, and FSIM.

[0057] Table 4. Comparison of different structural feature extraction methods on the ISD dataset .

[0058] Applicability across generative frameworks. Table 5 shows pairwise comparisons between the latent space diffusion backbone BBDM and the image spatial diffusion backbone Fast-DDPM. Adding SFDC to both diffusion backbones improves PSNR, SSIM, and FSIM, while decreasing MAE, indicating that SFDC can be used as an additional structural constraint to integrate with different diffusion frameworks.

[0059] Table 5. Application results of SFDC in different diffusion frameworks .

[0060] Figure 3 and Figure 4 The pairwise comparison results in Table 5 are supplemented from the perspectives of visualization and sample distribution. In the BBDM framework, the addition of SFDC makes the local high-signal regions within the ROI more pronounced, and the local brightness transition is closer to the Ground Truth, corresponding to a reduction in some high-error regions in the absolute error plot. In the Fast-DDPM framework, the local structural transitions in representative cases are also clearer than the corresponding baselines. The changes in the index distribution in the box plots are consistent with the results in the tables, supporting the integration of SFDC as a supplementary structural constraint into different diffusion backbones. The results of both diffusion models are interpreted according to the pairwise relationships within their respective frameworks.

[0061] External center validation at YHD. Table 6 presents paired results from another private clinical center, YHD. MultiCycleGAN, UKAN, and Base cGAN all showed improvement in all four metrics after incorporating SFDC; ARGAN and HisGAN showed improvement in three metrics, respectively. Base cGAN's PSNR increased from 21.2551 to 22.2575, MAE decreased from 12.3040 to 11.3496, SSIM increased from 0.5955 to 0.6599, and FSIM increased from 0.8081 to 0.8214.

[0062] Table 6. Pairwise quantitative results of five GAN backbones before and after adding SFDC on the YHD dataset. .

[0063] Figure 5 and Figure 6 The table results are supplemented from two levels: local error and patient-level distribution. In representative cases, the composite results after adding SFDC are generally closer to GT DWI within the indicated ROI, and the high error areas in the absolute error plot are correspondingly reduced. The error change plot retains both local improvement and regression information, where blue indicates a decrease in error after adding SFDC, and red indicates an increase in error. Patient-level box plots show the distribution of indicators and outliers on different backbones, avoiding judgment of model performance based solely on representative cases.

[0064] Bidirectional cross-center evaluation using ISD and YHD. Models trained and selected on ISD were directly used for YHD testing, and vice versa, models trained and selected on YHD were directly used for ISD testing. Fine-tuning was not performed in either direction at the target center. Most backbone metric combinations in both directions improved after incorporating SFDC, indicating that SFDC is generally applicable under center distribution shifts.

[0065] Table 7. Two-way cross-center quantitative results between ISD and YHD .

[0066] CT-to-T1 weighted MRI synthesis on the RAD dataset. Table 8 presents the paired results for the five GAN backbones. HisGAN, MultiCycleGAN, UKAN, and Base cGAN all showed improvements in all four metrics; ARGAN showed improvements in PSNR, MAE, and SSIM, with its FSIM being 0.0001 lower than the original model. The results indicate that SFDC can still provide additional structural constraints after the target modality is changed.

[0067] Table 8. Paired quantitative results of five GAN backbones before and after adding SFDC on the RAD dataset. .

[0068] Figure 7 and Figure 8 The table results are supplemented from two levels: local error and patient-level distribution. RAD (Radar Adaptive Disorder) is a CT to T1-weighted MRI synthesis task, and its local differences are mainly reflected in ventricular boundaries, sulcus contours, and tissue transition regions. In representative cases, the synthesized results after adding SFDC (Small Fractional Diagnostic Capture) are generally closer to GTT (Gross-T1 Weighted MRI) within the indicated ROI, and the high-error areas in the absolute error map are correspondingly reduced. The error change map simultaneously shows local improvement and regression, where blue indicates a decrease in error after adding SFDC, and red indicates an increase in error. Patient-level box plots are used to show the overall distribution on different skeletons, without directly extrapolating the visual manifestations of individual cases to overall conclusions.

[0069] Randomization and patient-level robustness analysis. A multi-source mixed dataset was formed by randomly selecting 250 cases each from ISD and YHD, and 100 cases from ISLES. Cases were randomly partitioned at an 8:2 ratio, repeated three times. Table 9 shows that adding SFDC to the three independent partitions improved PSNR, MAE, and SSIM, with FSIM showing a similar trend. After patient-level aggregation, PSNR increased by 0.2243, 0.3359, and 0.4040, respectively; MAE decreased by 0.3695, 0.4328, and 0.3879, respectively; and SSIM increased by 0.0169, 0.0209, and 0.0057, respectively. The Wilcoxon signed-rank test for all three indicators showed p < 0.001.

[0070] Table 9. Robustness results under three case-level randomizations .

[0071] ISLES Fixed U-Net Prediction Consistency Evaluation. Table 10 shows the CT-to-DWI synthesis results on ISLES. After adding SFDC to the two generation chains, PSNR, MAE, SSIM, and FSIM were all improved. The fixed U-Net segmentation model was trained using only real DWI and its lesion mask, and then all parameters were frozen. During testing, real DWI and synthesized DWI were input into the same fixed model, and the prediction mask corresponding to the real DWI was used as the reference for consistency comparison.

[0072] Table 10. Quantitative results of CT-to-DWI synthesis on the ISLES dataset. .

[0073] Table 11 compares the consistency between the synthetic DWI prediction mask and the real DWI reference prediction mask. Dice, Precision, and Recall were calculated on 328 slices of the real DWI reference prediction; HD95 was calculated only on slices where both predictions included the foreground. Each method was paired one-to-one with the real DWI reference prediction on 538 test slices. After adding SFDC, the overlap index of both generative chains increased and HD95 decreased, indicating that the fixed model response induced by synthetic DWI is closer to the response induced by real DWI.

[0074] Table 11. Consistency between synthetic DWI and real DWI predicted by fixed U-Net on ISLES .

[0075] The above experiments examined the applicability of SFDC to different GAN backbones and diffuse backbones, and investigated its stability through experiments involving external centers, bidirectional cross-centers, different target modalities, case-level randomization, and fixed U-Net prediction consistency. The results show that SFDC, as an additional structural supervision method, has cross-frame applicability.

[0076] This embodiment describes structural supervision in cross-modal brain image synthesis as a comparison of structural feature distributions in a shared space. Unlike loss calculation methods that primarily rely on positional or local correspondences, this embodiment constrains the overall distribution of structural features in the image space through structural feature distribution distance and structural weighted feature distribution distance. This constraint is decoupled from the generator structure and can be used as an additional supervisory signal in different image generation frameworks.

[0077] In summary, this embodiment first compares the overall distribution of structural features between the generated and target ends on a shared spatial grid. Then, it constructs positional weights based on the intensity of local structural changes at the target end, and applies these positional weights to both the generated and target distributions before renormalizing them to improve the contribution of key locations in the distribution comparison. Finally, it combines the overall distribution distance and the structure-weighted distribution distance into the SFDC loss. During the training phase, this loss is integrated into the original generation framework; during the inference phase, no real target domain brain image is required, and only the source domain brain image to be synthesized is input into the trained generation network to obtain the synthesized target domain brain image.

[0078] The aforementioned structural feature distribution constraint loss is only used for training the generative network and does not change the main structure of the generative network. For generative adversarial networks, this loss is jointly optimized with the original training loss of the generator; for diffusion models, the structural feature distribution constraint loss between the estimated clean image recovered from the output of the diffusion network and the real target domain brain image is calculated and jointly optimized with the original training loss of the diffusion model.

[0079] Experiments on four datasets—ISD, YHD, ISLES, and RAD—cover CT-to-DWI synthesis, CT-to-T1 weighted MRI synthesis, external center and cross-center evaluation, random partitioning, GAN and diffuse backbone, and fixed U-Net prediction consistency assessment. Experimental results show that the SFDC proposed in this embodiment can improve the quality of medical image synthesis and structural response consistency.

[0080] This embodiment also provides a cross-modal brain image synthesis system based on structural feature distribution constraints, used to implement the above-mentioned cross-modal brain image synthesis method based on structural feature distribution constraints, including: The brain image acquisition and synthesis module is used to acquire source domain brain images and real target domain brain images, input the source domain brain images into the generation network, and obtain the target domain synthesized brain images. The structural feature extraction module is used to extract structural features from the synthetic brain image of the target domain and the real brain image of the target domain, and obtain the generated structural feature map and the target structural feature map. The structural feature distribution constraint loss generation module maps the generated and target structural feature maps to a shared space grid, constructs a unified structural feature distribution representation, and obtains the generated and target structural feature distributions. On the shared space grid, it calculates the structural feature distribution distance between the generated and target structural feature distributions. It constructs position weights based on the intensity of local structural changes in the target structural feature map, applies these position weights to both the generated and target structural feature distributions, and normalizes them to obtain a structure-weighted generated distribution and a structure-weighted target distribution. Based on these two distributions, it calculates the structure-weighted feature distribution distance. Finally, it weights and combines the structural feature distribution distances to obtain the structural feature distribution constraint loss. The model training and target domain synthetic brain image generation module is used to combine the structural feature distribution constraint loss as an additional supervision signal with the original training objective of the generator network to form a joint training objective; the generator network is a generative adversarial network or a diffusion model; during the training phase, the generator or denoising network parameters are updated according to the generator network until a preset number of training rounds is reached or a preset stopping condition is met; after training is completed, the source domain brain image to be synthesized is input into the trained generator network, and the target domain synthetic brain image is output.

[0081] This embodiment also provides a cross-modal brain image synthesis device based on structural feature distribution constraints, including a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the above-described cross-modal brain image synthesis method based on structural feature distribution constraints.

[0082] This embodiment also provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the above-described cross-modal brain image synthesis method based on structural feature distribution constraints.

[0083] While exemplary embodiments of the present invention have been described above, many other variations or modifications conforming to the principles of the present invention can be directly determined or derived from the disclosure of the present invention without departing from the spirit and scope thereof. Therefore, the scope of the present invention should be understood and recognized to cover all such other variations or modifications.

Claims

1. A cross-modal brain image synthesis method based on structural feature distribution constraints, characterized in that, This includes the following operations: S1. Obtain source domain brain images and real target domain brain images. Input the source domain brain images into the generator network to obtain the target domain synthetic brain images. Extract the structural features of the target domain synthetic brain images and the real target domain brain images respectively to obtain the generated structural feature map and the target structural feature map. S2. Map the generated structural feature map and the target structural feature map to a shared space grid to construct a unified structural feature distribution representation, and obtain the generated structural feature distribution and the target structural feature distribution. On the shared spatial grid, the structural feature distribution distance between the generated structural feature distribution and the target structural feature distribution is calculated. Position weights are constructed based on the intensity of local structural changes in the target structural feature map, and these weights are applied to both the generated and target structural feature distributions. After normalization, a structure-weighted generated distribution and a structure-weighted target distribution are obtained. Based on these two distributions, the structure-weighted feature distribution distance is calculated. Finally, the structural feature distribution distance and the structure-weighted feature distribution distance are weighted and combined to obtain the structural feature distribution constraint loss. S3. The structural feature distribution constraint loss is used as an additional supervision signal and combined with the original training objective of the generator network to form a joint training objective; the generator network is a generative adversarial network or a diffusion model; during the training phase, the generator or denoising network parameters are updated according to the generator network until the preset number of training rounds is reached or the preset stopping condition is met. After training is complete, the source domain brain image to be synthesized is input into the trained generative network, and the target domain synthesized brain image is output.

2. The cross-modal brain image synthesis method based on structural feature distribution constraints according to claim 1, characterized in that, In S1, the operation for extracting structural features is as follows: calculate the gradient response of the brain image in the horizontal and vertical directions respectively, construct the gradient magnitude, obtain the average gradient magnitude of all spatial locations, and after normalization, obtain the structural feature map through thresholded Sigmoid mapping.

3. The cross-modal brain image synthesis method based on structural feature distribution constraints according to claim 1, characterized in that, In S2, the operation for constructing a unified structural feature distribution representation is as follows: Scaling the generated structural feature map and the target structural feature map to an auxiliary resolution. H × W , H, W Representing height and width respectively, a discrete space is constructed on the auxiliary mesh. , For the first i Each grid location N=H×W ;exist At this point, the feature values ​​of the generated structural feature map and the target structural feature map are denoted as follows: , , , The generated structural feature map and the target structural feature map are located at the grid positions, respectively. eigenvalues, , These are the generated structural feature map and the target structural feature map, respectively. The generated structural feature map and the target structural feature map are normalized into discrete structural feature distributions, respectively, to obtain the generated structural feature distribution and the target structural feature distribution.

4. The cross-modal brain image synthesis method based on structural feature distribution constraints according to claim 1, characterized in that, In S2, the structural feature distribution distance is calculated using the following formula: ， ， Distance of structural feature distribution , These represent the generated structural feature distribution and the target structural feature distribution at the grid positions, respectively. The distribution weights, , These represent the generated structural feature distribution and the target structural feature distribution at the grid positions, respectively. The distribution weights, For grid position and The normalized Euclidean distance between them It is a two-dimensional Euclidean distance. The diagonal length of the auxiliary grid is used to normalize the distance to a uniform scale.

5. The cross-modal brain image synthesis method based on structural feature distribution constraints according to claim 4, characterized in that, In S2, the structure-weighted generation distribution and the structure-weighted target distribution are calculated using the following formulas: , , , These represent the structure-weighted generation distribution and the structure-weighted target at the grid location, respectively. The distribution weights, , These represent the generated structural feature distribution and the target structural feature distribution at the grid positions, respectively. The distribution weights, , Grid positions , Position weights.

6. The cross-modal brain image synthesis method based on structural feature distribution constraints according to claim 1, characterized in that, Brain imaging includes any two modalities of CT images, DWI images, and T1-weighted MRI images.

7. The cross-modal brain image synthesis method based on structural feature distribution constraints according to claim 1, characterized in that, The structural feature distribution constraint loss is applied to the clean image estimate finally recovered by the diffusion model. Target loss function , This represents the original definition loss of the corresponding diffusion backbone. Represents the target image. The loss weights are constrained by the structural feature distribution. These are warm-up weights that are gradually added during training; for image spatial diffusion models, The latent variables are directly recovered from the network prediction results; for the latent space diffusion model, the target latent variables are first recovered, and then mapped to the image space through the decoder. .

8. A cross-modal brain image synthesis system based on structural feature distribution constraints, used to implement the cross-modal brain image synthesis method based on structural feature distribution constraints as described in claim 1, characterized in that, include: The brain image acquisition and synthesis module is used to acquire source domain brain images and real target domain brain images, input the source domain brain images into the generation network, and obtain the target domain synthesized brain images. The structural feature extraction module is used to extract structural features from the synthetic brain image of the target domain and the real brain image of the target domain, and obtain the generated structural feature map and the target structural feature map. The structural feature distribution constraint loss generation module is used to map the generated structural feature map and the target structural feature map to a shared space grid, and to construct the generated structural feature distribution and the target structural feature distribution. On the shared spatial grid, the structural feature distribution distance between the generated structural feature distribution and the target structural feature distribution is calculated. Position weights are constructed based on the intensity of local structural changes in the target structural feature map, and these weights are applied to both the generated and target structural feature distributions. After normalization, a structure-weighted generated distribution and a structure-weighted target distribution are obtained. Based on these two distributions, the structure-weighted feature distribution distance is calculated. Finally, the structural feature distribution distance and the structure-weighted feature distribution distance are weighted and combined to obtain the structural feature distribution constraint loss. The model training and target domain synthetic brain image generation module is used to combine the structural feature distribution constraint loss as an additional supervision signal with the original training objective of the generator network to form a joint training objective; the generator network is a generative adversarial network or a diffusion model; during the training phase, the generator or denoising network parameters are updated according to the generator network until the preset number of training rounds is reached or the preset stopping condition is met. After training is complete, the source domain brain image to be synthesized is input into the trained generative network, and the target domain synthesized brain image is output.

9. A cross-modal brain image synthesis device based on structural feature distribution constraints, characterized in that, It includes a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the cross-modal brain image synthesis method based on structural feature distribution constraints as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the cross-modal brain image synthesis method based on structural feature distribution constraints as described in any one of claims 1-7.

Images