Medical image cross-modal synthesis method, terminal equipment and storage medium

Abstract

This invention discloses a cross-modal synthesis method, terminal device, and storage medium for medical images. It establishes a bridge-type diffusion generation path with the source modality and target modality as endpoints to improve cross-modal alignment and structural fidelity. It proposes an internal and external joint condition injection strategy, which uses external conditions to generate preliminary estimates and refines them based on self-conditions to achieve reuse of intermediate results and stable detail recovery. By constraining the consistency of output predictions at adjacent time steps, it enhances the temporal coherence and robustness of the model in the generation process.

Description

Technical Field

[0001] This invention relates to medical image synthesis technology, and in particular to a method, terminal device and storage medium for cross-modal synthesis of medical images. Background Technology

[0002] Cross-modal MRI image synthesis involves synthesizing target modality (e.g., T2, FLAIR, PD) images from existing source modality (e.g., T1) images. The clinical motivation for this task is to reduce repetitive or high-risk MRI scans, shorten examination time, reduce patient burden, and provide a complete and consistent modal dataset for downstream tasks such as segmentation, quantitative analysis, and radiotherapy planning. This task requires accurately reproducing the intensity distribution, contrast relationships, and detailed textures of the target modality while maintaining geometric consistency of anatomical structures. It also demands stable synthesis quality under domain shifts caused by different equipment, centers, and acquisition protocols.

[0003] To achieve the aforementioned goals, traditional approaches often start with statistical or physical modeling to approximate the relationships between modalities in an interpretable way. Common practices include intensity mapping and histogram matching, which approximate the target modality by monotonically transforming the pixel value distribution; prior-based regularized reconstruction methods, which optimize the task by balancing data consistency and prior constraints; and dictionary learning methods that rely on precise registration, which synthesize the target image by combining external samples or retrieving data from a dictionary into structural blocks. These methods have clear implementation paths and low debugging costs, but they are sensitive to registration quality and prior assumptions, and struggle to fully characterize the strongly nonlinear and anatomically related relationships between modalities. They often exhibit details missing, contrast bias, or artifacts at tissue boundaries, in fine structures, and around lesions. As differences in acquisition protocols and individual patient variations increase, these limitations are further amplified, leading to insufficient robustness of the methods in real-world scenarios.

[0004] Deep learning methods construct cross-modal synthesis as an end-to-end supervised regression model. Convolutional neural networks, exemplified by U-Net, can learn complex mappings from source to target modalities under conditions of sufficient paired data, achieving stable optimization in overall contrast and local texture, through joint optimization of pixel reconstruction loss (L1 / L2), structural similarity (SSIM), and perceptual loss. To alleviate the long-range dependency and oversmoothing problems caused by the local receptive field of convolutional kernels, existing works have introduced structures such as feature pyramids, channel / spatial attention, dilated convolutions, and multi-scale fusion to improve the perception of large-scale anatomical context and fine-grained texture. However, when paired samples are insufficient, scanning protocols change significantly, or there is a shift between the test and training domains, the generalization performance of convolutional models is still prone to degradation, manifesting as unstable contrast reproduction, softened details, or local structural deformation.

[0005] The approach based on generative adversarial networks introduces adversarial loss in addition to reconstruction loss, which can improve the realism of synthesized images from the discriminator's perspective, and usually has advantages in terms of texture detail and subjective quality. When paired data is insufficient, the cycle consistency framework can learn cross-modal mappings under unpaired conditions, thereby expanding the range of usable data. It should be noted that adversarial training itself is prone to instability and mode collapse; when the loss weights are set inappropriately, the model may overemphasize the realism of high-frequency textures while neglecting the constraints on the position and shape of anatomical structures, specifically manifesting as boundary shifts and local deformations. Cycle consistency constraints are also not equivalent to anatomical structure consistency, and semantic shifts may still occur in actual results. For medical imaging applications that emphasize structural fidelity, these phenomena directly affect the reliability and feasibility of the method.

[0006] To better model long-range dependencies and global context, researchers have introduced Transformer and self-attention mechanisms into cross-modal generation. Window self-attention and shifted window designs expand the effective receptive field while controlling computational costs; combined with multi-scale feature pyramids, they can, to some extent, simultaneously consider both overall structure and local details. However, medical images are high-resolution and structurally complex, and self-attention still incurs significant computational and memory overhead at high resolutions; window segmentation also easily leads to discontinuous block boundaries and insufficient characterization of cross-block relationships. Furthermore, these methods typically supervise each time step separately during training, while inference requires sequential generation of results step-by-step. The mechanisms of these two approaches are inconsistent, and small errors can easily accumulate during iteration, eventually manifesting as visible structural biases.

[0007] Denoising diffusion models have been gradually applied to cross-modal generation of medical images due to their relatively stable training, high generation quality, and controllable sampling process. Conditional diffusion injects the source modality as a condition at each time step, and can generally reproduce the contrast and texture of the target modality well. However, the forward process of standard diffusion usually starts from pure noise, which is naturally decoupled from the distribution of the source modality; even if conditions are added at the network input, the sampling path itself is not a constrained path "from source to target", thus lacking continuous and direct guidance for geometric alignment. At the same time, mainstream training methods supervise each time step separately, lacking explicit temporal consistency constraints; in the inference stage, small deviations will gradually amplify along the time dimension, often manifesting as line drift, softened boundaries, or texture breaks.

[0008] To better align with the task attribute of "gradually transitioning from the source distribution to the target distribution," diffusion bridges and the Schrödinger Bridge paradigm have been used for cross-modal generation. These methods construct a stochastic process connecting the source and target distributions within a finite time frame, allowing sampling to migrate from the source state to the target state along the bridge trajectory. Compared to starting with noise, this path offers better task alignment and facilitates the continuous use of the source modality to constrain the geometry. Existing research has shown that the bridge paradigm helps shorten the effective sampling path and improves structural alignment and interpretability. However, existing methods often still rely on independent supervision at each time step, while the inference phase depends on sequential sampling, leading to a mechanistic inconsistency between training and inference. Furthermore, insufficient reuse of intermediate predictions from the previous step weakens the stability and temporal coherence of iterative sampling. In sensitive areas such as ventricular boundaries, cortical folds, white-gray matter junctions, and lesion margins, contrast shift, boundary blurring, or softening of details are still common, all of which affect usability in clinical settings.

[0009] The existing technology has the following drawbacks:

[0010] (1) Limited source information constraints: Most existing methods only use the source mode as input or loss condition, without fully integrating the source image into the sampling trajectory. The initial or intermediate states lack geometric guidance from the source mode, which easily leads to local structural offset and alignment errors.

[0011] (2) Insufficient reuse of historical information: Ordinary diffusion methods lack effective self-conditioning mechanisms, do not utilize the model's own historical prediction information, cannot continuously refine and correct on the sampling chain, and are prone to error accumulation.

[0012] (3) Training and inference mismatch: The training phase is independently supervised by time step, while the inference phase is iterated sequentially and lacks cross-step consistency constraints. Errors are easy to accumulate and drift, affecting structure fidelity and sampling stability. Summary of the Invention

[0013] The technical problem to be solved by the present invention is to provide a method, terminal device and storage medium for cross-modal synthesis of medical images, which addresses the shortcomings of the existing technology and ensures structural alignment and fidelity.

[0014] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a method for cross-modal synthesis of medical images, wherein model training includes the following steps:

[0015] S1. After registering the volumetric data of the medical image, slice the volumetric data along a fixed direction, and preprocess the sliced ​​data to obtain the source modality image. and target modal image ;

[0016] S2. Calculate the intermediate state at any time step t using the following formula. : ;in, For noise, I is with An identity matrix of the same dimension; t∈{0,…,T}, where T is the number of time steps; As time weight, For noise variance; y represents the target modality image, and y represents the source modality image;

[0017] S3, intermediate state Source modal images Input a neural network, train the neural network, and obtain a synthetic model; wherein, the loss function of the neural network is: ; ; ; For pixel-level reconstruction loss, For trajectory consistency constraint loss;

[0018] The specific acquisition process includes: obtaining the intermediate state Source modal images Inputting the data into a neural network yields a rough preliminary target estimate. ;right With probability Applying a Bernoulli mask yields the self-conditional ;Will Source modal images and self-condition Simultaneously, the input is given to the neural network to obtain a refined target estimate. ;

[0019] The specific acquisition process includes: utilizing the refined target estimate of the current step. With source modal image Online posterior recursion generates the state of the previous time step. ;Will , and target estimation The data is input into the neural network again, and a prediction is obtained. .

[0020] This invention establishes a constrained generation process using the source mode as the geometric anchor point, ensuring that the sampling trajectory aligns with the "from source to target" task attributes, thus guaranteeing structural alignment and fidelity. This invention introduces a joint internal and external condition mechanism of "external conditions + self-conditions," efficiently combining external conditions and historical predictions during iterative sampling to suppress the feedback and amplification of historical errors, thereby improving the detail richness and stability of the synthesized target. This invention alleviates the mismatch between model construction training and inference sampling through trajectory consistency constraints, making the outputs of adjacent time steps coherent to reduce error accumulation. The generated target images obtained using the method of this invention exhibit strong structural alignment, high detail fidelity, and stable sampling. This invention can improve the geometric consistency and visual quality of synthesized images under different devices and imaging conditions.

[0021] ; T is the number of time steps.

[0022] , T is the number of time steps.

[0023] The method of the present invention further includes:

[0024] During model inference, for a new source modality image, the new source modality image y and its intermediate states are... Input the synthetic model, perform reverse reasoning, and obtain the generated target modality image. "Reverse reasoning" refers to, within the framework of the diffusion bridge, using... Starting from T, decrease from T to 0 at time step t; at each time step, the self-condition from the previous back-reasoning step is... Source mode and current status Input the synthesis model to obtain And perform posterior recursive calculations. This process is repeated until the target estimate is finally output. .

[0025] In this invention, the neural network is a U-Net network.

[0026] As an inventive concept, the present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the steps of the above method.

[0027] As an inventive concept, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon; when the computer program / instructions are executed by a processor, they implement the steps of the above-described method.

[0028] Compared with the prior art, the beneficial effects of this invention are as follows: By introducing a bridge diffusion model, joint internal and external conditional injection, and trajectory consistency constraints under online posterior recursion, this invention brings the following beneficial effects in terms of structure fidelity and temporal stability:

[0029] (1) Structure of aligned MRI cross-modal synthetic images: During the bridge diffusion process with endpoint constraints, the sampling trajectory gradually approaches the target modality from the source modality, significantly reducing geometric drift and deformation in cross-modal mapping; clearer aligned structural boundaries can be achieved in areas such as cortical swirl, ventricular boundaries, and white matter-gray matter junction.

[0030] (2) Improved details and contrast of cross-modal image synthesis: The mechanism of joint internal and external conditional injection first obtains a coarse estimate based on external source conditions, and then performs residual refinement under the guidance of self-conditions. This effectively enhances the recovery of high-frequency texture and contrast, reduces over-smoothing and local artifacts in the synthesized image, and has stronger adaptability to mild domain offset problems such as cross-device.

[0031] (3) Enhanced temporal coherence and robustness of image synthesis: online posterior recursion combined with trajectory consistency constraints ensures that the endpoint estimates of adjacent time steps remain consistent, suppressing error accumulation and boundary jitter in step-by-step iteration; a smoother generated sequence is obtained under the same number of steps, and stable quality can still be maintained when fewer steps are set.

[0032] (5) Improve the observability and engineering usability of cross-modal synthesis process: Bridge generation provides continuous intermediate states in the time dimension, which makes it easy for people to intuitively examine the evolution process of structure and comparison and locate problems; the overall interface is simple and easy to integrate with downstream modules such as segmentation and quantitative analysis. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating the overall method of an embodiment of the present invention;

[0034] Figure 2 This is a training architecture diagram of an embodiment of the present invention;

[0035] Figure 3 This is a diagram illustrating the reasoning architecture of an embodiment of the present invention;

[0036] Figure 4 This is a qualitative comparison image of multi-contrast MRI translation in an embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Example 1

[0039] Embodiment 1 of this invention proposes a multi-contrast MRI cross-modal generation method, CoTraBridge, based on diffusion bridging. Its core feature is the introduction of a bridge-type diffusion model into MRI image cross-modal generation, integrating an internal-external joint conditional injection strategy with trajectory consistency constraints. The overall process is as follows: Figure 1 As shown, the main steps include:

[0040] (1) Acquiring and preprocessing training data: This embodiment of the invention uses the publicly available IXI dataset and BraTS 2021 dataset as experimental data sources. First, rigid and affine registration is performed on the multimodal MRI volume data of each subject to ensure spatial consistency between different modalities. Subsequently, the three-dimensional volume data is sliced ​​along the axis and uniformly cropped or filled. The resolution was adjusted. To eliminate intensity differences between different scanning devices and parameters, the images were normalized within a brain parenchyma mask. Percentile range linearly scaled to The training data is subjected to random rotation, flipping, cropping, and brightness perturbation to enhance its robustness and generalization ability. Paired source modality images are input during the model training phase. With target modal image During the inference phase, only the source modality image is input. .

[0041] (2) Constructing the diffusion bridge and forward process: In the embodiment of the present invention, in the discrete time interval A linear-Gaussian bridge with endpoint constraints is defined above, used to bridge the source mode. With target mode Establish a continuous transition between them. For any time step The intermediate state is defined as:

[0042]

[0043] in As time weight, This represents the noise variance. The following default scheduling is used:

[0044]

[0045] in This scheduling satisfies the condition that the mean is linearly interpolated between the target and source, and the noise variance is zero at both ends and reaches its maximum at the middle time step, exhibiting a Brownian bridge shape. During the training phase, we use the above formula to select paired samples... structure ; and in the reasoning stage As the initial starting point, then proceeding step by step from... Decrease sequentially to Perform reverse updates to gradually approximate the appearance of the target modality.

[0046] (3) Joint internal and external condition injection: Embodiment 1 of the present invention adopts a time-conditional generation network based on the U-Net structure. As the core inference network, two forward predictions are performed at each time step during the training period in step (2). The first forward prediction takes the intermediate states... and external conditions Injecting into the network yields a rough preliminary target estimate. ; then on by Applying a Bernoulli mask with a probability of (default 50%) yields a conditional result. During the second forward movement, , external conditions and self-condition Simultaneously, it is injected into the network to obtain a more refined target estimate. This combined internal and external conditional injection mechanism enables the model to fully utilize source modal information and its own historical predictions at each time step, improving the model's ability to faithfully reproduce the structure and details of the target modal image. For training supervision, pixel-level reconstruction loss is used.

[0047]

[0048] in For the target estimate predicted by the model, For the true target, this loss drives the model's predictions to align with the true target mode.

[0049] (4) Trajectory Consistency Constraint: To enhance the coherence between time steps and alleviate the training-inference mismatch problem, this invention constructs a trajectory consistency constraint mechanism. The embodiments of this invention are based on a bridge-type Gaussian posterior distribution, utilizing the prediction results of the current step. With source mode Online posterior recursive generation of the state of the previous time step Then, , and target estimation Input the data into the inference network again, and the prediction is obtained. By calculating the L1 difference between outputs at adjacent time steps:

[0050]

[0051] As a loss function, it will drive the prediction results of the target mode to remain consistent between adjacent time steps, thereby reducing the accumulation of error offsets from stepwise sampling during the inference stage.

[0052] (5) Loss Function Design and Optimization Objective: The training objective of this embodiment of the invention consists of reconstruction loss and trajectory consistency loss. Reconstruction Loss Used to constrain model output Compared with real target modal images Pixel-level consistency between them; trajectory consistency loss This is used to maintain continuity between adjacent time steps. The total loss is:

[0053]

[0054] We use the Adam optimizer for training, and hyperparameters such as learning rate, batch size, and time steps can be adjusted according to the dataset.

[0055] The model training process in this embodiment of the invention is as follows: Figure 2 As shown.

[0056] (6) Model training and inference process: During the training phase, paired samples are input. The process sequentially executes sample sampling, joint internal and external condition injection in two forward passes, trajectory consistency constraints under online posterior recursion, and loss function calculation, followed by parameter updates using gradient descent until the model converges. The inference phase only requires the source modality image as input. ,from Initially, conditional prediction and state updates are performed sequentially along the reverse path of bridge diffusion, ultimately outputting the generated target modality image. The quality of the generated data can be evaluated using metrics such as PSNR and SSIM.

[0057] The reasoning process of this invention embodiment is as follows: Figure 3 As shown.

[0058] (7) System Implementation and Module Composition: The method of Embodiment 1 of the present invention can be implemented as a software system comprising a data processing module, a diffusion bridge modeling module, an internal and external joint condition injection module, a trajectory consistency constraint module, and a model training and inference module. Each module is interconnected through a standard tensor interface, supports GPU-accelerated computing, and can be integrated into a medical image analysis platform.

[0059] Experimental results show that Embodiment 1 of this invention achieves a stable improvement over commonly used baselines on the public datasets IXI and BraTS 2021 (see Tables 1 and 2). Taking BraTS as an example, this invention achieves the best results in PSNR / SSIM across all four generation directions; it also maintains a leading position on the IXI dataset. Compared to representative diffusion models and GAN baselines, PSNR is improved by 0.25–0.85 dB, and SSIM is improved by 0.11–0.57 percentage points.

[0060] Table 1: PSNR / SSIM (%) on BraTS 2021

[0061]

[0062] Table 2: PSNR / SSIM (%) on IXI

[0063]

[0064] Qualitative results such as Figure 4 As can be seen in the magnified area of ​​the IXI image, the method of Embodiment 1 of the present invention retains sharper texture and more stable gray-white matter contrast at the cortical sulci / cerebellar lobe fissure; in the BraTS image, the method of Embodiment 1 of the present invention shows more coherent signal transition at the lateral ventricle boundary and its adjacent cortical sulci, with less oversmoothing and artifacts.

[0065] Ablation experiments further validated the effectiveness of each component module (see Table 3). In representative tasks, the performance indicators improved progressively from the basic bridge model to the addition of external conditions, and then to the introduction of self-conditions. When trajectory consistency constraints were superimposed, further gains were obtained in PSNR and SSIM, indicating that the combined internal and external conditions and the stride consistency constraints have an effective synergistic effect.

[0066] Table 3: Ablation Experiment: T1→T2 Translation Results on IXI and BraTS

[0067]

[0068] In summary, Embodiment 1 of the present invention, while ensuring consistency of anatomical structure and realism of target comparison, improves the stability of step sequence and robustness of inference, reduces the reliance on complex adversarial training and excessive parameter tuning, and has good engineering feasibility and clinical application potential.

[0069] Example 2

[0070] Embodiment 2 of the present invention provides a terminal device corresponding to Embodiment 1 above. The terminal device can be a processing device for a client, such as a mobile phone, a laptop, a tablet computer, a desktop computer, etc., to execute the method of the above embodiments.

[0071] The terminal device in this embodiment includes a memory, a processor, and a computer program stored in the memory; the processor executes the computer program in the memory to implement the steps of the method in Embodiment 1 described above.

[0072] In some implementations, the memory may be high-speed random access memory (RAM), and may also include non-volatile memory, such as at least one disk storage device.

[0073] In other implementations, the processor can be any type of general-purpose processor, such as a central processing unit (CPU) or a digital signal processor (DSP), and there is no limitation here.

[0074] Example 3

[0075] Embodiment 3 of the present invention provides a computer-readable storage medium corresponding to Embodiment 1 above, on which a computer program / instructions are stored. When the computer program / instructions are executed by a processor, they implement the steps of the method of Embodiment 1 above.

[0076] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof.

[0077] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0078] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0079] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0080] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0081] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A medical image cross-modality synthesis method, characterized by, The method comprises the following steps: S1, after registration of medical image volume data, slice the volume data along a fixed direction, pre-process the sliced data to obtain source modality images and target modality images ; S2, compute the intermediate state at any time step t using the formula : ; where, is noise, I is an identity matrix of the same dimension as ; time step , T is the number of time steps; is a time weight, is the noise variance; S3, the intermediate state and source modality images inputting a neural network, training the neural network to obtain a synthesis model; wherein a loss function of the neural network is: ; ; ; is a pixel-level reconstruction loss, is a trajectory consistency constraint loss; The specific acquisition process includes: inputting the intermediate state and the source modality image into a neural network to obtain a rough preliminary target estimation ; applying a Bernoulli mask to the rough preliminary target estimation with a probability to obtain a self-conditioning ; inputting the rough preliminary target estimation , the source modality image and the self-conditioning into the neural network at the same time to obtain a refined target estimation ; The specific acquisition process includes: using the refined target estimation of the current step and source modality images Online posterior recursion, generate the state of the previous time step ; input , and target estimation into the neural network again, and predict .

2. The medical image cross-modality synthesis method of claim 1, wherein, ； 。 3. The medical image cross-modality synthesis method of claim 1, wherein, ， 。 4. The medical image cross-modality synthesis method of claim 1, characterized in that: For a new source modality image, the new source modality image and the intermediate state thereof are input into the synthesis model, reverse inference is performed, and a generated target modality image is obtained The reverse inference time steps are generated step by step from T to 0 starting from T.

5. The medical image cross-modality synthesis method according to one of claims 1 to 4, characterized in that, The neural network is a U-Net network.

6. A terminal device comprising a memory, a processor, and a computer program stored on the memory; characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1-5.

7. A computer readable storage medium having stored thereon computer programs / instructions; characterized in that, The computer program / instructions, when executed by the processor, implement the steps of the method of any one of claims 1-5.

Images