Method, device, medium and program product for ct to mri volumetric data cross-modality conversion

Abstract

The application provides a CT-to-MRI volume data cross-modality conversion method, device, medium and program product. The method of the application comprises: acquiring CT volume data to be converted, and inputting the CT volume data to be converted into an encoder of a three-dimensional vector quantization autoencoder for mapping to obtain CT latent variables to be converted; performing conditional prior initialization on the CT latent variables to be converted by superimposing random disturbance to obtain noisy latent variables; inputting the noisy latent variables and the CT latent variables to be converted into a conditional latent space diffusion model, and under the guidance of the CT latent variables to be converted, performing reverse denoising in a latent space constructed by the three-dimensional vector quantization autoencoder by using a skip sampling strategy to obtain target MRI latent variables; and inputting the target MRI latent variables into a decoder of the three-dimensional vector quantization autoencoder for decoding and restoring to obtain target MRI volume data.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, specifically to a method, device, medium, and program product for cross-modal conversion of CT to MRI volumetric data. Background Technology

[0002] In the fields of medical image processing and computer-aided diagnosis, cross-modal conversion technology for computed tomography (CT) and magnetic resonance imaging (MRI) volumetric data aims to generate virtual images of one modality from images of another, thus compensating for the lack of multimodal imaging in clinical examinations. However, in real-world clinical scenarios involving cross-modal conversion of CT to MRI volumetric data, lesion areas are often small, with complex edge transitions and unstable contrast in surrounding tissues. Existing cross-modal conversion models can generally reconstruct normal anatomical contours well, but they often fail to stably preserve the boundaries, morphology, and grayscale differences of abnormal areas. This makes it difficult to directly apply the generated MRI volumetric data to downstream clinical tasks such as diagnostic support, treatment planning, or lesion segmentation. Therefore, ensuring the structural integrity and semantic consistency of lesion areas during cross-modal conversion, rather than merely satisfying the usability of the overall appearance, is a significant challenge in this field.

[0003] The existing technology mainly faces the following two technical bottlenecks: (1) Existing generative models (such as diffusion models) tend to fit the overall data distribution. During training, large areas of normal tissue features often dominate, which can easily dilute the gradient contribution of lesion-related features, resulting in insufficient learning of local small abnormal areas by the model. The generated results are prone to lesion boundary blunting or excessive smoothing of local details.

[0004] (2) There are significant differences between publicly available datasets and real clinical data from hospitals in terms of scanning protocols, reconstruction parameters, and case composition. If existing methods lack an intra-domain adaptation mechanism for real hospital data, the model is prone to mislearning non-pathological factors such as equipment differences, slice thickness differences, and grayscale fluctuations as inherent features of cross-modal mapping, which leads to highly unstable expression of local abnormal structures.

[0005] In summary, existing cross-modal conversion methods for CT to MRI volumetric data cannot stably preserve the boundaries, textures, and grayscale differences of key lesion areas while maintaining the overall anatomical structure stability when dealing with real clinical data. How to provide a cross-modal conversion method that can enhance the structural fidelity and semantic stability of lesion areas under real clinical data conditions is a technical challenge that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] This invention provides a method, device, medium, and program product for cross-modal conversion of CT to MRI volumetric data.

[0007] According to one aspect of the present invention, a method for cross-modal conversion of CT to MRI volumetric data is provided, comprising: acquiring CT volumetric data to be converted and inputting it into an encoder in a three-dimensional vector quantization autoencoder for mapping to obtain latent variables of the CT to be converted; performing conditional prior initialization on the latent variables of the CT to be converted by superimposing random perturbations to obtain noisy latent variables; inputting the noisy latent variables and the latent variables of the CT to be converted together into a conditional latent space diffusion model based on a three-dimensional U-Net diffusion network, and, guided by the latent variables of the CT to be converted, performing inverse denoising in the latent space constructed by the three-dimensional vector quantization autoencoder using a skip sampling strategy to obtain target MRI latent variables; and inputting the target MRI latent variables into the decoder of the three-dimensional vector quantization autoencoder for decoding and restoration to obtain target MRI volumetric data that preserves lesion boundaries and texture details.

[0008] According to at least one embodiment of the present invention, a cross-modal conversion method for CT to MRI volume data is provided, wherein a conditional latent spatial diffusion model is obtained through the following steps: acquiring paired original CT volume data and original MRI volume data and performing standardization processing to obtain standard CT volume data and standard MRI volume data; inputting the standard CT volume data and standard MRI volume data into the encoder of a three-dimensional vector quantization autoencoder for mapping to obtain CT latent variables and MRI latent variables; training a three-dimensional U-Net diffusion network with the CT latent variables as conditional inputs and the MRI latent variables as denoising targets, and using the trained three-dimensional U-Net diffusion network as a conditional latent spatial diffusion model.

[0009] According to at least one embodiment of the present invention, the loss function used when training the three-dimensional U-Net diffusion network in the CT to MRI volume data cross-modal conversion method is: a first joint loss function composed of signal-to-noise ratio weighted noise prediction loss and latent variable L1 reconstruction loss.

[0010] According to at least one embodiment of the present invention, the method for cross-modal conversion of CT to MRI volume data further includes, before acquiring the CT volume data to be converted and inputting it into the encoder of a three-dimensional vector quantization autoencoder for mapping: fine-tuning the three-dimensional vector quantization autoencoder, including: loading the three-dimensional vector quantization autoencoder with pre-trained weights, freezing the encoder parameters of the three-dimensional vector quantization autoencoder, and updating the parameters only of the decoder and quantization codebook of the three-dimensional vector quantization autoencoder. The loss function used when updating the parameters of the decoder and quantization codebook of the three-dimensional vector quantization autoencoder is a second joint loss function composed of reconstruction loss, vector quantization loss, perceptual loss, and three-dimensional structural similarity loss. The reconstruction loss uses mean squared error; the vector quantization loss is obtained by weighted summation of codebook loss and commitment loss. The codebook loss is used to push the codebook closer to the encoder output, and the commitment loss is used to constrain the encoder output from frequently deviating from the selected discrete codeword; the perceptual loss measures the semantic offset between the reconstruction result and the real sample by comparing differences in the high-level feature space; and the three-dimensional structural similarity loss is used to constrain the consistency between the reconstruction result and the real sample in local brightness, contrast, and structural relationships.

[0011] According to at least one embodiment of the CT-to-MRI volumetric data cross-modal conversion method of the present invention, the second joint loss function further includes adversarial loss, wherein the adversarial loss adopts a delayed activation strategy, specifically including: when the number of training rounds is less than a set threshold, only reconstruction loss, vector quantization loss, perceptual loss, and 3D structural similarity loss are used for optimization; when the number of training rounds is greater than or equal to the set threshold, adversarial loss is combined with the reconstruction loss, vector quantization loss, perceptual loss, and 3D structural similarity loss for optimization; wherein the adversarial loss is configured with adaptive adversarial weights, the adaptive adversarial weights are calculated based on the gradient norm ratio of the reconstruction target and the adversarial target on the last layer of the decoder, and after being smoothed and amplified by a preheating strategy, are used to perform weighted optimization of the adversarial loss.

[0012] A cross-modal conversion method for CT-MRI volumetric data according to at least one embodiment of the present invention acquires paired original CT volumetric data and original MRI volumetric data and performs standardization processing to obtain standard CT volumetric data and standard MRI volumetric data, comprising: uniformly converting the original CT volumetric data and original MRI volumetric data into a preset format and resampling the voxel spacing to a preset physical scale to obtain first CT volumetric data and first MRI volumetric data; performing bias field correction on the first MRI volumetric data to obtain second MRI volumetric data; aligning the first CT volumetric data to MRI space using an ANTs-based SyNRA registration method to obtain second CT volumetric data; replacing invalid regions in the second CT volumetric data and limiting window width and window level to obtain third CT volumetric data; and performing normalization processing to uniformly map the third CT volumetric data and the second MRI volumetric data to a preset intensity range to obtain standardized standard CT volumetric data and standard MRI volumetric data.

[0013] The CT-to-MRI volume data cross-modal conversion method according to at least one embodiment of the present invention further includes: performing total variational denoising on the acquired original CT volume data and original MRI volume data during the acquisition of paired original CT volume data and original MRI volume data; adjusting the original CT volume data and original MRI volume data to a fixed size by adopting a center clipping or symmetric filling strategy during the standardization process; and adopting a single-axis random halving clipping and axial rotation strategy during the training of the three-dimensional U-Net diffusion network, randomly clipping one spatial dimension each time during training, and remapping the clipped axis to the depth direction.

[0014] According to another aspect of the present invention, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, such that the processor performs a CT-to-MRI volumetric data cross-modal conversion method according to any embodiment of the present invention.

[0015] According to another aspect of the present invention, a readable storage medium is provided, wherein execution instructions are stored therein, which, when executed by a processor, are used to implement the CT-to-MRI volumetric data cross-modal conversion method of any embodiment of the present invention.

[0016] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements a CT-to-MRI volumetric data cross-modal conversion method according to any embodiment of the present invention.

[0017] This invention obtains noisy latent variables by conditionally initializing them with random perturbations superimposed on the latent variables of the CT to be transformed. This allows the inverse denoising process to start near a priori coordinates consistent with the spatial layout of the input CT, reducing the risk of overall anatomical structure drift. By continuously guiding the inverse denoising process within the latent space under the guidance of the latent variables of the CT to be transformed, the denoising prediction at each time step is always constrained by the priori CT structure, improving the positional consistency of the lesion region. By employing a skip-sampling strategy to directly jump to an earlier target time step, skipping some intermediate time steps, the number of network calls to the 3D U-Net diffusion network can be significantly reduced, lowering the inference computational overhead of the 3D volume data and achieving a reasonable balance between generation quality and inference efficiency. Attached Figure Description

[0018] The accompanying drawings illustrate exemplary embodiments of the invention and, together with the description thereof, serve to explain the principles of the invention. These drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification.

[0019] Figure 1 This is a flowchart illustrating a method for cross-modal conversion of CT to MRI volumetric data according to an embodiment of the present invention.

[0020] Figure 2 This is a model overall architecture diagram according to one embodiment of the present invention.

[0021] Figure 3 This is a flowchart illustrating a method for obtaining a conditional latent space diffusion model according to an embodiment of the present invention.

[0022] Figure 4 This is a schematic diagram of the network architecture of a three-dimensional U-Net diffusion network according to an embodiment of the present invention.

[0023] Figure 5 This is a comparison chart of CT-guided MRI generation results on a public dataset and a hospital dataset according to an embodiment of the present invention.

[0024] Figure 6 This is a flowchart illustrating a standardized processing method according to an embodiment of the present invention.

[0025] Figure 7 This is a schematic diagram of the network architecture of a three-dimensional vector quantization autoencoder according to an embodiment of the present invention.

[0026] Figure 8 This is a comparison chart of data reconstruction results of a three-dimensional vector quantization autoencoder according to an embodiment of the present invention on public datasets and hospital data.

[0027] Figure 9This is a schematic structural block diagram of an electronic device employing a processor-based hardware implementation according to an embodiment of the present invention. Detailed Implementation

[0028] The present invention will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present invention are shown in the accompanying drawings.

[0029] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other. The technical solution of this invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0030] Figure 1 This diagram illustrates the overall flow of a CT-to-MRI volumetric data cross-modal conversion method according to one embodiment of the present invention, based on... Figure 2 The overall architecture of the model is shown below. Figure 1 The method shown includes steps S110 to S140. This method can be executed by electronic devices such as mobile phones and tablets.

[0031] S110: Acquire the CT volume data to be converted and input it into the encoder in the three-dimensional vector quantization autoencoder (3D-VQ-VAE) for mapping to obtain the latent variables of the CT to be converted.

[0032] The CT volume data to be converted is the data that needs to be converted into MRI volume data in practical applications.

[0033] S120: Perform conditional prior initialization of the latent variables of the CT to be transformed by superimposing random perturbations to obtain noisy latent variables.

[0034] S130: The noisy latent variables and the CT latent variables to be converted are input into the conditional latent space diffusion model obtained based on the three-dimensional U-Net diffusion network. Under the guidance of the CT latent variables to be converted, the skip sampling strategy is used to perform reverse denoising in the latent space constructed by the three-dimensional vector quantization autoencoder to obtain the target MRI latent variables.

[0035] S140: Input the latent variables of the target MRI into the decoder of the three-dimensional vector quantization autoencoder for decoding and restoration, and obtain the target MRI volume data that preserves the lesion boundary and texture details.

[0036] In the task of cross-modal generation of 3D medical images, the sampling stage not only affects inference efficiency but also directly relates to the structural consistency and detail restoration quality of the generated results. For CT-to-MR conversion tasks, the goal is not to generate a completely new image under unconstrained conditions, but to gradually restore the tissue contrast and local texture of the target MR image based on the anatomical structure provided by the input CT. If the sampling process still adopts a method of starting from a completely random state and gradually backtracking along the complete time series, it often faces two problems in 3D volumetric data scenarios: first, the number of backsampling steps is too large, resulting in high inference overhead; second, in the long sequence denoising process, if there is a lack of sufficiently clear structural constraints, contour shifts, local blurring, and unstable restoration of lesion areas are likely to occur. Based on this task characteristic, this embodiment has made targeted adjustments to the DDPM sampling path, making the sampling process closer to the generation method of structurally constrained and gradually restored details required for cross-modal conversion of medical images.

[0037] This embodiment improves the initial sampling state. Unlike reverse generation starting directly from pure Gaussian noise, this embodiment uses conditional information for initialization, using the conditional modal representation as the basis for the sampling starting point and superimposing a certain intensity of random perturbation (corresponding to step S120). This design ensures that the sampling process no longer starts from a completely unstructured random state, but instead initiates reverse recovery near a priori location consistent with the spatial layout of the input CT. The reason for this approach is that CT and MR have a strong correspondence in organ contours, tissue boundaries, and lesion locations. If this structural prior can be preserved at the sampling starting point, subsequent reverse denoising can focus more on compensating for modal differences, rather than repeatedly completing global structural reconstruction throughout the entire sampling process. For 3D volumetric data generation, this conditional initialization reduces the risk of overall anatomical structure drift and helps improve the positional consistency of lesion regions and tissue boundaries during cross-modal conversion.

[0038] During the sampling process, the conditional information is not used only once at the initial moment, but continuously participates in noise estimation throughout the entire backsampling step. That is, the denoising prediction at each time step is guided by the conditional modality prior, and the update direction of the current state is always associated with the structural information of the input CT. In this way, the sampling process is no longer essentially an unconditional free generation, but a target modality recovery process with explicit structural constraints. The significance of this mechanism lies in splitting the cross-modal generation task into two mutually cooperating processes: the conditional information stably provides anatomical constraints such as organ morphology, spatial layout, and lesion location; and backdiffusion gradually supplements the grayscale distribution, soft tissue contrast, and local texture unique to MR. Compared to methods that rely entirely on the model to infer structures from random states, this continuous conditional guidance is more conducive to maintaining the global consistency of 3D volumetric data, reducing discontinuities between slices, and improving the recovery stability of local abnormal regions.

[0039] In addition to the constraints, this embodiment also compresses the reverse time step update path. Traditional DDPM typically denoises stepwise along the complete time series, that is, it only backtracks from the current time to the adjacent previous time each time. This embodiment, however, resamples the sampled time series by setting a step size, so that the reverse process can skip some intermediate time points and directly jump from the current state to an earlier target time (i.e., the skip sampling strategy in step S130). Let the current time be... The moment before the jump is Then its recovery form can be expressed as:

[0040] in, Indicates the latent variable from the current time. Jump directly to (restore) the target time. The new state afterwards; Indicates time The corresponding cumulative signal coefficients; Indicates time The corresponding cumulative signal coefficients; This represents the current noise predicted by the conditional latent spatial diffusion model; This represents the network parameters in the conditional latent spatial diffusion model.

[0041] Based on this, when Before reaching the endpoint, a random term matching the current noise level is added to maintain the necessary randomness of the sampling process. When sampling reaches the final moment, the restored result is directly output without adding new perturbations. The core purpose of this strategy is to reduce the number of network calls for backsampling, thereby reducing the overall inference cost. For 3D medical image generation, since a single network forward pass itself has high computational overhead, shortening the number of backsampling steps can directly improve inference efficiency, making the diffusion model more practically applicable in 3D cross-modal generation scenarios. At the same time, step sampling is not simply compressing the number of steps, but reducing redundant updates while retaining conditional constraints, achieving a more reasonable balance between sampling efficiency and restoration quality. Here, the random term can be understood as a Gaussian perturbation in the backsampling stage, used to characterize the generation uncertainty that still needs to be retained when backtracking from the current noise state to the previous state; in the conditional generation task in this paper, this perturbation is always subject to the prior constraints of CT structure and does not represent arbitrary changes to the anatomical structure.

[0042] As one possible implementation, before inputting the noisy latent variables and the CT latent variables to be transformed into the conditional latent spatial diffusion model obtained based on the 3D U-Net diffusion network, it is also necessary to construct a conditional latent spatial diffusion model. Figure 3A flowchart illustrating a method for obtaining a conditional latent spatial diffusion model according to one embodiment of the present invention is shown. Figure 3 The method shown includes steps S210 to S230.

[0043] S210: Acquire paired raw CT volume data and raw MRI volume data and perform standardization processing to obtain standard CT volume data and standard MRI volume data.

[0044] S220: Input the standard CT volume data and standard MRI volume data into the encoder of the three-dimensional vector quantization autoencoder for mapping to obtain CT latent variables and MRI latent variables.

[0045] S230: Using CT latent variables as conditional inputs and MRI latent variables as denoising targets, train a 3D U-Net diffusion network and use the trained 3D U-Net diffusion network as a conditional latent spatial diffusion model.

[0046] One possible implementation is to use two sets of 3D vector quantization autoencoders to map standard CT volume data and standard MRI volume data into a latent space, respectively, to obtain CT latent variables and MRI latent variables. Subsequently, using the CT latent variables as conditional inputs and the MRI latent variables as denoising targets, the 3D U-Net diffusion network is trained in the latent space. Compared to direct diffusion in pixel space, this embodiment significantly reduces memory consumption through latent space generation and focuses training on more semantically concentrated structural features.

[0047] Figure 4 A schematic diagram of the network architecture of a three-dimensional U-Net diffusion network according to one embodiment of the present invention is shown. This embodiment employs a conditional generation mode, continuously feeding CT latent variables into the three-dimensional U-Net diffusion network, ensuring that the MRI denoising process is always constrained by the original structural cues. This is particularly important for lesion-related tasks, because the lesion location, boundary orientation, and surrounding tissue relationships often already exist in some form in CT; conditional diffusion can reduce positional drift and morphological blurring caused by unconditional sampling. In other words, lesion perception does not originate from additional branches, but rather from the continuous preservation and utilization of conditional information in the latent space.

[0048] like Figure 5 As shown in the visualization results from the public dataset and the hospital dataset, the generated MRI volume data maintains a high degree of consistency with the target MRI in terms of overall anatomical structure, grayscale distribution, and major tissue boundaries. This indicates that the latent representation adapted from the hospital data can provide a more stable structural prior for the conditional diffusion model and improve the model's cross-modal generation capability on real clinical samples.

[0049] As one possible implementation, the loss function used when training the 3D U-Net diffusion network is a first joint loss function consisting of a signal-to-noise ratio-weighted noise prediction loss and a latent variable L1 reconstruction loss.

[0050] In the process of training the 3D U-Net diffusion network, the training objective is no longer to directly reconstruct MRI volume data, but to learn to accurately predict diffusion noise in the MRI latent space given CT latent variables, and further recover the latent representation of the target MRI volume data.

[0051] Regarding the technical problem this invention aims to solve, if only the 3D U-Net diffusion network is required to fit the global noise distribution without constraining the distance between the denoised potential result and the actual MRI latent variables, the generated result, while statistically reasonable, is still prone to deviations at lesion boundaries and local detail locations. Therefore, this embodiment combines the noise prediction loss and the latent variable reconstruction loss to form the optimization objective of the conditional latent spatial diffusion model.

[0052] During the training phase, let the MRI latent variables after encoder mapping of standard MRI volume data be denoted as . At any moment Adding Gaussian noise yields noisy latent variables. The corresponding noise is denoted as The core task of conditional latent space diffusion models is to base them on noisy latent variables. Time step And the CT latent variables predict the current noise, denoted as the prediction result. Therefore, the basic noise prediction error It can be represented as:

[0053] in, This represents the total number of elements involved in the calculation within a single latent variable sample. Indicates the first in the batch One sample, This represents the latent variables in CT. The basic noise prediction error can be used to directly constrain the model's prediction accuracy for diffused noise and is a fundamental objective for training latent spatial diffusion.

[0054] However, this embodiment does not simply average the noise error across all time steps, but instead introduces a weighting mechanism based on the signal-to-noise ratio. When time step... The corresponding cumulative signal coefficient is denoted as At that time, its signal-to-noise ratio It can be represented as:

[0055] Based on this, a threshold is used The loss weights at different time steps are truncated to obtain the loss weights for each time step. :

[0056] Therefore, the noise prediction loss based on signal-to-noise ratio weighting It can be represented as:

[0057] in, This indicates the batch size. The purpose of this design is to reduce the excessive dominance of high signal-to-noise ratio time steps on the overall training, allowing the 3D U-Net diffusion network to obtain a more balanced learning signal at different diffusion stages, thereby improving training stability. Noise prediction constraints alone are still insufficient to fully guarantee that the denoising results closely approximate the true MRI representation in the latent spatial semantics.

[0058] Therefore, after obtaining the predicted noise, the initial latent variables predicted by the model are recovered according to the diffusion inversion formula, denoted as... and compare it with the true latent variables. conduct L1 Constraints. The latent variable is the L1 reconstruction loss. It can be represented as:

[0059] in, This represents the initial MRI latent variables obtained from the model recovery. This represents the true MRI latent variables. Compared to noise prediction loss, the latent variable L1 reconstruction loss can more directly constrain the generated results from the semantic level of the latent space, reducing structural drift that occurs during the denoising process and making the recovered initial MRI latent variables closer to the true target. Combining the above two constraints, the first joint loss function of the conditional latent space diffusion model... It can be represented as:

[0060] in, This represents the weights of the latent variable L1 reconstruction loss. In actual training, This weight can be used to balance the accuracy of noise prediction and the accuracy of latent variable reconstruction: when its value is too small, the model tends to be more biased towards statistical noise fitting; when its value is too large, it may weaken the learning stability of the diffusion process itself. Therefore, by explicitly setting this weight, the model can achieve a balance between correctly predicting noise and correctly recovering latent variables.

[0061] Figure 6A flowchart illustrating a standardized processing method according to one embodiment of the present invention is shown. Figure 6 The method shown includes steps S410 to S450.

[0062] S410: Convert the original CT volume data and the original MRI volume data into a preset format, and resample the voxel spacing to a preset physical scale to obtain the first CT volume data and the first MRI volume data.

[0063] For example, the preset format can be NIfTI format, and the preset physical size can be 1mm×1mm×1mm.

[0064] Step S410 eliminates spatial resolution differences caused by different devices and scanning protocols, enabling subsequent volumetric data comparison and processing to be based on a unified physical scale benchmark.

[0065] S420: Perform bias field correction on the first MRI volume data to obtain the second MRI volume data.

[0066] For example, the bias field correction uses N4 bias field correction.

[0067] Step S420 can reduce low-frequency grayscale drift caused by non-uniformity of the radio frequency coil and prevent the model from misidentifying slow changes in grayscale as tissue structure features.

[0068] S430: Using the ANTs-based SyNRA registration method, the first CT volume data is aligned to the MRI space to obtain the second CT volume data.

[0069] The registration method used in step S430 combines rigid and non-rigid transformations, which can effectively correct local deformations while preserving global geometric relationships.

[0070] S440: Replace invalid regions (such as zero-value regions) in the second CT volume data and limit the window width and window level to obtain the third CT volume data.

[0071] Step S440 can truncate the volume data intensity to a clinically interpretable range, thereby reducing the interference of registration boundaries and extreme gray values ​​on model training and preventing the generative network from learning background holes or abnormally bright pseudo-films as effective structural features.

[0072] S450: Normalization processing is used to map the third CT volume data and the second MRI volume data to a preset intensity range to obtain standardized CT volume data and standard MRI volume data.

[0073] For example, the preset intensity range is [ The interval is [1, 1].

[0074] Compared to existing standardization processes, the standardization process comprising steps S410 to S450 places greater emphasis on constraining case-level differences in real hospital data. This is because lesion-related feature representations are easily masked by grayscale drift. Through a continuous processing flow of format conversion, resampling, bias correction, spatial registration, grayscale cropping, and normalization, the input data acquired by the model is no longer primarily dominated by differences in equipment and scanning protocols, but rather retains anatomical structure and lesion change information more effectively.

[0075] To expand the spatial diversity of training samples, improve the model's generalization ability to different scanning protocols, equipment differences, and case compositions, and prevent network overfitting, this embodiment also introduces data augmentation measures. However, in the real clinical scenarios addressed by this invention, lesion areas are often small in size and frequently exhibit small-scale gray-scale abrupt changes or complex boundary morphological alterations. If traditional data augmentation strategies with strong perturbation techniques (such as large-scale affine deformation, multi-axis synchronous scaling, or random strong pruning) are used, it is easy to cause irreversible geometric distortions to weak abnormal structures or to simultaneously dilute lesion features in multiple spatial dimensions, causing the model to lose key local pathological clues in the early stages of training. Therefore, how to control the memory burden of 3D training while avoiding strong data augmentation from destroying the geometric morphology and spatial continuity of small lesions is the main technical challenge faced by this invention in the data augmentation stage.

[0076] To address this technical problem, one possible implementation employs a data augmentation strategy centered on edge preservation, including: (1) During the process of acquiring paired raw CT volume data and raw MRI volume data (i.e., the data reading stage), total variational denoising is performed on the acquired raw CT volume data and raw MRI volume data to suppress random noise and preserve structural edges, providing an optional smoothing processing path for high-noise cases.

[0077] (2) During the standardization process (i.e., in the stage of spatial scale unification), the original CT volume data and the original MRI volume data are adjusted to a fixed size by adopting a central clipping or symmetrical filling strategy, rather than using excessive affine deformation, so as to avoid the lesion geometry being distorted additionally.

[0078] (3) During the training of the 3D U-Net diffusion network, a single-axis random halving and axial rotation strategy is adopted. Specifically, only one spatial dimension is randomly truncated in each training session, and the truncated axis is remapped to the depth direction, thereby preserving the complete structural information of the other two directions while controlling the memory overhead. Compared with truncating all three directions at the same time, the above strategy is more conducive to the preservation of local abnormal structures, because lesions are not diluted simultaneously in multiple dimensions, which can avoid excessive compression of small lesions caused by simultaneous downsampling in three dimensions. The halved axis is rotated to the depth direction before entering the network, so that the model can learn the structural changes in different spatial directions under a uniform input specification. For lesions with slender shapes or those extending along a single direction, this treatment helps to enhance the model's adaptability to directional differences. If necessary, the training input can also be kept divisible by geometric dimensions through edge-padding mechanism to reduce information truncation caused by boundary truncating.

[0079] Through the above data augmentation measures, this embodiment can effectively avoid the damage to small lesions and multi-dimensional simultaneous dilution caused by conventional augmentation strategies, while ensuring that the 3D training computing resources are controllable. This allows the model to continuously focus on learning the edge, texture and grayscale transition features of local abnormal areas, providing a robust and structurally accurate high-quality input for the subsequent potential representation of lesion perception and cross-modal generation.

[0080] As one possible implementation, before acquiring the CT body data to be converted and inputting it into the encoder in the three-dimensional vector quantization autoencoder for mapping, the method further includes: fine-tuning the three-dimensional vector quantization autoencoder. Figure 7 A schematic diagram of the network architecture of a three-dimensional vector quantization autoencoder according to one embodiment of the present invention is shown.

[0081] Since the quality of latent representation directly determines the upper limit of the conditional latent space diffusion model, if the 3D vector quantization autoencoder has already smoothed out the lesion boundaries during the encoding stage, even if the subsequent diffusion process can generate a reasonably sized MRI, it will be difficult to recover the true local abnormal structures. Based on this, this embodiment improves the traditional training process of the 3D vector quantization autoencoder by making it lesion-oriented. The network still uses the original encoder, vector quantizer, and decoder structure, but the training method is changed from full-parameter relearning to fine-tuning optimization based on existing weights. That is, a pre-trained 3D vector quantization autoencoder with pre-trained weights is loaded, the encoder parameters of the 3D vector quantization autoencoder are frozen, and only the decoder and quantization codebook of the 3D vector quantization autoencoder are updated. The purpose of this approach is to preserve the basic structural representation capability while allowing the model to prioritize the recovery of local details in hospital data with its limited learning capacity.

[0082] To verify the improvement effect of the above-mentioned improvement strategy on the potential representation quality, this embodiment further visualizes and analyzes the reconstruction results of the three-dimensional vector quantization autoencoder after fine-tuning the hospital data. For example... Figure 8 As shown in the MRI reconstruction results, the three-dimensional vector quantization autoencoder can recover the overall gray-scale distribution, ventricular structure, and cortical sulcus contours of brain tissue relatively well. This indicates that the strategy of freezing the encoder, fine-tuning the decoder, and quantizing the codebook did not destroy the basic anatomical representation ability in the original latent space.

[0083] As one possible implementation, the loss function used when updating the parameters of the decoder and quantization codebook of the 3D vector quantization autoencoder is a second joint loss function consisting of reconstruction loss, vector quantization loss, perceptual loss and 3D structural similarity loss.

[0084] The model uses mean squared error as the reconstruction loss to directly constrain the voxel deviation between the reconstruction result and the original input. Let the input volume data be... The reconstruction result is Then the reconstruction loss It can be represented as:

[0085] in, This represents the total number of voxels involved in the calculation within the volume data; This represents the square of the L2 norm. This loss is responsible for ensuring that the reconstruction result is consistent with the real data at the overall gray level, and is a fundamental term in the entire optimization process.

[0086] To ensure the stability of the discrete latent space representation, this embodiment retains the codebook loss and commitment loss from the vector quantization module. These two losses are combined into the vector quantization loss. :

[0087] in, This represents the codebook loss, used to push the codebook closer to the encoder output; This represents the committed loss, used to constrain the encoder output from frequently deviating from the selected discrete codeword; and These represent the weights of the codebook loss and the commitment loss, respectively. The role of vector quantization loss is to maintain the stability of the latent space quantization process, so that the latent representation is both discriminative and can be stably used by subsequent networks.

[0088] Building on this, this embodiment further introduces perceptual loss and 3D structural similarity (SSIM) loss to enhance the model's ability to recover local structures.

[0089] Perceptual loss measures the semantic shift between the reconstructed result and the real sample by comparing differences in the high-level feature space. It can be represented as:

[0090] in, This represents the perceptual feature comparison function. Unlike voxel-level error, the perceptual loss term is more sensitive to combinations of edges, textures, and local structures, thus being more conducive to preserving details in lesion areas.

[0091] The 3D structural similarity (SSIM) loss is used to further constrain the consistency of the reconstruction results in local brightness, contrast, and structural relationships. 3D structural similarity can be expressed as:

[0092] in, This represents the similarity of three-dimensional structures. Since a larger SSIM value indicates greater structural similarity, it is used... It can be transformed into a minimizeable loss term.

[0093] By combining reconstruction loss, vector quantization loss, perceptual loss, and 3D structural similarity loss, a second joint loss function can be obtained. , as the joint reconstruction target of the three-dimensional vector quantization autoencoder.

[0094]

[0095] in, , and represents the weights of vector quantization loss, perceptual loss, and 3D structural similarity loss, respectively. By combining these losses, the 3D vector quantization autoencoder is no longer limited to grayscale similarity, but simultaneously focuses on the structural consistency and perceptual realism of the volume data.

[0096] As a further implementation, the model also incorporates a 3D discriminator to improve the realism of the reconstruction results in high-frequency details. For the discriminator, this embodiment uses Hinge Loss as the optimization objective. Let the discriminator's outputs for real samples and generated samples be respectively... and The discriminator loses It can be represented as:

[0097] in, This represents the mathematical expectation. The discriminator loss can cause the discriminator to tend to increase the score of real samples and decrease the score of generated samples, thereby providing the generator with a more discriminative feedback signal.

[0098] For the generator, the adversarial target does not directly employ the above form, but rather aims to generate samples that elicit higher discriminator responses. Therefore, the generator's adversarial loss... Defined as:

[0099] The role of adversarial loss is to make the reconstruction results closer to real medical body data in terms of local texture and visual details, compensating for the shortcomings of simple reconstruction loss which tends to produce smooth results. Considering that adversarial training on small sample data from hospitals can easily introduce instability, this embodiment sets the second joint loss function to include adversarial loss, and the adversarial loss adopts a delayed activation strategy to control the timing of its activation. Specifically, when the number of training epochs is less than a set threshold, only reconstruction loss, vector quantization loss, perceptual loss, and 3D structural similarity loss are used for optimization. When the number of training epochs is greater than or equal to the set threshold, adversarial loss is combined with reconstruction loss, vector quantization loss, perceptual loss, and 3D structural similarity loss for optimization. Among them, the adversarial loss is configured with adaptive adversarial weights, which are calculated based on the ratio of the gradient norm of the reconstruction target and the adversarial target on the last layer of the decoder, and after being smoothed and amplified by a warm-up strategy, they are used to perform weighted optimization of the adversarial loss.

[0100] The final second joint loss function It can be represented as:

[0101] in, This represents the adaptive adversarial weights obtained through adaptive calculation.

[0102] Through the above design, the 3D vector quantization autoencoder can prioritize learning stable volume data reconstruction and latent space quantization in the early stages of training, and then gradually supplement the constraints on local realism in the later stages of training, thereby reducing the disruption of the overall convergence process by adversarial learning.

[0103] The present invention has the following beneficial effects: (1) By introducing edge-preserving enhancement processing and a specific three-dimensional vector quantization autoencoder fine-tuning strategy, the small lesions are effectively prevented from being destroyed or over-compressed during the data augmentation and encoding stages, and the latent representation’s ability to preserve lesion boundaries and local textures is significantly enhanced.

[0104] Specifically, by employing total variational denoising, center pruning or symmetrical filling, and single-axis random halving pruning and axial rotation strategies, the system controls the GPU memory overhead of 3D training while avoiding the destruction of lesion geometry and multi-dimensional simultaneous dilution caused by strong perturbations. By freezing the encoder parameters after loading pre-trained weights and updating the parameters only for the decoder and quantized codebook, the system prioritizes the learning capacity for local detail restoration while preserving the basic anatomical structure representation capability. Furthermore, by combining perceptual loss and 3D structural similarity loss, and using an adversarial loss delayed activation and warm-up strategy based on the gradient norm ratio of the reconstructed target and adversarial target at the last layer of the decoder, the system effectively avoids the instability caused by prematurely introducing adversarial training on small sample data. This forces the model to remain highly sensitive to local edges and textures, overcoming the problem of dilution of lesion feature gradients by large areas of normal tissue.

[0105] (2) By using the first joint loss function consisting of the noise prediction loss and the latent variable L1 reconstruction loss weighted by the signal-to-noise ratio to train the three-dimensional U-Net diffusion network, the structural fidelity and semantic stability of the conditional latent spatial diffusion model in the lesion area are significantly improved.

[0106] Specifically, the signal-to-noise ratio (SNR) weighted noise prediction loss weakens the excessive dominance of high SNR time steps on the overall training, enabling the 3D U-Net diffusion network to obtain more balanced learning signals at different diffusion stages and improving training stability. The latent variable L1 reconstruction loss recovers the initial latent variables predicted by the model according to the diffusion inversion formula and applies L1 constraints, directly constraining the generated results from the semantic level of the latent space. This effectively reduces the structural drift that occurs in the model during the reverse denoising process, making the recovered target MRI latent variables closer to the real target in terms of morphology and grayscale in the lesion neighborhood.

[0107] (3) In the reasoning stage, the conditional prior initialization and skip sampling strategy are adopted, which greatly improves the reasoning efficiency of three-dimensional volume data while ensuring the global consistency of the generated results and the positional stability of the lesion area.

[0108] Specifically, noisy latent variables are obtained by conditionally initializing the latent variables of the CT to be transformed with superimposed random perturbations. This ensures that the inverse denoising process starts near the prior, consistent with the spatial layout of the input CT, reducing the risk of overall anatomical structure drift. During the inverse denoising process in the latent space, it is continuously guided by the latent variables of the CT to be transformed, ensuring that the denoising prediction at each time step is always constrained by the prior of the CT structure, improving the positional consistency of the lesion region. Simultaneously, a skip-sampling strategy is adopted to directly jump to an earlier target time, bypassing some intermediate time points. This significantly reduces the number of network calls to the 3D U-Net diffusion network, lowers the inference computation overhead of the 3D volume data, and achieves a reasonable balance between generation quality and inference efficiency.

[0109] According to a further embodiment of the present invention, an electronic device is also provided. Figure 9 This diagram illustrates a schematic block diagram of an electronic device employing a processor-based hardware implementation according to an embodiment of the present invention. The hardware structure of the electronic device of the present invention can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc. Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one connecting line is used in this diagram, but this does not indicate that there is only one bus or one type of bus. The memory 1300 stores a computer program. When the processor 1200 executes the computer program, the processor 1200 is able to execute the CT to MRI volume data cross-modal conversion method of the above embodiments of the present invention.

[0110] The present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection (electronic device) having one or more wires, a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0111] This invention also provides a computer program product. The method of this invention can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, the processes or functions of this invention are performed wholly or partially.

[0112] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0113] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0115] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0116] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0117] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0118] Those skilled in the art should understand that the above embodiments are merely for illustrating the present invention and are not intended to limit the scope of the invention. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present invention.

Claims

1. A method for cross-modal conversion of CT to MRI volumetric data, characterized in that, include: Acquire the CT volume data to be converted and input it into the encoder in the three-dimensional vector quantization autoencoder for mapping to obtain the latent variables of the CT to be converted; Conditional prior initialization is performed on the superimposed random perturbation of the CT latent variables to be converted to obtain noisy latent variables; The noisy latent variables and the CT latent variables to be converted are input into the conditional latent space diffusion model obtained based on the three-dimensional U-Net diffusion network. Under the guidance of the CT latent variables to be converted, a skip sampling strategy is used to perform reverse denoising in the latent space constructed by the three-dimensional vector quantization autoencoder to obtain the target MRI latent variables. The latent variables of the target MRI are input into the decoder of the three-dimensional vector quantization autoencoder for decoding and restoration, so as to obtain the target MRI volume data that preserves the lesion boundary and texture details.

2. The cross-modal conversion method for CT to MRI volumetric data as described in claim 1, characterized in that, The conditional latent spatial diffusion model is obtained through the following steps: Acquire paired raw CT volume data and raw MRI volume data and perform standardization processing to obtain standard CT volume data and standard MRI volume data; The standard CT volume data and standard MRI volume data are respectively input into the encoder of the three-dimensional vector quantization autoencoder for mapping to obtain CT latent variables and MRI latent variables. Using the CT latent variables as conditional inputs and the MRI latent variables as denoising targets, a three-dimensional U-Net diffusion network is trained, and the trained three-dimensional U-Net diffusion network is used as a conditional latent spatial diffusion model.

3. The cross-modal conversion method for CT to MRI volumetric data as described in claim 2, characterized in that, The loss function used when training the 3D U-Net diffusion network is the first joint loss function consisting of the noise prediction loss based on the signal-to-noise ratio and the latent variable L1 reconstruction loss.

4. The cross-modal conversion method for CT to MRI volumetric data as described in claim 1, characterized in that, Before acquiring the CT volume data to be converted and inputting it into the encoder of the 3D vector quantization autoencoder for mapping, the process also includes: fine-tuning the 3D vector quantization autoencoder, including: A 3D vector quantization autoencoder with pre-trained weights is loaded, and its encoder parameters are frozen. Parameter updates are performed only on the decoder and quantization codebook of the 3D vector quantization autoencoder. The loss function used for updating the parameters of the decoder and quantization codebook is a second joint loss function composed of reconstruction loss, vector quantization loss, perceptual loss, and 3D structural similarity loss. The reconstruction loss is represented by the mean square error. The vector quantization loss is obtained by weighted summation of codebook loss and commitment loss. Codebook loss is used to push the codebook closer to the encoder output, and commitment loss is used to constrain the encoder output from frequently deviating from the selected discrete codeword. Perceptual loss measures the semantic shift between the reconstructed result and the real sample by comparing differences in the high-level feature space; Three-dimensional structural similarity loss is used to constrain the consistency between the reconstruction results and the real samples in terms of local brightness, contrast, and structural relationships.

5. The cross-modal conversion method for CT to MRI volumetric data as described in claim 4, characterized in that, The second joint loss function also includes adversarial loss, wherein the adversarial loss employs a delayed activation strategy, specifically including: When the number of training epochs is less than a set threshold, optimization is performed using only reconstruction loss, vector quantization loss, perceptual loss, and 3D structural similarity loss. When the number of training rounds is greater than or equal to a set threshold, adversarial loss is combined with the reconstruction loss, vector quantization loss, perceptual loss, and 3D structural similarity loss for optimization. The adversarial loss is configured with adaptive adversarial weights, which are calculated based on the ratio of the gradient norm of the reconstruction target and the adversarial target at the last layer of the decoder. After being smoothed and amplified by a preheating strategy, the adaptive adversarial loss is used to perform weighted optimization.

6. The method for cross-modal conversion of CT to MRI volumetric data as described in claim 2, characterized in that, Acquire paired raw CT volume data and raw MRI volume data and perform standardization processing to obtain standard CT volume data and standard MRI volume data, including: The original CT volume data and the original MRI volume data are uniformly converted into a preset format, and the voxel spacing is resampled to a preset physical scale to obtain the first CT volume data and the first MRI volume data. Bias field correction is performed on the first MRI volume data to obtain the second MRI volume data; The SyNRA registration method based on ANTs was used to align the first CT volume data to the MRI space to obtain the second CT volume data; The invalid regions in the second CT volume data are replaced and the window width and window level are limited to obtain the third CT volume data; Normalization is used to map the third CT volume data and the second MRI volume data to a preset intensity range, resulting in standardized CT volume data and standardized MRI volume data.

7. The cross-modal conversion method for CT to MRI volumetric data as described in claim 6, characterized in that, Also includes: During the acquisition of paired raw CT volume data and raw MRI volume data, total variational denoising is performed on the acquired raw CT volume data and raw MRI volume data. During the standardization process, a center-cropping or symmetrical filling strategy is used to adjust the original CT volume data and the original MRI volume data to a fixed size; During the training of the 3D U-Net diffusion network, a single-axis random halving and axis rotation strategy is adopted. In each training session, a spatial dimension is randomly truncated, and the truncated axis is remapped to the depth direction.

8. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the CT-to-MRI volumetric data cross-modal conversion method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the CT-to-MRI volumetric data cross-modal conversion method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the CT to MRI volumetric data cross-modal conversion method according to any one of claims 1 to 7.

Images