PET-MR combined reconstruction method and system based on expert network modular regularizer

Abstract

The application provides a PET-MR joint reconstruction method and system based on an expert network modular regularizer, comprising: constructing a PET-MR joint reconstruction main network based on a decoupled iterative framework; constructing and integrating a U-Net joint regularizer which is modularized in an expert network, and integrating the U-Net joint regularizer into the iterative reconstruction framework of step S1 to obtain a final joint reconstruction network model; applying a comprehensive loss function to supervise an end-to-end learning process of the joint reconstruction network model; constructing a training data set, and using the loss function to perform end-to-end joint training on the joint reconstruction network model; in an inference stage, inputting PET and MR measurement data to be reconstructed into the trained joint reconstruction network model, and finally outputting reconstructed PET and MR images. The application enhances the functional metabolic information accuracy of the reconstructed PET image and the anatomical structure detail definition of the MR image, ensures the high feature fidelity of the images of various modalities, and thus improves the overall quality of the reconstructed images and the clinical diagnostic value.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to a method and system for joint reconstruction of positron emission tomography (PET) and magnetic resonance (MR) images based on an expert network modular regularizer. Background Technology

[0002] In recent years, deep learning technology has made significant progress in the field of medical image reconstruction due to its powerful nonlinear modeling capabilities, especially in the joint reconstruction of multimodal medical images (such as positron emission tomography (PET) and magnetic resonance imaging (MR)). PET / MR joint imaging systems can simultaneously provide highly sensitive functional metabolic information (from PET) and high-resolution anatomical structural information (from MR). The effective fusion and synergistic reconstruction of these two information is crucial for improving the accuracy and reliability of disease diagnosis. Therefore, developing advanced PET-MR joint reconstruction algorithms to fully utilize complementary information and improve image quality has become a hot topic in current medical imaging technology research.

[0003] PET and MR images differ significantly in imaging principles, noise characteristics, spatial resolution, and the biological information they reflect. PET images typically have lower resolution and higher noise levels, but can reflect physiological function and metabolic activity; while MR images offer excellent soft tissue contrast and high spatial resolution, providing detailed anatomical structures. Traditional independent reconstruction methods often struggle to fully utilize the prior information provided by the other modality. Joint reconstruction aims to overcome the limitations of each modality by sharing information or through collaborative constraints. For example, it can utilize structural information from MR images to guide noise suppression and edge preservation in PET images, or leverage functional information from PET images to assist in the analysis of specific regions in MR images. The key to achieving this goal lies in designing an effective mechanism to fuse these two heterogeneous datasets.

[0004] In current joint reconstruction methods, the "unrolled network" structure based on model iterative optimization has attracted much attention due to its good interpretability and performance. These methods typically unfold the iterative steps of traditional iterative reconstruction algorithms (such as the MAPEM algorithm for PET and the Landweber algorithm for MR) into network layers and introduce learnable parameters or modules, particularly using neural networks (such as the U-Net structure) as regularization terms (i.e., "joint priors") to learn complex image prior knowledge. However, some existing unrolled network frameworks, while employing networks like U-Net as joint regularizers, often have a single U-Net simultaneously responsible for learning and imposing regularization constraints for both PET and MR modalities. This "one-size-fits-all" shared regularization strategy, while simplifying model design, easily leads to the confusion and loss of modality-specific features. For example, fine structural texture features in MR images may be inappropriately introduced into the inherently smoother PET images, or the noise patterns in PET images may in turn affect the regularization process of MR images, ultimately "inevitably introducing MR features into PET, failing to maintain modality-specific features well," and vice versa, thereby impairing the fidelity of each modality image and the accuracy of clinical interpretation.

[0005] To achieve better joint reconstruction results, it is essential to fully recognize the unique physical characteristics and information content of PET and MR images. While cross-modal information sharing is a core advantage of joint reconstruction, the core challenge currently facing joint reconstruction technology is how to effectively utilize complementary information between modalities within a unified framework, while accurately preserving or even enhancing the key features of each modality and avoiding improper or "negative" information transfer. An ideal joint regularizer should possess modality-aware capabilities, able to apply the most appropriate constraints based on the data characteristics of different modalities. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a PET-MR joint reconstruction method and system based on an expert network modular regularizer.

[0007] A PET-MR joint reconstruction method based on an expert network modular regularizer, provided by the present invention, includes:

[0008] Step S1: Construct a PET-MR joint reconstruction backbone network based on a decoupled iterative framework;

[0009] Step S2: Construct and integrate the modular U-Net joint regularizer of the expert network, and integrate it into the iterative reconstruction framework described in Step S1 to obtain the final joint reconstruction network model;

[0010] Step S3: Apply the integrated loss function to supervise the end-to-end learning process of the joint reconstruction network model;

[0011] Step S4: Construct a training dataset and perform end-to-end joint training on the joint reconstruction network model using the loss function;

[0012] Step S5: In the inference phase, the PET and MR measurement data to be reconstructed are input into the trained joint reconstruction network model, and the reconstructed PET and MR images are finally output.

[0013] Preferably, step S1 includes a PET data fidelity update step and an MR data fidelity update step;

[0014] The PET data fidelity update step includes updating the PET image by executing a PET reconstruction algorithm with one or more sub-iterations based on the current PET image estimate, the acquired raw PET measurement data, and the PET system model.

[0015] The MR data fidelity update step runs in parallel with the PET data fidelity update step, and includes updating the MR image by executing an MR reconstruction algorithm with one or more sub-iterations based on the estimation of the current MR image and the acquired MR k-space data.

[0016] Preferably, the U-Net joint regularizer adopts the classic U-Net structure, including an encoder, a decoder, and skip connections connecting corresponding layers, for fusing multi-scale features;

[0017] The encoder includes convolutional layers, normalization, and the Leaky ReLU activation function;

[0018] The decoder is symmetrical to the encoder and gradually restores spatial resolution by combining transposed convolution with convolutional layers;

[0019] The U-Net joint regularizer also includes an expert network module; the expert network module is used to implement differentiated processing, and to perform independent feature extraction on the PET feature channel and MR feature channel at different levels, and learn modality-specific attention weights or affine transformation parameters.

[0020] Preferably, the expert network module includes a fusion prompting block (FPGB) and an expert regularization block (EFB).

[0021] The fusion cue block FPGB adaptively extracts beneficial joint information from the feature maps obtained from each layer of U-Net to generate fusion cue as a guiding signal. The cue captures complementary structural and functional features across modalities. The fusion cue is then combined with the current feature map, and the combination result is then input into a series of parallel expert network modules for processing.

[0022] The expert regularization block EFB uses prompts to select experts or weight expert branches that are more suitable for the current task and modality, thereby performing differential processing on features.

[0023] Preferably, the loss function includes a core fidelity loss term and an expert network balance regularization term;

[0024] The core fidelity loss term includes calculating the difference between the reconstructed PET image and the gold standard high-quality PET reference image, as well as the corresponding difference between the reconstructed MR image and the gold standard high-quality MR reference image.

[0025] The expert network balance regularization term is used to optimize the performance of the expert network module and prevent static routing. This is achieved by calculating and minimizing the squared variation coefficient, which includes the total frequency of each expert being selected or activated throughout the training batch and the actual number of samples processed by each expert. The calculation formula is as follows:

[0026] L Balance =CV 2 (importance)+CV 2 (load)

[0027] Where importance represents the total frequency at which each expert is selected or activated throughout the entire training batch, and load represents the actual number of samples processed by each expert;

[0028] The final loss function formula is as follows:

[0029] L = L mae +γL balance

[0030] Among them, L mae L represents the core fidelity loss term. balance This represents the expert network balance regularization term, where γ is a weighting parameter used to balance the relative contributions of fidelity loss and expert module balance loss.

[0031] Preferably, the training dataset includes multiple sets of paired raw PET measurement data, raw MR k-space data, and strictly corresponding gold standard high-quality PET reference images and MR reference images.

[0032] A PET-MR joint reconstruction system based on an expert network modular regularizer provided by the present invention includes:

[0033] Module M1: Constructing a PET-MR joint reconstruction backbone network based on a decoupled iterative framework;

[0034] Module M2: Constructs and integrates the modular U-Net joint regularizer of the expert network, and integrates it into the iterative reconstruction framework described in Module M1 to obtain the final joint reconstruction network model;

[0035] Module M3: Applies a comprehensive loss function to supervise the end-to-end learning process of the joint reconstruction network model;

[0036] Module M4: Constructs the training dataset and performs end-to-end joint training on the joint reconstruction network model using the loss function;

[0037] Module M5: During the inference phase, the PET and MR measurement data to be reconstructed are input into the trained joint reconstruction network model, and the reconstructed PET and MR images are finally output.

[0038] Preferably, the module M1 includes a PET data fidelity update module and an MR data fidelity update module;

[0039] The PET data fidelity update module includes updating the PET image by executing a PET reconstruction algorithm with one or more sub-iterations based on the current PET image estimation, the acquired raw PET measurement data, and the PET system model.

[0040] The MR data fidelity update module runs in parallel with the PET data fidelity update module. It includes updating the MR image by executing an MR reconstruction algorithm with one or more sub-iterations based on the estimation of the current MR image and the acquired MR k-space data.

[0041] Preferably, the expert network module includes a fusion prompting block (FPGB) and an expert regularization block (EFB).

[0042] The fusion cue block FPGB adaptively extracts beneficial joint information from the feature maps obtained from each layer of U-Net to generate fusion cue as a guiding signal. The cue captures complementary structural and functional features across modalities. The fusion cue is then combined with the current feature map, and the combination result is then input into a series of parallel expert network modules for processing.

[0043] The expert regularization block EFB uses prompts to select experts or weight expert branches that are more suitable for the current task and modality, thereby performing differential processing on features.

[0044] Preferably, the loss function includes a core fidelity loss term and an expert network balance regularization term;

[0045] The core fidelity loss term includes calculating the difference between the reconstructed PET image and the gold standard high-quality PET reference image, as well as the corresponding difference between the reconstructed MR image and the gold standard high-quality MR reference image.

[0046] The expert network balance regularization term is used to optimize the performance of the expert network module and prevent static routing. This is achieved by calculating and minimizing the squared variation coefficient, which includes the total frequency of each expert being selected or activated throughout the training batch and the actual number of samples processed by each expert. The calculation formula is as follows:

[0047] L Balance =CV 2 (importance)+CV 2 (load)

[0048] Where importance represents the total frequency at which each expert is selected or activated throughout the entire training batch, and load represents the actual number of samples processed by each expert;

[0049] The final loss function formula is as follows:

[0050] L = L mae +γL balance

[0051] Among them, L mae L represents the core fidelity loss term. balance This represents the expert network balance regularization term, where γ is a weighting parameter used to balance the relative contributions of fidelity loss and expert module balance loss.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. The regularizer in this invention introduces parallel, parameter-independent expert network modules within the U-Net structure, tailored to the data characteristics of the PET and MR channels. This allows for the application of differentiated regularization constraints to features of different modalities, effectively avoiding the problem of modality-specific feature fusion or even loss that can easily occur with a single shared regularization network during joint reconstruction. This significantly enhances the accuracy of functional metabolic information in reconstructed PET images and the clarity of anatomical structural details in MR images, ensuring high feature fidelity across all modalities, thereby improving the overall quality and clinical diagnostic value of the reconstructed images.

[0054] 2. This invention employs a PET-MR joint reconstruction backbone network based on a decoupled iterative framework, deeply integrating traditional PET reconstruction algorithms (such as MAPEM) and MR reconstruction algorithms (such as Landweber) with a deep learning regularizer. Through a data fidelity update module, the physical consistency between the reconstructed image and the original measurement data (PET sinogram and MR k-space data) is ensured, thereby guaranteeing the physical rationality and reliability of the reconstruction results.

[0055] 3. This invention combines the powerful image prior knowledge learned by the modular regularizer of the expert network, which can efficiently suppress noise and artifacts. It achieves high reconstruction quality while maintaining the structural clarity and signal-to-noise ratio of each modality of the image, providing a reliable image foundation for accurate diagnosis.

[0056] 4. The end-to-end PET-MR joint reconstruction method proposed in this invention can efficiently acquire high-quality PET and MR images while significantly reducing the dose of radioactive tracer (achieving low-dose PET examinations). The output images possess advantages such as clear structure, rich detail, strong physical consistency, and high modal feature fidelity, overcoming the challenge of image quality degradation under low-dose conditions in traditional methods. This provides a highly efficient, low-risk, and highly reliable innovative solution for clinical low-dose PET examinations and multimodal image fusion analysis, with broad clinical application prospects and significant promotional value. Attached Figure Description

[0057] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0058] Figure 1 This is a schematic diagram of the working method of the present invention;

[0059] Figure 2 This is a schematic diagram of the network structure of the U-Net joint regularizer with integrated expert network modules proposed in this invention;

[0060] Figure 3 This is a diagram illustrating the PET-MR combined reconstruction results of one embodiment of the present invention;

[0061] Figure 4 This is a diagram illustrating the PET-MR combined reconstruction results of another embodiment of the present invention. Detailed Implementation

[0062] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0063] Advances in deep learning have demonstrated that modular and specialized network designs often lead to performance improvements. For example, in multi-task learning or processing heterogeneous data, designing specialized processing units or parameter subsets for different tasks or data types allows the network to learn and express specific information more effectively. This "divide and conquer" strategy allows different parts of the network to focus on their strengths while collaborating as a whole through subsequent feature fusion or parameter sharing mechanisms. Therefore, this invention introduces modality-specific processing mechanisms into the U-Net regularizer for PET-MR joint reconstruction, which is an effective way to solve the aforementioned feature confusion problem. By designing "Mixture-of-ExpertsModules," dedicated or at least partially parameter-independent neural computation pathways are customized for PET and MR data streams within the shared U-Net framework, enabling the regularization network to learn more discriminative prior knowledge that closely matches the intrinsic characteristics of each modality. These expert modules can be designed as parallel branches, processing the feature maps of their respective modalities and generating or adjusting the regularization strength or pattern for that modality, before being integrated in some way to jointly guide the reconstruction process. This U-Net regularizer, optimized with an expert network module, is embedded into an iterative PET-MR joint reconstruction pipeline, for example, by applying this regularization step after each MAPEM iteration updates the PET image and each Landweber iteration updates the MR image. Because the regularizer acts more precisely on each modality, it more effectively promotes the recovery of functional information from the PET image and suppresses noise, while simultaneously assisting in the reconstruction of structural details in the MR image and maintaining its high resolution, without introducing artifacts or blurring due to inappropriate feature sharing. This design promises to leverage the advantages of joint information while maximizing the diagnostic value of both PET and MR images, thereby significantly improving the overall quality and clinical applicability of joint reconstruction.

[0064] Example 1

[0065] According to the present invention, a PET-MR joint reconstruction method based on an expert network modular regularizer is provided, such as... Figure 1As shown, the decoupled iterative framework adopted includes inputting PET and MR k-space data, iteratively updating data fidelity, applying modality-specific regularization through a deep learning-driven expert network modular regularizer, and finally outputting high-quality PET and MR images. The specific process includes the following steps:

[0066] Step S1: Construct a PET-MR joint reconstruction backbone network based on a decoupled iterative framework. The network employs the MAPEM algorithm for PET reconstruction and the Landweber algorithm for MR reconstruction, and performs iterative optimization. In each iteration, the backbone network is responsible for executing the reconstruction algorithm for each modality based on the current image estimate and the original measurement data, and generating a preliminary image estimate for the next iteration. This step provides the basis for iterative optimization of the subsequent expert network modular regularizer.

[0067] Step S1 includes a PET data fidelity update step and an MR data fidelity update step. The PET data fidelity update step involves updating the PET image based on the current PET image estimate, the acquired raw PET measurement data (sinogram), and the PET system model by executing a traditional PET reconstruction algorithm, such as the Expectation-Maximization (MAPEM) algorithm, with one or more sub-iterations. Its core purpose is to ensure a high degree of consistency between the reconstructed PET image and the physical measurement data, providing a starting point based on accurate measurement data for subsequent regularization processing. The MR data fidelity update step runs parallel to the PET data fidelity update step. It involves updating the MR image based on the current MR image estimate and the acquired MR k-space data by executing an MR reconstruction algorithm, such as the Landweber algorithm, with one or more sub-iterations. Step S1 ensures a high degree of consistency between the reconstructed MR image and the original k-space measurement data, providing an accurate MR image estimate for subsequent regularization processing.

[0068] Step S2: Construct and integrate the modular U-Net joint regularizer of expert networks, and integrate it into the iterative reconstruction framework described in Step S1 to obtain the final joint reconstruction network model. The U-Net regularizer internally designs parallel or partially parameter-independent expert network modules for the PET and MR data channels respectively. In each iteration, the current PET image estimate and MR image estimate are fed into the U-Net joint regularizer as input. The expert network module is responsible for in-depth analysis and processing of the feature information of the corresponding modality, and generating or modulating specific regularization terms or gradients for the modality, thereby achieving refined and differentiated regularization constraints for different modalities, effectively preventing the fusion or loss of modality-specific features.

[0069] The U-Net joint regularizer adopts the classic U-Net structure, including an encoder (downsampling path), a decoder (upsampling path), and skip connections connecting corresponding layers to effectively fuse multi-scale features. The encoder includes convolutional layers, normalization, and the Leaky ReLU activation function. The decoder is symmetrical to the encoder, gradually restoring spatial resolution through transposed convolutions combined with convolutional layers.

[0070] The U-Net joint regularizer also includes an expert network module, which performs differentiated processing by using a gating mechanism to independently extract features from the PET and MR feature channels at different levels and learn modality-specific attention weights or affine transformation parameters. The expert network module includes a Fusion Cueing Block (FPGB) and an Expert Regularization Block (EFB). The FPGB extracts modality-specific cues from the input PET and MR channels. Specifically, based on the feature maps obtained from each layer of the U-Net, it adaptively extracts beneficial joint information to generate a fusion prompt as a guidance signal. This prompt captures complementary structural and functional features across modalities. The fusion prompt is then combined with the current feature map, and the combined result is input into a series of parallel expert network modules for processing. The EFB uses the prompt to select expert branches or weights expert branches that are more suitable for the current task and modality, thereby performing differentiated feature processing.

[0071] Each expert is essentially a multilayer perceptron capable of learning specific nonlinear transformations. The gating mechanism dynamically calculates the importance of each expert based on input features and fusion cues, selecting the top n experts to process the current feature information. The input features are then precisely assigned to these selected experts for parallel computation, and their outputs are weighted and combined according to the gating weights. This dynamic expert selection and processing mechanism allows the network to adaptively select the most suitable expert based on the characteristics of local image regions (e.g., whether it favors PET functional information or MR anatomical information), thereby achieving refined intermodal differentiation processing in the spatial dimension.

[0072] Step S3: Apply a comprehensive loss function to supervise the end-to-end learning process of the joint reconstruction network model. The loss function includes a core fidelity loss term and an expert network balance regularization term.

[0073] The core fidelity loss term quantifies the difference between the model reconstruction result and the high-quality reference image, ensuring the accuracy of the reconstructed image. Specifically, it calculates the difference between the reconstructed PET image and the gold-standard high-quality PET reference image, as well as the corresponding difference between the reconstructed MR image and the gold-standard high-quality MR reference image. This invention uses the mean absolute error (MAE) metric.

[0074] The expert network balance regularization term is used to optimize the performance of expert network modules and prevent the static routing problem, where a minority of expert modules are always selected. This is achieved by calculating and minimizing the coefficient of variation squared (CV2), which includes the total frequency of each expert being selected or activated throughout the training batch and the actual number of samples processed by each expert. The calculation formula is as follows:

[0075] L Balance =CV 2 (importance)+CV 2 (load)

[0076] Here, importance represents the total frequency at which each expert is selected or activated throughout the entire training batch, and load represents the actual number of samples processed by each expert.

[0077] The final loss function formula is as follows:

[0078] L = L mae +γL balance

[0079] Among them, L mae L represents the core fidelity loss term. balance The term represents the expert network balancing regularization term, where γ is a weighting parameter used to balance the relative contributions of fidelity loss and expert module balancing loss. During training, different weighting factors can be set for the loss terms of PET and MR as needed, or their relative contributions can be dynamically adjusted to balance the reconstruction effects and convergence speed of the two modalities, thereby achieving optimal joint reconstruction performance.

[0080] Step S4: Construct a training dataset and perform end-to-end joint training of the joint reconstruction network model using the loss function. The training dataset includes multiple sets of paired raw PET measurement data, raw MR k-space data, and their strictly corresponding gold-standard high-quality PET and MR reference images. The goal of training is to optimize all learnable parameters in the network, including U-Net backbone network parameters, internal parameters of expert network modules, and possible hyperparameters such as regularization strength, so that the model can accurately recover high-quality PET and MR images with clear structure, good noise suppression, and excellent modal feature preservation from low-quality or incomplete input data.

[0081] Step S5: In the inference phase, the PET and MR measurement data to be reconstructed are input into the trained joint reconstruction network model. Through the combined action of iterative reconstruction and the modular regularizer of the expert network, the final output is a PET image and an MR image with high reconstruction quality and high modal feature fidelity. The model executes the reconstruction process according to a preset number of iterations. In each iteration, the estimation of the current PET and MR images is calculated through forward propagation, and then modality-specific regularization constraints are calculated and applied using the modular U-Net regularizer of the expert network to update the image estimates accordingly. After iteration, the model finally outputs high-quality PET images (functional metabolic information) and high-quality MR images (anatomical structure information) optimized by deep learning and regularized by experts. These images can better serve clinical diagnosis and scientific research analysis. Through the above steps, this invention effectively improves the structural accuracy and physical interpretability of PET images while reducing the dose of radioactive tracer, providing a solution for clinical low-dose PET examination that combines high efficiency, low risk, and high reliability.

[0082] This invention aims to address the problem in existing joint reconstruction methods where a single shared regularization network can easily lead to the fusion or even loss of modality-specific features. The goal is to achieve efficient PET-MR joint reconstruction while enhancing the feature fidelity of each modality's images. Specifically, when using a shared neural network as a regularizer, improper fusion and mutual interference of features from different modalities (such as structural information in MR and functional information in PET) can easily occur, resulting in the loss of unique modality features, blurred or distorted image details, and ultimately affecting the accuracy of clinical diagnosis.

[0083] In the regularization step of the iterative joint reconstruction framework, this invention introduces a U-Net structure with a built-in expert network module as a joint regularizer. These expert modules provide differentiated, modality-specific regularization processing tailored to the different data characteristics and regularization requirements of PET and MR. This effectively utilizes cross-modal complementary information while maximizing the preservation and enhancement of the inherent characteristics of each imaging modality, thereby improving the quality and fidelity of the final reconstructed image.

[0084] Example 2:

[0085] The following detailed description, in conjunction with the accompanying drawings, provides a specific embodiment of the PET-MR joint reconstruction method based on an expert network modular regularizer according to the present invention. Figure 1 This is an overall flowchart of the method of the present invention. Figure 2 This is a schematic diagram of the reconstruction framework used.

[0086] Steps S1 and S2: Model Structure Design. In this embodiment, the PET image denoising model adopts an end-to-end deep neural network architecture, which includes a multimodal cue generation module, a cue fusion module, a backbone denoising network, and a line integral projection constraint module. The specific settings of each module are as follows:

[0087] PET Data Consistency Update Module

[0088] This module is responsible for updating the PET image based on the current PET image estimate, the acquired raw PET measurement data (sinogram), and the PET system model, by executing the traditional PET maximum expectation algorithm MAPEM in one or several sub-iterations. Its core purpose is to ensure that the reconstructed PET image maintains a high degree of consistency with the physical measurement data, providing a starting point based on accurate measurement data for subsequent regularization processing.

[0089] MR Data Fidelity Update Module (Data Consistency for MRI)

[0090] Parallel to the PET data fidelity update module, this module is responsible for updating the MR image estimate in each iteration. It receives the MR image estimate for the current iteration (containing both real and imaginary channels) and the original MR k-space data. Internally, the module executes one or more steps of the Landweber algorithm, which can be represented as:

[0091] x k+1 =x k -αA T (Ax k -b)

[0092] Where, x k For the current MR image estimation, A is the MR forward model (including Fourier transform and sampling operations), A T Let α be its conjugate transpose (including zero-padding and inverse Fourier transform), b be the original MR k-space data, and α be the step size. This module also aims to ensure that the reconstructed MR image maintains high fidelity with the k-space data. Its output serves as the input to the MR channel in the U-Net joint regularizer.

[0093] The U-Net joint regularizer, integrating expert network modules, learns and applies modality-specific regularization priors. It receives intermediate PET image estimates (256x256, channel 1) from the PET data fidelity update module and intermediate MR image estimates (256x256, channel 2) from the MR data fidelity update module. Its basic architecture is the classic U-Net structure, comprising: an encoder (downsampling path): consisting of a series of convolutional layers, normalization, and Leaky ReLU activation functions. With each downsampling block (2x2 max pooling), the spatial resolution of the feature map is halved, and the number of channels is doubled. Input channel 2 (1 PET channel, 1 MR channel) is expanded to 64, then successively to 128, 256, and 512. The decoder (upsampling path): symmetric to the encoder, it progressively restores spatial resolution through transposed convolutions combined with convolutional layers. Skip Connections: Feature maps of the corresponding layers of the encoder are passed to the decoder by concatenation to preserve multi-scale feature information and help restore details.

[0094] The U-Net joint regularizer deeply embeds Expert Fusion Modules. These modules are specifically designed for the characteristics of PET and MR data streams, performing independent feature extraction on PET and MR feature channels at different levels and learning modality-specific attention weights or affine transformation parameters. The Expert Fusion Module mainly comprises two functions: First, the Fusion Cueing Block (FPGB) adaptively extracts beneficial joint information from the feature maps obtained from each layer of U-Net, generating "fusion prompts" as guiding signals. These prompts capture complementary structural and functional features across modalities. Subsequently, these generated fusion prompts are combined with the current feature map and fed into a series of parallel "expert models" for processing. Through this prompt-guided mechanism, the Expert Regularization Block (EFB) can intelligently select or weight expert branches more suitable for the current task and modality, thereby differentiating features and effectively avoiding the introduction of interference information or artifacts from other modalities during information fusion, ensuring the accurate preservation of modality-unique features. The processed features are then passed through a 1x1 convolutional layer to adjust the number of channels, and further through a 3x3 convolutional layer for local feature extraction, ultimately restoring the feature map to the same size as the input feature map for subsequent network layer processing. In this embodiment, each expert consists of two fully connected layers, an activation layer, and a dropout layer, with a total of four experts; only two experts are used for data processing simultaneously. Finally, the U-Net regularizer outputs regularization information for two channels: one channel corresponding to the regularization term for the PET image, and the other two channels corresponding to the regularization term for the MR image. These regularization terms will guide subsequent image updates to preserve modality-specific features and suppress noise.

[0095] The fusion update module is responsible for effectively combining the intermediate image estimates obtained from the PET and MR data fidelity update modules with the modality-specific regularization terms output by the U-Net joint regularizer of the integrated expert network module to generate the final PET and MR images for the current main iteration step. This module is a key step in the entire joint reconstruction framework for fusing data fidelity and learned prior knowledge (i.e., the regularizer output). It aims to accurately balance the contributions of both, thereby outputting high-quality PET and MR images with effectively suppressed noise, clearer structure, and well-preserved modal features.

[0096]

[0097] u k+1 =u k -α(u Reg +λu DC )

[0098] Where xk+1 s is the new estimate of the PET image after the current subset iteration is completed; x EM This is an image estimate from the PET data fidelity update module, representing an image update based solely on the original PET measurement data; x Reg This is a regularization term for the PET modal output, which incorporates the expert model U-net module. The parameter ν is used to balance x. EM and x Reg The relative contribution. This updated formula ensures that the PET images not only conform to the physical measurement data during the iteration process, but also incorporate modality-specific priors learned by deep learning.

[0099] Among them, u k+1 This represents the latest estimate of the MR image after the current subset iteration is completed. This estimate is in u k This is obtained by correcting the current estimate of the MR image. The correction amount consists of two parts: one part is u Reg This is also a regularization term for the MR modality output by the U-Net joint regularizer, which integrates expert network modules. It carries prior knowledge learned by the deep neural network that helps improve MR image quality and anatomical details; another part is u DC It comes directly from the MR data fidelity module, guiding the image to be corrected to better match the measurement data. The parameter α in the formula, acting as the learning rate or step size, controls the magnitude of each update; while λ (lambda) is a learnable weight parameter that dynamically balances the deep learning regularization term u. Reg and data consistency gradient u DC The relative contribution to the total correction.

[0100] Step S3: Loss Function Design. The comprehensive loss function used in this embodiment aims to supervise the end-to-end learning process of the entire PET-MR joint reconstruction network and consists of two core parts. First is the core fidelity loss term, which directly quantifies the difference between the model's reconstruction result and the high-quality reference image, ensuring the accuracy of the reconstructed image. Specifically, it calculates the mean absolute error (MAE) between the reconstructed PET image and the gold-standard high-quality PET reference image, as well as the MAE between the reconstructed MR image and the gold-standard high-quality MR reference image. This pixel-level difference metric effectively reflects the overall image reconstruction accuracy and, by averaging the absolute errors of all pixels, maintains insensitivity to outliers and provides good numerical stability. This suppresses large-scale systematic biases while maintaining the global structure and signal intensity distribution, ultimately improving the overall quality and reliability of the reconstructed image. During training, different weighting factors can be set for the MAE loss terms of PET and MR as needed, or their relative contributions can be dynamically adjusted to balance the reconstruction effects and convergence speed of the two modalities, thereby achieving optimal joint reconstruction performance.

[0101]

[0102] Secondly, to optimize the performance of the expert network modules and prevent the "static routing" problem (i.e., a few expert modules are always selected), this invention introduces an expert network balance regularization term L. balance This regularization term is achieved by calculating and minimizing the coefficient of squared variation (CV). 2 This is achieved through [the following]. Specifically, the model evaluates the variability of two key metrics: the total frequency with which each expert is selected or activated throughout the entire training batch, and the actual number of samples processed by each expert. When an expert is overused or ignored, the distribution of these frequencies or sample sizes becomes uneven, leading to an increase in the corresponding coefficient of squared variation. Therefore, L […]. balance It is defined as the sum of the squared coefficients of variation of these two items. This is achieved by backpropagation and minimizing this L. balace The network is strongly incentivized to ensure that all expert modules are selected and processed approximately evenly during training. This mechanism ensures that each expert module is fully trained and utilized, thereby improving the overall robustness and generalization ability of the model. It avoids the potential problem of relying solely on a few "elite" experts while neglecting other modules, ultimately enabling multimodal features to be processed more comprehensively and meticulously through all expert modules.

[0103] L Balance =CV 2 (importance)+CV2 (load)

[0104] Here, importance represents the total frequency at which each expert is selected or activated throughout the entire training batch, and load represents the actual number of samples processed by each expert.

[0105] The final loss function is:

[0106] L=Lmae+γL Balance

[0107] Here, γ is a weighting parameter used to balance the relative contributions of the two types of losses.

[0108] Step S4: This embodiment constructs multiple sets of paired PET-MR medical image datasets. Each set of samples in the dataset contains the original sinogram data of the PET scan and the corresponding PET reconstructed image, as well as the original k-space data of the MR scan and the corresponding MR reconstructed image. All samples are strictly paired and precisely registered. Furthermore, each set of samples must correspond to its gold-standard high-quality PET reference image and gold-standard high-quality MR reference image (high-resolution T1-weighted). These paired data ensure that the model can learn the accurate anatomical correspondences and complementary information between different modalities during joint reconstruction.

[0109] To construct a paired training dataset consisting of low-dose PET images, standard-dose PET images, and their corresponding MR images of the brain, the first PET image was represented using Standardized Uptake Value (SUV) to directly retain its original physical values ​​and maintain the medical interpretability of metabolic information. The MRI images were T1-weighted images of the brain, which are greatly affected by scanning parameters and equipment. To eliminate amplitude differences between images, a linear normalization method was used to scale their pixel values ​​to the [0,1] interval, facilitating subsequent multimodal fusion processing. Considering the diverse sources of the original image data and the inconsistencies in spatial resolution and image size, all images were adjusted to a standard size of 256×256 to ensure the uniformity of the model input. For larger images, a center-cropping method was used to retain the main anatomical structures and minimize the interference of background areas on model training.

[0110] To test the model's reconstruction capability, dose data was simulated during dataset construction. Specifically, for the original projection data corresponding to the gold standard high-quality PET images, a Poisson noise sampling method was used to simulate reduced radioactivity counts. The total count of the original projection data was directly reduced by a set scaling factor (0.1 × standard dose) to simulate the statistical noise characteristics under low-dose PET acquisition conditions.

[0111] Low-quality MR k-space data is achieved by simulating six-fold equidistant undersampling of the original high-resolution MR k-space data, retaining only one-sixth of the data volume in the non-central region of the original k-space. Simultaneously, Gaussian noise can be selectively added to simulate data acquisition under real-world conditions of rapid scanning or low signal-to-noise ratio.

[0112] Step S5: During the model training phase, to further improve the generalization ability and robustness of the model of this invention, various data augmentation strategies were adopted for the preprocessed multimodal medical image dataset. Specifically, in addition to the previously mentioned random horizontal / vertical flipping and random scaling, each input image was rotated at a random angle (between -15 degrees and +15 degrees) to simulate various orientation changes that may occur during actual clinical acquisition. The brightness and contrast of the images were also randomly adjusted to enhance the network's adaptability to data under different imaging conditions. Based on these data augmentations, the deep learning model was jointly trained using the preprocessed PET and MR multimodal medical image data. The entire training process consisted of 60 iterations (unrolled 60 rounds).

[0113] During training, the model calculates loss strictly according to the comprehensive loss function defined in step S3, based on the outputs of each relevant module (including the PET and MR data fidelity update module, and the U-Net joint regularizer integrating the expert network module). The core fidelity loss term primarily measures the pixel-level difference (using mean absolute error MAE) between the reconstructed PET image and the gold-standard PET reference image, as well as the corresponding pixel-level difference between the reconstructed MR image and the gold-standard MR reference image. Meanwhile, the implementation of the expert network balance regularization term depends on the settings of four expert modules. In each forward propagation, only the two best-performing expert modules (i.e., top-k = 2) are selected for processing. This regularization term is achieved by calculating and minimizing the coefficient of variation of the squared variation of the selection frequency (importance) and actual load of these expert modules. It aims to encourage all expert modules to be selected and utilized approximately uniformly during training. By minimizing this comprehensive loss function and using the backpropagation algorithm to continuously update all learnable parameters of the model (including U-Net backbone network parameters, internal parameters of expert network modules, and regularization strength parameters), the joint reconstruction capability and multimodal feature representation capability of the model are gradually improved.

[0114] To ensure efficiency and convergence during training, this embodiment uses the Adam optimizer for parameter optimization. The initial learning rate is set to 0.0002, and after every 10 iterations, the learning rate is reduced to 0.5 times its original value to ensure more stable convergence and better identification of the optimal solution in the later stages of training. During the 60 training iterations, performance is evaluated on both the training and validation sets after each iteration, with real-time monitoring of the loss function's trend and the model's reconstruction quality. Once the loss function on the validation set reaches the convergence criterion and stabilizes, the final trained deep learning model is obtained. This model possesses the ability to recover high-quality, high-feature-fidelity joint images from low-quality PET and MR measurement data.

[0115] Figure 3 The diagram illustrates the results of this embodiment, where the PET dose is 0.1 times the standard dose, and the MR K-space is undersampled six times at equal intervals. Alternatively, the input PET data is 0.2 times the standard dose data, and the MR K-space is undersampled five times at equal intervals; the results are illustrated below. Figure 4 As shown.

[0116] Example 3

[0117] The present invention also provides a PET-MR joint reconstruction system based on an expert network modular regularizer. The PET-MR joint reconstruction system based on an expert network modular regularizer can be implemented by executing the process steps of the PET-MR joint reconstruction method based on an expert network modular regularizer. That is, those skilled in the art can understand the PET-MR joint reconstruction method based on an expert network modular regularizer as a preferred embodiment of the PET-MR joint reconstruction system based on an expert network modular regularizer.

[0118] A PET-MR joint reconstruction system based on an expert network modular regularizer provided by the present invention includes:

[0119] Module M1: Constructs a PET-MR joint reconstruction backbone network based on a decoupled iterative framework; Module M1 includes a PET data fidelity update module and an MR data fidelity update module; The PET data fidelity update module includes updating the PET image by executing a PET reconstruction algorithm with one or several sub-iterations based on the estimation of the current PET image, the acquired raw PET measurement data, and the PET system model; The MR data fidelity update module runs in parallel with the PET data fidelity update module, including updating the MR image by executing an MR reconstruction algorithm with one or several sub-iterations based on the estimation of the current MR image and the acquired MR k-space data.

[0120] Module M2: Constructs and integrates the modular U-Net joint regularizer of the expert network, and integrates it into the iterative reconstruction framework described in Module M1 to obtain the final joint reconstruction network model. The expert network module includes a fusion cue block (FPGB) and an expert regularization block (EFB). The FPGB adaptively extracts useful joint information from the feature maps obtained from each layer of U-Net to generate fusion cue as a guiding signal. The cue captures complementary structural and functional features across modalities. Then, the fusion cue is combined with the current feature map, and the combination result is input into a series of parallel expert network modules for processing. The EFB uses the cue to select expert or weighted expert branches that are more suitable for the current task and modality, thereby performing differential processing on the features.

[0121] Module M3: Applies a comprehensive loss function to supervise the end-to-end learning process of the joint reconstruction network model; the loss function includes a core fidelity loss term and an expert network balance regularization term; the core fidelity loss term calculates the difference between the reconstructed PET image and the gold-standard high-quality PET reference image, and the corresponding difference between the reconstructed MR image and the gold-standard high-quality MR reference image; the expert network balance regularization term optimizes the performance of the expert network module and prevents static routing, achieved by calculating and minimizing the squared coefficient of variation, which includes the total frequency of each expert being selected or activated throughout the training batch and the actual number of samples processed by each expert, calculated as follows:

[0122] L Balance =CV 2 (importance)+CV 2 (load)

[0123] Where importance represents the total frequency at which each expert is selected or activated throughout the entire training batch, and load represents the actual number of samples processed by each expert;

[0124] The final loss function formula is as follows:

[0125] L = L mae +γL balance

[0126] Among them, L mae L represents the core fidelity loss term. balance This represents the expert network balance regularization term, where γ is a weighting parameter used to balance the relative contributions of fidelity loss and expert module balance loss.

[0127] Module M4: Constructs the training dataset and performs end-to-end joint training on the joint reconstruction network model using the loss function;

[0128] Module M5: During the inference phase, the PET and MR measurement data to be reconstructed are input into the trained joint reconstruction network model, and the reconstructed PET and MR images are finally output.

[0129] The core innovation of this invention is the U-Net Joint Regularizer (Expert Module Unet), which integrates an expert network module. This module is responsible for learning and applying a joint, modality-specific regularization prior to significantly improve the quality of PET and MR images and accurately ensure the features of their respective modalities. It receives intermediate PET image estimates from the PET data fidelity update module and intermediate MR image estimates from the MR data fidelity update module as its multi-channel inputs. The Fusion Update Module is responsible for effectively combining the intermediate images obtained from the data fidelity update module with the modality-specific regularization terms output by the U-Net Joint Regularizer to generate the final PET and MR images for the current main iteration step. Specifically, it executes specific update rules according to the overall optimization algorithm framework adopted (an expansion of the MAPEM and Landweber algorithms). Through these update rules and learnable regularization strength parameters, this module aims to precisely balance the contribution between data fidelity and regularization priors, ultimately outputting PET and MR images with effectively suppressed noise, clearer structure, and well-preserved modal features, as the result of the current iteration or the final reconstruction output.

[0130] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0131] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A PET-MR joint reconstruction method based on an expert network modular regularizer, characterized in that, include: Step S1: Construct a PET-MR joint reconstruction backbone network based on a decoupled iterative framework; Step S2: Construct and integrate the modular U-Net joint regularizer of the expert network, and integrate it into the iterative reconstruction framework described in Step S1 to obtain the final joint reconstruction network model; Step S3: Apply the integrated loss function to supervise the end-to-end learning process of the joint reconstruction network model; Step S4: Construct a training dataset and perform end-to-end joint training on the joint reconstruction network model using the loss function; Step S5: In the inference phase, the PET and MR measurement data to be reconstructed are input into the trained joint reconstruction network model, and the reconstructed PET and MR images are finally output. The U-Net joint regularizer adopts the classic U-Net structure, including an encoder, a decoder, and skip connections connecting corresponding layers, for fusing multi-scale features; The encoder includes convolutional layers, normalization, and the Leaky ReLU activation function; The decoder is symmetrical to the encoder and gradually restores spatial resolution by combining transposed convolution with convolutional layers; The U-Net joint regularizer also includes an expert network module; the expert network module is used to implement differentiated processing, and to perform independent feature extraction on the PET feature channel and MR feature channel at different levels, and learn modality-specific attention weights or affine transformation parameters. The expert network module includes a fusion prompting block FPGB and an expert regularization block EFB; The fusion cue block FPGB adaptively extracts beneficial joint information from the feature maps obtained from each layer of U-Net to generate fusion cue as a guiding signal. The cue captures complementary structural and functional features across modalities. The fusion cue is then combined with the current feature map, and the combination result is then input into a series of parallel expert network modules for processing. The expert regularization block EFB uses prompts to select experts or weight expert branches that are more suitable for the current task and modality, thereby performing differential processing on features.

2. The PET-MR joint reconstruction method based on expert network modular regularizer according to claim 1, characterized in that, Step S1 includes a PET data fidelity update step and an MR data fidelity update step; The PET data fidelity update step includes updating the PET image by executing a PET reconstruction algorithm with one or more sub-iterations based on the current PET image estimate, the acquired raw PET measurement data, and the PET system model. The MR data fidelity update step runs in parallel with the PET data fidelity update step, and includes updating the MR image by executing an MR reconstruction algorithm with one or more sub-iterations based on the estimation of the current MR image and the acquired MR k-space data.

3. The PET-MR joint reconstruction method based on expert network modular regularizer according to claim 1, characterized in that, The loss function includes a core fidelity loss term and an expert network balance regularization term. The core fidelity loss term includes calculating the difference between the reconstructed PET image and the gold standard high-quality PET reference image, as well as the corresponding difference between the reconstructed MR image and the gold standard high-quality MR reference image. The expert network balance regularization term is used to optimize the performance of the expert network module and prevent static routing. This is achieved by calculating and minimizing the squared variation coefficient, which includes the total frequency of each expert being selected or activated throughout the training batch and the actual number of samples processed by each expert. The calculation formula is as follows: in, This represents the total frequency with which each expert is selected or activated throughout the entire training batch. This indicates the actual number of samples processed by each expert; The final loss function L is formulated as follows: L=L mae +γL balance Among them, L mae L represents the core fidelity loss term. balance This represents the expert network balance regularization term, where γ is a weighting parameter used to balance the relative contributions of fidelity loss and expert module balance loss.

4. The PET-MR joint reconstruction method based on expert network modular regularizer according to claim 1, characterized in that, The training dataset includes multiple pairs of raw PET measurement data, raw MR k-space data, and corresponding gold-standard high-quality PET reference images and MR reference images.

5. A PET-MR joint reconstruction system based on an expert network modular regularizer, characterized in that, The PET-MR joint reconstruction method based on expert network modular regularizer as described in any one of claims 1-4, the system comprising: Module M1: Constructing a PET-MR joint reconstruction backbone network based on a decoupled iterative framework; Module M2: Constructs and integrates the modular U-Net joint regularizer of the expert network, and integrates it into the iterative reconstruction framework described in Module M1 to obtain the final joint reconstruction network model; Module M3: Applies a comprehensive loss function to supervise the end-to-end learning process of the joint reconstruction network model; Module M4: Constructs the training dataset and performs end-to-end joint training on the joint reconstruction network model using the loss function; Module M5: During the inference phase, the PET and MR measurement data to be reconstructed are input into the trained joint reconstruction network model, and the reconstructed PET and MR images are finally output.

6. The PET-MR joint reconstruction system based on expert network modular regularizer according to claim 5, characterized in that, The module M1 includes a PET data fidelity update module and an MR data fidelity update module; The PET data fidelity update module includes updating the PET image by executing a PET reconstruction algorithm with one or more sub-iterations based on the current PET image estimation, the acquired raw PET measurement data, and the PET system model. The MR data fidelity update module runs in parallel with the PET data fidelity update module. It includes updating the MR image by executing an MR reconstruction algorithm with one or more sub-iterations based on the estimation of the current MR image and the acquired MR k-space data.

7. The PET-MR joint reconstruction system based on expert network modular regularizer according to claim 5, characterized in that, The loss function includes a core fidelity loss term and an expert network balance regularization term. The core fidelity loss term includes calculating the difference between the reconstructed PET image and the gold standard high-quality PET reference image, as well as the corresponding difference between the reconstructed MR image and the gold standard high-quality MR reference image. The expert network balance regularization term is used to optimize the performance of the expert network module and prevent static routing. This is achieved by calculating and minimizing the squared variation coefficient, which includes the total frequency of each expert being selected or activated throughout the training batch and the actual number of samples processed by each expert. The calculation formula is as follows: in, This represents the total frequency with which each expert is selected or activated throughout the entire training batch. This indicates the actual number of samples processed by each expert; The final loss function formula is as follows: L=L mae +γL balance Among them, L mae L represents the core fidelity loss term. balance This represents the expert network balance regularization term, where γ is a weighting parameter used to balance the relative contributions of fidelity loss and expert module balance loss.

Images