Liver and gall period magnetic resonance image generation model and generation method

Abstract

The application discloses a liver and gallbladder phase magnetic resonance image generation model and a generation method. The model comprises a generator and multiple discriminators. The generator comprises multiple encoders, a fusion module and a decoder. Each encoder extracts image features of real modality images and liver and gallbladder phase images in different vascular phases respectively. The fusion module fuses the image features of the real modality images into fused image features. The decoder reconstructs modality images and real liver and gallbladder phase images in different vascular phases based on the fused image features and liver and gallbladder phase image features. In the model training process, the multiple discriminators respectively determine the authenticity probability of each modality reconstructed image and liver and gallbladder phase reconstructed image according to the real modality images, and optimize the module parameters of the corresponding modules in the generator. The real modality images of patients in different vascular phases are input into the trained model, the model fuses the modality image features in different vascular phases, synthesizes the liver and gallbladder phase magnetic resonance image, and improves the robustness and fidelity of the synthesis result.

Description

Model and method for generating hepatobiliary magnetic resonance images Technical Field

[0001] This invention relates to the field of image or video recognition technology, specifically to a model and method for generating hepatobiliary magnetic resonance images. Background Technology

[0002] Primary liver cancer (mainly hepatocellular carcinoma, HCC) is the sixth most common malignant tumor and the third leading cause of cancer-related deaths. Nearly half of HCC cases are caused by hepatitis B virus (HBV) infection, posing a significant challenge to clinical diagnosis and treatment.

[0003] Magnetic resonance imaging (MRI) is a key tool for the diagnosis of hepatocellular carcinoma. Gadoxetate disodium (Gd-EOB-DTPA)-enhanced MRI combines the properties of extracellular contrast agents with hepatocellular specificity, significantly improving lesion contrast in the hepatobiliary phase (HBP). Compared to extracellular contrast agents, hepatobiliary phase MRI images represent the core advantage of liver-specific contrast agents, providing not only functional information about the liver but also serving as a recognized non-invasive biomarker for diagnosing hepatobiliary diseases.

[0004] Despite the significant clinical value of hepatobiliary phase imaging, its routine acquisition faces numerous practical challenges. First, after contrast agent injection, hepatobiliary phase imaging requires an additional 20-minute wait to ensure adequate contrast agent uptake by hepatocytes, significantly prolonging the examination time and increasing patient discomfort. Furthermore, in high-throughput imaging centers, this extended waiting time further limits the clinical application and promotion of gadoxetate disodium. Second, patients with special conditions (such as those with impaired liver function or elevated bilirubin) often fail to achieve ideal hepatobiliary phase enhancement, resulting in image quality that is suboptimal or insufficient for diagnostic purposes. These issues highlight the urgent need for more efficient and standardized methods for acquiring hepatobiliary phase magnetic resonance imaging.

[0005] In recent years, deep learning-based hepatobiliary phase magnetic resonance imaging (MRI) image synthesis techniques have been considered a potential solution to overcome the aforementioned limitations. However, most existing methods heavily rely on single-modal imaging stages such as the transition phase, resulting in limited robustness. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a model and method for generating hepatobiliary phase magnetic resonance images. This model can fuse modal images from different vascular phases to synthesize hepatobiliary phase images, thereby improving the robustness and fidelity of the synthesized hepatobiliary phase images. The specific technical solution is as follows:

[0007] In a first aspect, a hepatobiliary magnetic resonance imaging (MRI) image generation model is provided, comprising a generator and multiple discriminators. In a first implementable embodiment of the first aspect, the generator includes:

[0008] Multiple encoders are configured to extract image features from real modal images and hepatobiliary images of different vascular phases, wherein the real modal images include T1-weighted images, arterial phase images, and portal venous phase images;

[0009] The fusion module is configured to fuse image features from all real modal images to obtain fused image features;

[0010] The decoder is configured to construct reconstructed images of each modality and reconstructed images of the hepatobiliary phase based on the features of the fused image, and to reconstruct real hepatobiliary phase images based on the image features of the hepatobiliary phase images;

[0011] Multiple discriminators, including a T1 discriminator, an AP discriminator, a VP discriminator, and an HBP discriminator, are configured to determine the authenticity probability of each modal reconstructed image based on the corresponding real modal image, and to determine the authenticity probability of the hepatobiliary reconstructed image based on the real hepatobiliary phase image.

[0012] The encoder includes:

[0013] The common feature extraction module is configured to extract common image features between different real modal images;

[0014] A dedicated feature extraction module is configured to extract dedicated image features from the corresponding real modal images;

[0015] The modal feature enhancement module is configured to enhance the corresponding features in the extracted image features using a feature enhancement method that matches the real modal image.

[0016] In conjunction with the first implementable method of the first aspect, the second implementable method of the first aspect enhances the corresponding features in the extracted image features, including:

[0017] Based on the global inter-class maximization criterion, a learnable multi-channel Otsu layer is used to enhance the high-response features of the T1-weighted image;

[0018] Arterial vascular texture features in arterial phase images were enhanced by using a superimposed Gabor filter;

[0019] And / or, jointly learnable multi-channel Otsu layers with Gabor filters enhance portal vein vascular texture features in portal venous phase images.

[0020] In conjunction with the first possible implementation of the first aspect, in the third possible implementation of the first aspect, the fusion module includes:

[0021] The multimodal attention module is configured to calculate the spatial attention weights corresponding to the image features of each real modality image;

[0022] The feature fusion block is configured to weight and fuse the image features of each real modality image according to the spatial attention weights corresponding to all real modality images to obtain the fused image features.

[0023] In conjunction with the third possible implementation of the first aspect, in the fourth possible implementation of the first aspect, the fusion module further includes:

[0024] The maximization block is configured to perform a maximum value operation on the image features of each real modality image, retain the high information content features across modalities, and then concatenate the high information content features with the fused image features.

[0025] In conjunction with the third possible implementation of the first aspect, in the fifth possible implementation of the first aspect, the fusion module further includes:

[0026] The clinical information fusion block is configured to encode clinical data into clinical feature vectors that match the features of the fused image, and to fuse the clinical feature vectors as described in the fused image features.

[0027] Secondly, a method for generating hepatobiliary magnetic resonance images is provided. In a first implementable embodiment of the second aspect, the method includes:

[0028] Construct a hepatobiliary magnetic resonance image generation model as described in any of the first to fifth implementable methods of the first aspect, and train the hepatobiliary magnetic resonance image generation model using the constructed dataset;

[0029] Based on the acquired real modal images, hepatobiliary magnetic resonance images are generated using a trained hepatobiliary magnetic resonance image generation model.

[0030] In conjunction with the first feasible approach of the second aspect, the second feasible approach of the second aspect trains the hepatobiliary phase magnetic resonance image generation model, including:

[0031] A loss function is constructed based on adversarial loss, pixel loss, and temporal continuity loss.

[0032] In conjunction with the first feasible approach of the second aspect, the third feasible approach of the second aspect generates hepatobiliary magnetic resonance images using a trained hepatobiliary magnetic resonance image generation model, including:

[0033] The acquired real modal images are registered and aligned.

[0034] Beneficial Effects: The hepatobiliary phase magnetic resonance image generation model and method of this invention can synthesize hepatobiliary phase magnetic resonance images by fusing modal image features from different vascular phases through a set hepatobiliary phase magnetic resonance image generation model. Compared with synthesizing hepatobiliary phase images from a single modal image, it can comprehensively utilize the complementary hemodynamic information from different vascular phases, ensuring the stability and availability of data, and generating more comprehensive and stable hepatobiliary phase magnetic resonance image information, thereby improving the robustness and fidelity of the synthesis results. Attached Figure Description

[0035] To more clearly illustrate the specific embodiments of the present invention, the accompanying drawings used in the specific embodiments will be briefly described below. In all the drawings, the elements or parts are not necessarily drawn to scale.

[0036] Figure 1 is an architecture diagram of a magnetic resonance imaging generation model for the hepatobiliary phase provided in an embodiment of the present invention;

[0037] Figure 2 is an architecture diagram of a fusion module provided in an embodiment of the present invention;

[0038] Figure 3 is a flowchart of a method for generating hepatobiliary magnetic resonance images according to an embodiment of the present invention;

[0039] Figure 4 shows a hepatobiliary magnetic resonance image of a patient generated using the hepatobiliary magnetic resonance image generation method provided by the present invention, wherein A-true T1 modal image, B-true arterial phase modal image, C-true portal venous phase modal image, D-true hepatobiliary phase modal image, and E-reconstructed true hepatobiliary phase image.

[0040] Figure 5 shows a hepatobiliary phase magnetic resonance image of a patient generated using the hepatobiliary phase magnetic resonance image generation method provided by the present invention, wherein A-true T1 modal image, B-true arterial phase modal image, C-true portal venous phase modal image, D-true hepatobiliary phase modal image, and E-reconstructed true hepatobiliary phase image.

[0041] Figure 6 shows a hepatobiliary phase magnetic resonance image of a patient generated using the hepatobiliary phase magnetic resonance image generation method provided by the present invention, wherein A-true T1 modal image, B-true arterial phase modal image, C-true portal venous phase modal image, D-true hepatobiliary phase modal image, and E-reconstructed true hepatobiliary phase image. Detailed Implementation

[0042] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are merely illustrative of the technical solution of the present invention and are therefore intended to limit the scope of protection of the present invention.

[0043] It should be understood that in this embodiment, T1 refers to Longitudinal Relaxation Time (T1), AP refers to Arterial Phase (AP), VP refers to Venous Phase (VP), and HBP refers to Hepatobiliary Phase (HBP).

[0044] Figure 1 shows the architecture of the hepatobiliary phase magnetic resonance image generation model. This model includes a generator and multiple discriminators. The generator includes:

[0045] Multiple encoders are configured to extract image features from real modal images and hepatobiliary images of different vascular phases. The real modal images include T1-weighted images, arterial phase images, and portal venous phase images.

[0046] The fusion module is configured to fuse image features from all real modal images to obtain fused image features;

[0047] The decoder is configured to construct reconstructed images of each modality and reconstructed images of the hepatobiliary phase based on the features of the fused image, and to reconstruct real hepatobiliary phase images based on the image features of the hepatobiliary phase images;

[0048] Multiple discriminators are configured to determine the authenticity probability of each modal reconstructed image based on the corresponding real modal image, and to determine the authenticity probability of the hepatobiliary reconstructed image based on the real hepatobiliary phase image.

[0049] Specifically, the generative model includes a generator and multiple discriminators. The generator comprises multiple encoders, a fusion module, and a decoder. In this embodiment, the multiple encoders may include a T1 encoder, an AP encoder, a VP encoder, and an HBP encoder. The T1 encoder is used to extract features from real T1-weighted images to obtain T1-weighted image features. The AP encoder is used to extract features from real arterial phase images to obtain arterial phase image features. The HBP encoder can extract features from input real hepatobiliary phase images during training to extract hepatobiliary phase image features from real hepatobiliary phase images.

[0050] The fusion module can fuse the extracted T1-weighted image features, arterial phase image features, and portal venous phase image features into a fused image feature. The decoder can reconstruct the T1-weighted, arterial, and portal venous phase MRI images based on the fused image feature, obtaining the T1-weighted reconstructed image, AP reconstructed image, VP reconstructed image, and hepatobiliary phase reconstructed image. Simultaneously, it can map the hepatobiliary phase image features to a true hepatobiliary phase image of the same size as the hepatobiliary phase reconstructed image, facilitating subsequent processing.

[0051] Multiple discriminators can include a T1 discriminator, an AP discriminator, a VP discriminator, and an HBP discriminator. During the training of the generative model, the T1 discriminator can determine the probability of authenticity of the T1-weighted reconstructed image generated by the decoder based on the input real T1-weighted image, and feed this probability back to the generator through a loss function to adjust the parameters of the corresponding modules in the generator to optimize the quality of the generated image. Similarly, the AP discriminator can determine the probability of authenticity of the AP reconstructed image generated by the decoder based on the input real arterial phase image, and optimize the module parameters of the corresponding modules in the generator through a loss function. The HBP discriminator can determine the probability of authenticity of the hepatobiliary phase reconstructed image generated by the decoder based on the reconstructed real hepatobiliary phase image, and optimize the module parameters of the corresponding modules in the generator through a loss function. After the generative model is trained, the patient's T1-weighted image, arterial phase image, and portal venous phase image can be input into the generative model. The generative model can then capture and fuse the complementary hemodynamic information in the T1-weighted image, arterial phase image, and portal venous phase image to synthesize a hepatobiliary phase magnetic resonance image, as shown in Figure 4-6.

[0052] In this embodiment, optionally, the real modal images include T1-weighted images, arterial phase images, and portal venous phase images.

[0053] Specifically, T1-weighted images, arterial phase images, and portal venous phase images are standard acquisition sequences in liver MRI examination protocols. These sequences not only provide stable acquisition but also capture complementary hemodynamic information across different vascular phases. Furthermore, both liver parenchyma and focal lesions exhibit characteristic dynamic signal changes. T1-weighted images reflect baseline contrast information. During the arterial phase, tumors typically show significant enhancement relative to the liver parenchyma. During the portal venous phase, liver parenchyma enhancement gradually increases, while some tumors show decreased enhancement or a relative decrease in signal intensity. The signal brightness of the tumor is significantly weaker than that of the significantly enhanced normal liver parenchyma at this time, creating a stark contrast; visually, the tumor area appears darker than the surrounding liver parenchyma. This temporal signal variation contains physiologically significant clues, crucial for guiding the accurate and clinically relevant synthesis of hepatobiliary phase images.

[0054] In this embodiment, optionally, the encoder includes:

[0055] The common feature extraction module is configured to extract common image features between different real modal images;

[0056] A dedicated feature extraction module is configured to extract dedicated image features from the corresponding real modal images;

[0057] The modal feature enhancement module is configured to enhance the corresponding features in the extracted image features using a feature enhancement method that matches the real modal image.

[0058] Specifically, each encoder includes a common feature extraction module, a specific feature extraction module, and a modal feature enhancement module. The common feature extraction module can extract common image features between different real modal images to ensure the consistency of the generated results at the basic structural level.

[0059] In this embodiment, the common feature extraction module adopts a multi-layer convolutional structure, including convolutional blocks and max-pooling layers. A multi-scale feature extraction mechanism is used to uniformly encode the input modal images, extracting the core features common to different modal images, providing fundamental shared feature support for subsequent multi-modal feature fusion.

[0060] The dedicated feature extraction module matches the input modal image to extract the unique detailed features of the input modal image to reflect its specific characteristics. For example, it can obtain the T1 relaxation, anatomical structure, and inherent T1-weighted signal intensity features of tissues such as liver parenchyma and tumors in the T1-weighted image; features such as tumor enhancement, blood perfusion, and vascular contour in the arterial phase (AP); and features such as tumor enhancement reduction, homogeneous enhancement of liver parenchyma, and lesion boundaries in the portal venous phase (VP).

[0061] The dedicated feature extraction module can adopt a multi-scale convolutional structure that matches the common feature extraction module, and some parameters can be shared in the dedicated feature extraction modules of different encoders. This ensures targeted capture of modality-specific information, and the sharing of parameters maintains a certain similarity in the high-level features of different modalities, while reducing network memory consumption.

[0062] The exclusive image features extracted by the exclusive feature extraction module and the common image features extracted by the common feature extraction module are gradually fused through the fusion layer, thereby achieving an effective combination of shared and unique information. Finally, a multi-scale feature set of each modality image is output, providing refined modality-specific features for the subsequent fusion module and modality feature enhancement module.

[0063] Specifically, the dedicated image features include feature maps and attention maps corresponding to the modal images. First, the feature maps and attention maps extracted from the modal images by each dedicated feature extraction module are obtained. Then, the attention maps of different modal images are concatenated and normalized using softmax to obtain the normalized attention weights corresponding to different modal images. The specific calculation formula is as follows:

[0064] ;

[0065] in, To normalize the attention weight map, Attention map extracted by the dedicated feature extraction module. This is the natural exponential function, used to convert the values ​​of the original attention map into non-negative numbers, in preparation for subsequent normalization. These represent T1-weighted images, arterial phase images, and portal venous phase images, respectively.

[0066] Next, the feature maps of each modality image are multiplied element-wise with their own normalized attention weights to obtain the corresponding single-modality weighted feature maps. Then, the single-modality weighted feature maps corresponding to each modality image are summed and fused with the cross-modality shared feature maps output by the common feature extraction module to obtain the basic fused features, i.e., the multi-scale feature set. The specific calculation formula is as follows:

[0067]

[0068] in, Based on the fusion feature map, , , These are feature maps corresponding to the T1-weighted image, arterial phase image, and portal venous phase image, respectively. , , These are the normalized attention weight maps corresponding to the T1-weighted image, the arterial phase image, and the portal venous phase image, respectively. This is the cross-modal shared feature map output by the common feature extraction module.

[0069] Different encoders match their modal feature enhancement modules to the input modal images, allowing for the use of adaptive modal image feature enhancement methods to enhance corresponding features in the multi-scale feature set extracted from the modal images. For example, the T1 encoder includes a learnable multi-channel Otsu layer, which can optimize liver tissue region segmentation in T1-weighted images, highlighting the distinction between liver parenchyma and surrounding tissues. The AP encoder includes stacked Gabor filters, which can enhance the texture features of arterial vessels in the multi-scale feature set of arterial phase images, strengthening vessel peaks. This provides high-quality single-modal feature input for subsequent processing.

[0070] In this embodiment, optionally, the corresponding features in the extracted image features are enhanced, including:

[0071] Based on the global inter-class maximization criterion, a learnable multi-channel Otsu layer is used to enhance the high-response features of the T1-weighted image;

[0072] Arterial vascular texture features in arterial phase images were enhanced by using a superimposed Gabor filter;

[0073] And / or, jointly learnable multi-channel Otsu layers with Gabor filters enhance portal vein vascular texture features in portal venous phase images.

[0074] Specifically, the feature enhancement methods used in the modal feature enhancement modules of different encoders are adapted to the modal images of the input encoder. Among them, the T1 encoder includes a learnable multi-channel Otsu layer, which can optimize the segmentation of liver tissue regions in the T1-weighted image, highlighting the distinction between liver parenchyma and surrounding tissues.

[0075] Specifically, the learnable multi-channel Otsu layer can first globally normalize the multi-scale feature set corresponding to the T1 weighted image to [0, 1]. The specific calculation formula is as follows:

[0076] ;

[0077] in, For the multi-scale feature set of the T1 weighted image, The minimum value in the feature set of the T1 weighted image. The maximum value in the feature set of the T1 weighted image.

[0078] Then, the feature categories in the multi-scale feature set are divided by a learnable threshold set, and the global inter-class variance is calculated. The specific calculation formula is as follows:

[0079] ;

[0080] ;

[0081] ;

[0082] ;

[0083] in, This represents the average signal value of all feature pixels after T1 mode normalization. This is the set of flattened feature maps of the normalized T1-weighted image. for The value of a single pixel in the middle. It is the sum of signal strengths of all T1 feature pixels, and N is the total number of pixels in the feature map, which is the number of channels of the feature map of the T1 weighted image × height × width.

[0084] For the first Weights of class features For the first The pixel mask function of class features, i.e., when pixel Belongs to the The value is 1 if it represents a class feature, and 0 otherwise, and is used to filter pixels of that class. To accommodate numerical stability terms with a denominator of 0, The global inter-class variance is used to measure the signal difference between different classes in the T1 feature map. The global mean. The number of feature categories for dividing a multi-scale feature set.

[0085] The final enhanced feature set selected from the multi-scale feature set corresponding to the T1 weighted image is:

[0086] ;

[0087] in, This is the learnable optimal threshold. In the "otherwise" case, the output is a feature map with a value of 0. When the pixel value is less than the threshold, the enhanced feature set is set to 0, and the output... The shape is completely identical to the input feature map, so subsequent network layers can still receive tensors of the complete shape without interrupting computation. Simultaneously, because of the threshold set... As the parameters are learnable, the network will automatically optimize the thresholds to avoid the extreme case where all thresholds are set to 0.

[0088] The AP encoder includes a superimposed Gabor filter, which enhances the texture features of arterial vessels in the multi-scale feature set of arterial phase images.

[0089] Specifically, the Gabor kernel of the Gabor filter is defined by a Gaussian-modulated cosine wave, and the specific calculation formula is as follows:

[0090] ;

[0091] in, The standard deviation is Gaussian, used to control the width of the filter. Ellipticity is used to control the flattening of the filter. This is a phase shift, used to control the starting position of the cosine wave. The spatial frequency is used to control the density of the cosine wave.

[0092] The Gaussian envelope is used to limit the range of the cosine wave, making the filter effective only locally. It is a cosine carrier wave used to capture textures in specific directions and frequencies. The coordinates are the rotated coordinates, i.e., the original pixel coordinates. Rotation angle around the filter center The new coordinates obtained are calculated using the following formula:

[0093] ;

[0094] .

[0095] Specifically, Gabor filters with four different directions (0°, 45°, 90°, and 135°) are generated and convolved on the input features to capture texture information from different angles.

[0096] Different combinations of Gaussian standard deviation and spatial frequency can allow the filter to respond to textures of varying coarseness; fine veins use high frequency and low Gaussian standard deviation, while coarse veins use low frequency and high Gaussian standard deviation.

[0097] The Gaussian envelope allows the filter to work only locally, avoiding global noise interference and highlighting the local texture edges of blood vessels.

[0098] By setting the Gaussian standard deviation and spatial frequency as learnable parameters, the Gaussian standard deviation and spatial frequency can be automatically adjusted during model training, allowing the filter to more accurately match the distribution of blood vessel texture. Other parameters, such as ellipticity, can be set to common values ​​for physician image textures and do not require dynamic adjustment. Phase does not affect core features such as texture direction or frequency; the phase offset is set to 0 by default.

[0099] The VP encoder includes a learnable multi-channel Otsu layer and a Gabor filter. These elements enable simultaneous optimization of lesion region segmentation and portal vein texture extraction. This allows for the selection of high-response features from each modality of the image, providing high-quality single-modality feature input for subsequent fusion.

[0100] In this embodiment, optionally, the fusion module includes:

[0101] The multimodal attention module is configured to calculate the spatial attention weights corresponding to the image features of each real modality image;

[0102] The feature fusion block is configured to weight and fuse the image features of each real modality image according to the spatial attention weights corresponding to all real modality images to obtain the fused image features.

[0103] Specifically, as shown in Figure 2, the fusion module includes a multimodal attention module, a clinical information fusion block, a maximization integration block, and a feature fusion block. The multimodal attention block adapts to the multi-scale feature sets of three modalities: T1-weighted, arterial phase (AP), and portal venous phase (VP). It extracts features of different ranges through 3×3, 5×5, and 7×7 multi-receptive field convolutions, concatenates them, performs 1×1 convolution dimensionality reduction and Sigmoid activation to generate an initial attention map, and then uses Softmax normalization to obtain the spatial attention weights for each modality. The feature fusion block performs a weighted summation of the features from the three modalities based on these weights, then performs residual fusion with the output of the clinical information fusion block, concatenates it with the output of the maximization integration block, and finally completes feature integration through a 1×1 convolution to obtain the final fused image features.

[0104] In this embodiment, optionally, the fusion module further includes:

[0105] The clinical information fusion block is configured to encode clinical data into clinical feature vectors that match the features of the fused image, and to fuse the clinical feature vectors as described in the fused image features.

[0106] Specifically, the fusion module also includes a clinical information fusion block. This block linearly and normally processes patient clinical data, such as age, gender, and 22 key liver function indicators including serum total bilirubin and albumin. This clinical data is encoded into clinical feature vectors that match the number of feature channels in the fused image. Finally, a 1×1 convolution is used to adjust the dimension of the clinical feature vectors, and the adjusted vectors are residually fused with the fused image features. This supplements the fused image features with physiological background information, improving the physiological consistency of the subsequently generated images. This results in hepatobiliary images that are not only visually realistic but also physiologically consistent with the individual patient's condition, greatly enhancing the physiological rationality of the generated results and their practical application value in clinical diagnosis and assessment.

[0107] In this embodiment, optionally, the fusion module further includes:

[0108] The maximization block is configured to perform a maximum value operation on the image features of each real modality image, retain the high information content features across modalities, and then concatenate the high information content features with the fused image features.

[0109] Specifically, the fusion module also includes a maximization ensemble block. This maximization ensemble block performs a maximum value operation on the image features extracted from the T1-weighted image, arterial phase image, and portal venous phase image. It retains the vascular peak enhancement features in the arterial phase image and the lesion high signal features in the portal venous phase image, while discarding low-response information. These retained high-information features are then concatenated with the fused image features to supplement the modal peak information that may be lost during weighted fusion. This improves the completeness of the fused image features, including T1-weighted, AP arterial phase, VP portal venous phase multimodal weighted fusion features, and clinical information coding features. The fused image features form a complete fusion feature that includes both precisely weighted image features and supplements clinical background and modal peak information.

[0110] In this embodiment, the decoder can reconstruct the TI-weighted image, arterial phase image, and portal venous phase image based on the fused image features, as well as generate the hepatobiliary phase image. It mainly includes an upsampling block and T1 output layers, AP output layers, VP output layers, and HBP output layers. The upsampling block can gradually restore the fused image features output by the fusion module from low resolution to a size consistent with the original image through transposed convolutional layers, ensuring the spatial dimensionality consistency of the generated images across different modalities.

[0111] The T1 output layer adapts to the grayscale range of anatomical structures, mapping the restored fused image features to a T1-weighted reconstructed image. The AP and VP output layers match the enhancement signal range, mapping the fused image features to arterial and portal venous phase reconstructed images, respectively. The HBP output layer optimizes grayscale mapping based on the high signal intensity of the liver parenchyma and the low signal intensity of lesions during the hepatobiliary phase, mapping the fused image features to a hepatobiliary phase reconstructed image.

[0112] The T1 output layer, AP output layer, VP output layer, and HBP output layer are all connected in series with the tanh() activation function, which can normalize the pixel values ​​of the generated reconstructed image to the range of [-1,1], so as to ensure that the pixel range of the T1 weighted reconstructed image, the arterial phase reconstructed image, the portal venous phase reconstructed image, and the hepatobiliary phase reconstructed image are consistent, providing a unified benchmark for subsequent discriminator discrimination and loss calculation.

[0113] Figure 3 shows a flowchart of a method for generating hepatobiliary magnetic resonance images. This method includes:

[0114] Step 1: Construct the above-mentioned hepatobiliary magnetic resonance image generation model, and train the hepatobiliary magnetic resonance image generation model using the constructed dataset;

[0115] Step 2: Based on the acquired real modal images, generate hepatobiliary magnetic resonance images using the trained hepatobiliary magnetic resonance image generation model.

[0116] Specifically, firstly, a hepatobiliary phase MRI image generation model, as described above, can be constructed based on a generative adversarial network (GAN) framework. This model is then trained using a pre-built dataset to optimize the parameters of the generator and discriminator, thereby improving the quality of the generated images. Next, various real-world modal images of the patient, such as T1-weighted images, arterial phase images, and portal venous phase images, can be acquired and input into the trained hepatobiliary phase MRI image generation model. This model then generates the patient's hepatobiliary phase MRI images.

[0117] The hepatobiliary phase MRI image generation model can fuse modal image features from different vascular phases to synthesize hepatobiliary phase MRI images. Compared to synthesizing hepatobiliary phase images from single-modal images, it can comprehensively utilize complementary hemodynamic information from different vascular phases, ensuring data stability and availability, and generating more comprehensive and stable hepatobiliary phase MRI images, thus improving the robustness and fidelity of the synthesis results.

[0118] In this embodiment, when constructing the dataset, T1-weighted images, arterial phase images, portal venous phase images, and hepatobiliary phase images corresponding to multiple patients can be obtained. Then, using the T1-weighted images as the target template images, the SyN registration algorithm is used to register and align the arterial phase images, portal venous phase images, and hepatobiliary phase images with the target template images, respectively.

[0119] Since liver MRI images are 3D data (H×W×Z, where Z is the number of slices), but the vast majority of slices have no clinical significance, processing all slices would lead to the following three major problems:

[0120] (1) The proportion of invalid data is extremely high. A patient's 3D MRI usually has 50 to 200 slices, but only a dozen or so slices containing the liver and tumor are diagnostically valuable. The rest of the slices are just background, such as air, bones, and other non-target organs in the abdomen.

[0121] (2) Introducing a lot of noise. The pixel values ​​of pure background slices are meaningless and will interfere with the calculation of intra-case normalization, such as lowering vmin or raising vmax, resulting in the compression of the gray range of effective slices and loss of texture (such as blood vessel) features;

[0122] (3) Wasting computing resources, processing or storing hundreds of meaningless slices will increase the memory or time cost of subsequent model training, and may also cause the model to learn irrelevant features of background noise, such as the scan bed and artifacts, reducing the model's ability to recognize tumors or blood vessels.

[0123] Therefore, this method only retains the core slice containing both the tumor and the liver. The specific operation is as follows:

[0124] By calculating the sum of non-zero pixel images in the tumor mask, the image range with values ​​for each modality of the tumor mask is located. Based on this, eight images are extended forward and eight images backward to determine the initial effective image range. This eight-layer extension effectively covers the complete anatomical context of the tumor, ensuring a sufficient number of slices and compatibility with differences in scan slice thickness. Subsequently, other modalities are matched and adjusted based on the effective slice range of the T1-weighted image. If the number of slices is insufficient, the last slice index is repeated; if the number is excessive, the first half is truncated. This solves the problem of inconsistent slice numbers caused by differences in scanning parameters between different modalities, ensuring a one-to-one correspondence between multimodal slices at the same anatomical location. Finally, the processed arterial phase images, portal venous phase images, hepatobiliary phase images, and T1-weighted images are included as training samples in the dataset, providing a foundation for training the hepatobiliary phase MRI image generation model.

[0125] Once the dataset is constructed, the hepatobiliary MRI image generation model can be trained using training samples from the dataset based on the set loss function. In this embodiment, the loss function can be constructed based on adversarial loss, pixel loss, and temporal continuity loss, and the specific calculation formula is as follows:

[0126]

[0127] Among them, combating losses The loss includes that of the T1 discriminator, AP discriminator, VP discriminator, and HBP discriminator, with each discriminator responsible for determining the authenticity of the corresponding modality image. The specific calculation formula is as follows:

[0128] ;

[0129] ;

[0130] in, These are the discriminator numbers, corresponding to the T1 discriminator, AP discriminator, VP discriminator, and HBP discriminator, respectively. The total number of discriminators, This serves as the input mask for the discriminator. For the first A discriminator for real images The discriminant output, For the first A discriminator pairs with the generator. The discriminative output of the generated reconstructed image, Represents the image in the true data distribution Take the expected value, which is the mean of the loss calculated from all real images. Indicates the generator The generated refactoring Take the expected value, which is to calculate the mean of the loss over all generated images;

[0131] Pixel loss includes the general pixel loss from input modality reconstruction, as well as the region-guided loss for generating hepatobiliary images, enhancing the accuracy of liver and tumor regions. The specific calculation formula is as follows:

[0132] ;

[0133] in, The first generated by the generator Modal reconstruction image of each modality, For the first The true modal image of each modality. These are real images of the hepatobiliary phase. The reconstructed hepatobiliary phase image generated by the generator. For a weighted system, the HBP modes that need to be generated are... The input data actually contains modalities. , For regional loss weights, Additional pixel loss in the tumor and liver regions of the hepatobiliary phase images is weighted.

[0134] The region-guided loss is used to enhance the generation accuracy of tumor and liver regions, and the specific calculation formula is as follows:

[0135]

[0136] in, For tumor region masking, This is the liver region mask, and ⊙ represents element-wise multiplication.

[0137] For multiple modal clinical scan time series, temporal trend constraints and regional intensity relationship constraints are designed, and optimization is achieved through temporal continuity loss. The specific calculation formula is as follows:

[0138] ;

[0139] in, The specific calculation formula for the temporal trend of tumor regions, based on clinical characteristics, is as follows:

[0140] ;

[0141] ;

[0142] The average intensity of the region corresponding to different modal images. , , and The values ​​represent the average intensity of the tumor or liver region corresponding to the T1-weighted image, arterial phase image, portal venous phase image, and hepatobiliary phase image, respectively.

[0143] The time-series trend of the liver region is calculated using the following formula:

[0144] ;

[0145] The relative intensity constraint for liver tumors is used to ensure that the intensity relationship between the liver and tumor in different modalities conforms to clinical norms. The specific calculation formula is as follows:

[0146] ;

[0147] in, The average intensity of the tumor region in the T1-weighted image. The average intensity of the liver region in the T1-weighted image. The average intensity of the tumor region in the arterial phase image. The average intensity of the liver region in the arterial phase image. This represents the average intensity of the tumor region in the portal venous phase image. The average intensity of the liver region in the portal venous phase image. This represents the average intensity of the tumor region in the hepatobiliary phase images. This represents the average intensity of the liver region in the hepatobiliary phase image.

[0148] By using the Adam optimizer and backpropagating to update the parameters of the generator and discriminator groups based on the aforementioned loss function, the training of the hepatobiliary MRI image generation model can be completed. After training, the trained hepatobiliary MRI image generation model can be validated using sample data from the dataset. The model synthesizes hepatobiliary images from the training samples, and combined with real hepatobiliary images from the training samples, the mean absolute error (MAE), peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and Pearson correlation coefficient of the synthesized images are calculated, along with their mean and standard deviation. Simultaneously, the Fréchet initial distance (FID) is calculated. The model's generation performance is quantitatively evaluated from dimensions such as pixel error, image quality, structural similarity, pixel correlation, and generation diversity.

[0149] After training the hepatobiliary phase magnetic resonance image generation model, T1-weighted images, arterial phase images, and portal venous phase images of the patient can be obtained. The obtained T1-weighted images, arterial phase images, and portal venous phase images are then registered and aligned. Finally, the registered and aligned T1-weighted images, arterial phase images, and portal venous phase images are input into the hepatobiliary phase magnetic resonance image generation model, which generates hepatobiliary phase magnetic resonance images.

[0150] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A magnetic resonance imaging (MRI) image generation model for the hepatobiliary phase, comprising a generator and multiple discriminators, characterized in that, The generator includes: multiple encoders, configured to extract image features from real modal images and hepatobiliary phase images of different vascular phases, the real modal images including T1-weighted images, arterial phase images, and portal venous phase images; a fusion module, configured to fuse the image features of all real modal images to obtain fused image features; a decoder, configured to construct reconstructed images of each modality and reconstructed images of the hepatobiliary phase based on the fused image features, and to reconstruct real hepatobiliary phase images based on the image features of the hepatobiliary phase images; and multiple discriminators, including a T1 discriminator, an AP discriminator, a VP discriminator, and an HBP discriminator, configured to determine the authenticity probability of each reconstructed image based on the corresponding real modal image, and to determine the authenticity probability of the reconstructed hepatobiliary phase image based on the real hepatobiliary phase image; the encoders include: a common feature extraction module, configured to extract common image features among different real modal images; a specific feature extraction module, configured to extract specific image features reflecting the real modal images; and a modality feature enhancement module, configured to enhance the corresponding features in the extracted image features using a feature enhancement method matching the real modal images.

2. The hepatobiliary phase magnetic resonance image generation model according to claim 1, characterized in that, The corresponding features in the extracted image features are enhanced, including: enhancing the high response features of the T1-weighted image by using a learnable multi-channel Otsu layer based on the global inter-class maximization criterion; enhancing the arterial vascular texture features of the arterial phase image by using a superimposed Gabor filter; and / or, jointly enhancing the portal venous vascular texture features of the portal venous phase image by combining the learnable multi-channel Otsu layer and the Gabor filter.

3. The hepatobiliary phase magnetic resonance image generation model according to claim 1, characterized in that, The fusion module includes: a multimodal attention module, configured to calculate the spatial attention weights corresponding to the image features of each real modal image; and a feature fusion block, configured to weight and fuse the image features of each real modal image according to the spatial attention weights corresponding to all real modal images to obtain the fused image features.

4. The hepatobiliary phase magnetic resonance image generation model according to claim 3, characterized in that, The fusion module further includes a maximization integration block, configured to perform a maximum value operation on the image features of each real modality image, retain the high information content features across modalities, and concatenate the high information content features with the fused image features.

5. The hepatobiliary phase magnetic resonance image generation model according to claim 3, characterized in that, The fusion module further includes a clinical information fusion block, configured to encode clinical data into clinical feature vectors that match the fused image features, and to fuse the clinical feature vectors as the fused image features.

6. A method for generating hepatobiliary phase magnetic resonance images, characterized in that, include: A hepatobiliary phase magnetic resonance image generation model as described in any one of claims 1-5 is constructed, and the hepatobiliary phase magnetic resonance image generation model is trained using the constructed dataset; based on the acquired real modal images, hepatobiliary phase magnetic resonance images are generated using the trained hepatobiliary phase magnetic resonance image generation model.

7. The method for generating hepatobiliary phase magnetic resonance images according to claim 6, characterized in that, The model for generating hepatobiliary magnetic resonance images was trained by constructing a loss function based on adversarial loss, pixel loss, and temporal continuity loss.

8. The method for generating hepatobiliary phase magnetic resonance images according to claim 6, characterized in that, Hepatobiliary MRI images are generated using a trained hepatobiliary MRI image generation model, including registration and alignment of the acquired real modal images.

Images