Method and system for synthesizing HBP images from dynamic enhanced images based on deep learning

Abstract

The present application discloses a method and system for synthesizing HBP images from dynamic enhanced images based on deep learning, the method comprising: acquiring a multimodal medical image and performing data preprocessing to obtain a preprocessed multimodal medical image; introducing a multimodal implicit alignment module based on weighted fusion of spatiotemporal latent space to construct a hepatobiliary medical image synthesis network model; performing hepatobiliary medical image synthesis processing on the preprocessed multimodal medical image based on the hepatobiliary medical image synthesis network model, and outputting a hepatobiliary medical synthesis image. The embodiment of the present application can weight the importance output of each mode of the image, thereby balancing the contribution of each mode to suppressing negative fusion and improving the synthesis quality of the HBP image. The present application can be widely used in the field of multimodal medical imaging technology.

Description

Technical Field

[0001] The present application relates to the field of multimodal medical imaging technology, and in particular to a method and system for synthesizing HBP images from dynamic enhanced images based on deep learning. Background Art

[0002] Clinical HBP imaging is a multi-phase sequence image process, and HBP synthesis can be regarded as a many-to-one synthesis process. At present, there are many studies on improving synthesis performance through multimodal fusion, which is called many-to-one synthesis. The existing single-modal sequence image generation method is difficult to process multimodal time series data, and although the existing image synthesis technology considers the input of multimodal information, it does not consider the correlation between multimodal information to improve synthesis performance and does not consider the contribution of different modalities in clinical practice.

[0003] First, due to the different sampling times, MRI images of different phases show different spatial dislocations and contrasts. Second, due to the interference of the patient's breath holding during the arterial phase sampling, motion artifacts are easily generated, the imaging quality is poor, and how to balance the contribution of each modality to suppress negative fusion is not considered. Therefore, the existing HBP synthetic image results are not ideal.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a method and system for synthesizing HBP images from dynamically enhanced images based on deep learning, which can weight the importance output of each mode of the image, thereby balancing the contribution of each mode to suppressing negative fusion and improving the synthesis quality of the HBP image.

[0006] To achieve the above object, an embodiment of the present application provides a method for synthesizing an HBP image from a dynamic enhanced image based on deep learning, the method comprising:

[0007] Acquiring a multimodal medical image and performing data preprocessing to obtain a preprocessed multimodal medical image, wherein the multimodal medical image includes a pre-enhancement medical image, an early arterial medical image, a late arterial medical image, a portal venous medical image, and a delayed medical image;

[0008] A multimodal implicit alignment module based on weighted fusion of spatiotemporal latent space was introduced to construct a network model for medical image synthesis in the hepatobiliary stage;

[0009] Based on the hepatobiliary phase medical image synthesis network model, the preprocessed multimodal medical image is subjected to hepatobiliary phase medical image synthesis processing to output a hepatobiliary phase medical synthesis image.

[0010] In some embodiments, the acquiring of a multimodal medical image and performing data preprocessing to obtain a preprocessed multimodal medical image includes:

[0011] Acquire multimodal medical images;

[0012] Scaling the multimodal medical image to obtain a scaled multimodal medical image;

[0013] The scaled multimodal medical image is scanned and inferred by a sliding window method to obtain a preprocessed multimodal medical image.

[0014] In some embodiments, the hepatobiliary stage medical image synthesis network model includes a multimodal parallel input diffusion encoder, a multimodal implicit alignment module based on spatiotemporal latent space weighted fusion, and a denoising output diffusion decoder, the output end of the multimodal parallel input diffusion encoder is connected to the input end of the multimodal implicit alignment module based on spatiotemporal latent space weighted fusion, and the output end of the multimodal implicit alignment module based on spatiotemporal latent space weighted fusion is connected to the input end of the denoising output diffusion decoder.

[0015] In some embodiments, performing hepatobiliary phase medical image synthesis processing on the preprocessed multimodal medical image based on the hepatobiliary phase medical image synthesis network model to output the hepatobiliary phase medical synthesis image includes:

[0016] Inputting the preprocessed multimodal medical image into the hepatobiliary phase medical image synthesis network model;

[0017] Based on the multimodal parallel input diffusion encoder of the hepatobiliary phase medical image synthesis network model, performing latent space coding alignment processing on the preprocessed multimodal medical image to obtain latent alignment features of the multimodal medical image;

[0018] Based on the multimodal implicit alignment module of the spatiotemporal latent space weighted fusion of the hepatobiliary stage medical image synthesis network model, multimodal weighted feature fusion processing is performed on the latent alignment features of the multimodal medical image to obtain a balanced multimodal medical feature fusion image;

[0019] Based on the denoising output diffusion decoder of the hepatobiliary stage medical image synthesis network model, the balanced multimodal medical feature fusion image is denoised to obtain a hepatobiliary stage medical synthesis image.

[0020] In some embodiments, the multimodal implicit alignment module based on the spatiotemporal latent space weighted fusion of the hepatobiliary stage medical image synthesis network model performs multimodal weight feature fusion processing on the potential alignment features of the multimodal medical image to obtain a balanced multimodal medical feature fusion image, including:

[0021] Inputting the potential alignment features of the multimodal medical images into the multimodal implicit alignment module of the spatiotemporal latent space weighted fusion of the hepatobiliary medical image synthesis network model;

[0022] Based on the transformer structure of the multimodal implicit alignment module of spatiotemporal latent space weighted fusion, linear operation processing is performed on the latent alignment features of the multimodal medical image to obtain latent feature information of the fused multimodal medical image;

[0023] Based on the multi-head attention mechanism of the multi-modal implicit alignment module of the spatiotemporal latent space weighted fusion, implicitly align the latent feature information of the fused multi-modal medical image to obtain the latent alignment feature information of the fused multi-modal medical image;

[0024] The importance value of the potential alignment feature information of the fused multimodal medical image is obtained through an adaptive fusion method of the attention mechanism and weighted calculation is performed to obtain a balanced multimodal medical feature fusion image.

[0025] In some embodiments, the denoising output diffusion decoder based on the hepatobiliary stage medical image synthesis network model performs denoising on the balanced multimodal medical feature fusion image to obtain a hepatobiliary stage medical synthesis image, including:

[0026] Based on the parametric method, set the denoising target;

[0027] Inputting the balanced multimodal medical feature fusion image into the denoising output diffusion decoder of the hepatobiliary stage medical image synthesis network model, performing denoising processing by a heavy parameter Gaussian sampling method, and obtaining a denoised multimodal medical feature fusion image;

[0028] Based on the denoising target, the denoised multimodal medical feature fusion image is mapped through a contrastive learning method to obtain a medical composite image of the hepatobiliary stage.

[0029] In some embodiments, the expression of the denoising target is specifically as follows:

[0030]

[0031] In the above formula, L D represents the denoising objective function, t represents the time step, ∈ θ represents the U-shaped backbone denoising model parameterized by θ, ∈ represents the denoising model, represents the L2 norm, z t Represents the image at time t.

[0032] In some embodiments, the expression for denoising by the re-parameterized Gaussian sampling method is specifically as follows:

[0033]

[0034] In the above formula, q(z t-1 |z t ) represents the inverse denoising process, N represents normal distribution, z t-1 represents the image at time t-1, σ t represents the standard deviation at time t, β t represents the diffusion speed at time t, and I represents the covariance matrix as the unit matrix.

[0035] In some embodiments, the hepatobiliary phase medical composite image is evaluated by image quality evaluation indicators, wherein the image quality evaluation indicators include a structural similarity loss function, a peak signal-to-noise ratio loss function, and a normalized root mean square error loss function.

[0036] To achieve the above purpose, another aspect of the embodiment of the present application proposes a system for synthesizing HBP images from dynamic enhanced images based on deep learning, the system comprising:

[0037] An acquisition module, used for acquiring multimodal medical images and performing data preprocessing to obtain preprocessed multimodal medical images, wherein the multimodal medical images include pre-enhancement medical images, early arterial medical images, late arterial medical images, portal venous medical images, and delayed medical images;

[0038] A construction module is used to introduce a multimodal implicit alignment module based on weighted fusion of spatiotemporal latent space to construct a network model for medical image synthesis in the hepatobiliary stage;

[0039] A synthesis module is used to perform hepatobiliary stage medical image synthesis processing on the preprocessed multimodal medical image based on the hepatobiliary stage medical image synthesis network model, and output a hepatobiliary stage medical synthesized image.

[0040] The embodiments of the present application include at least the following beneficial effects: The present application provides a method and system for synthesizing HBP images from dynamically enhanced images based on deep learning. The scheme obtains multimodal medical images and performs data preprocessing to obtain preprocessed multimodal medical images, further introduces a multimodal implicit alignment module based on weighted fusion of spatiotemporal latent space, constructs a network model for hepatobiliary medical image synthesis, and finally generates HBP images through multi-period dynamically enhanced images and a potential multimodal diffusion model. The multimodal implicit alignment module based on weighted fusion of spatiotemporal latent space realizes spatiotemporal implicit feature alignment and weights the importance output of each mode, thereby balancing the contribution of each modality to suppressing negative fusion and improving the synthesis quality of the HBP image. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of a method for synthesizing HBP images from dynamically enhanced images based on deep learning provided in an embodiment of the present application;

[0042] Figure 2 This is a schematic diagram of the structure of a network model for constructing a hepatobiliary medical image synthesis system according to an embodiment of the present application;

[0043] Figure 3 is a schematic diagram of ablation experiment results provided in an embodiment of the present application;

[0044] Figure 4 is a visualization diagram of ablation experiment results provided in an embodiment of the present application;

[0045] Figure 5 It is a structural diagram of the system for synthesizing HBP images from dynamically enhanced images based on deep learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0047] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in embodiments of the present invention, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0048] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0049] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present invention have the same meanings as those commonly understood by those skilled in the art to which the present application belongs. The terms used in the embodiments of the present invention are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0050] First of all, gadoxetate disodium is a new contrast agent mainly used in hepatobiliary system magnetic resonance imaging, and plays an important role in the examination and diagnosis of liver and biliary lesions. Gadoxetate disodium has the characteristics of both nonspecific extracellular contrast agents and hepatobiliary specific contrast agents. With a single injection, multi-phase dynamic scanning images of the liver and hepatobiliary phase (HBP) images can be obtained. The hepatobiliary phase plays an important role in the detection and diagnosis of liver lesions, especially greatly improving the specificity and sensitivity of detecting tiny liver lesions. The detection of tiny liver lesions by Gd-EOB-DTPA dynamic enhanced magnetic resonance provides assistance for the early detection and diagnosis of hepatocellular carcinoma, helps to provide appropriate treatment for patients with early hepatocellular carcinoma, and reduces the risk of postoperative recurrence. It plays an important clinical role in the diagnosis and treatment of patients with hepatocellular carcinoma. Therefore, Gd-EOB-DTPA dynamic enhanced magnetic resonance has been widely used in the preoperative evaluation of patients with hepatocellular carcinoma. Then, compared with the traditional MRI enhancement examination using extracellular space contrast agent, Gd-EOB-DTPA enhanced MRI takes a long time, and its total examination time is often more than 20 minutes, which reduces the patient's compliance, comfort and MRI examination throughput. Therefore, how to shorten the acquisition time of hepatobiliary phase images is a problem that radiologists and clinical researchers are concerned about.

[0051] Based on this, the embodiment of the present invention proposes to generate HBP images through multi-period dynamically enhanced images and potential multimodal diffusion models. By establishing a potential spatiotemporal alignment transformer module to achieve spatiotemporal implicit feature alignment and weight the importance output of each mode, the contribution of each mode to suppressing negative fusion is balanced, thereby improving the synthesis quality of the HBP image.

[0052] Reference Figure 1 , Figure 1 A flowchart of a method for synthesizing HBP images from dynamic enhanced images based on deep learning provided by an embodiment of the present invention, referring to Figure 1 , the method comprises the following steps:

[0053] S100, acquiring a multimodal medical image and performing data preprocessing to obtain a preprocessed multimodal medical image, wherein the multimodal medical image includes a pre-enhancement medical image, an early arterial medical image, a late arterial medical image, a portal venous medical image, and a delayed medical image;

[0054] It should be noted that, in some embodiments, step S100 may include: S110, acquiring a multimodal medical image; S120, scaling the multimodal medical image to obtain a scaled multimodal medical image; S130, scanning and reasoning the scaled multimodal medical image through a sliding window method to obtain a preprocessed multimodal medical image.

[0055] In some specific embodiments, the present invention uses data from 135 patients who underwent Gd-EOB-DTPA enhanced MRI examinations. The patients were randomly divided into a training group and a validation group (n=100, 89 males and 11 females, with an average age ± standard deviation of 50.3 ± 12.0 years old) and a test group (n=35, with 31 males and 4 females, 53.7 ± 11.9 years old). The size of each image is scaled to (80, 320, 320), and in each training iteration, the present invention randomly selects 100 patches of size (16, 64, 64) from each MRI scan to train the autoencoder. During the inference process, the present invention uses a sliding window method to generate a complete MRI scan, and the window size is equal to the size of the patch. The present invention sets the overlap between windows to 50% of the patch size and applies Gaussian weighting to the edges of the windows.

[0056] S200, introduce a multimodal implicit alignment module based on spatiotemporal latent space weighted fusion to construct a network model for medical image synthesis in the hepatobiliary stage;

[0057] It should be noted that, in some specific embodiments, the hepatobiliary stage medical image synthesis network model includes a multimodal parallel input diffusion encoder, a multimodal implicit alignment module based on spatiotemporal latent space weighted fusion, and a denoising output diffusion decoder. The output end of the multimodal parallel input diffusion encoder is connected to the input end of the multimodal implicit alignment module based on spatiotemporal latent space weighted fusion, and the output end of the multimodal implicit alignment module based on spatiotemporal latent space weighted fusion is connected to the input end of the denoising output diffusion decoder.

[0058] In some specific embodiments, Figure 2 As shown, Figure 2The structural parameters in are explained, where PP means pre-enhancement; APE means early arterial phase; APL means late arterial phase; PvP means portal venous phase; DP means delayed phase; HBP means hepatobiliary phase; ResBlock means residual block; Diffuser means diffuser; LSAT means late spatiotemporal alignment; GN means group normalization; SiLU means weighted linear unit; FC means fully connected layer; Add means addition operation; Linear means linear layer; Pool means pooling layer; Concatenate means connection; MatMul means matrix multiplication; Scale means scaling; Norm means normalization; MLP means multilayer perceptron.

[0059] The network structure of the HBP image synthesis of the embodiment of the present invention includes a multimodal parallel input diffusion encoder module, a multimodal implicit alignment (LSAT) module of spatiotemporal latent space weighted fusion, and a diffusion encoder module for denoising output. The preprocessed data image is input into the HBP image synthesis network of the embodiment of the present invention; the multimodal parallel input diffusion encoder module based on the HBP image synthesis model aligns the latent space encoders of each modality and uses them to extract the aligned latent features; the latent spatiotemporal alignment module (LSAT) based on the HBP image synthesis model obtains the balanced modal weights and the fused latent features; the denoising diffusion encoder based on the HBP image synthesis model denoises the above image to obtain the generated HBP image.

[0060] S300, performing hepatobiliary phase medical image synthesis processing on the preprocessed multimodal medical image based on the hepatobiliary phase medical image synthesis network model, and outputting a hepatobiliary phase medical synthesis image;

[0061] It should be noted that, in some embodiments, step S300 may include: S310, inputting the preprocessed multimodal medical image into the hepatobiliary medical image synthesis network model; S320, based on the multimodal parallel input diffusion encoder of the hepatobiliary medical image synthesis network model, performing latent space coding alignment processing on the preprocessed multimodal medical image to obtain the latent alignment features of the multimodal medical image; S330, based on the multimodal implicit alignment module of the spatiotemporal latent space weighted fusion of the hepatobiliary medical image synthesis network model, performing multimodal weight feature fusion processing on the latent alignment features of the multimodal medical image to obtain a balanced multimodal medical feature fusion image; S340, based on the denoising output diffusion decoder of the hepatobiliary medical image synthesis network model, performing denoising processing on the balanced multimodal medical feature fusion image to obtain a hepatobiliary medical synthetic image.

[0062] Among them, the multimodal weighted feature fusion processing is to input the potential alignment features of the multimodal medical image into the multimodal implicit alignment module of the spatiotemporal latent space weighted fusion of the hepatobiliary stage medical image synthesis network model; based on the transformer structure of the multimodal implicit alignment module of the spatiotemporal latent space weighted fusion, the potential alignment features of the multimodal medical image are linearly operated to obtain the potential feature information of the fused multimodal medical image; based on the multi-head attention mechanism of the multimodal implicit alignment module of the spatiotemporal latent space weighted fusion, the potential feature information of the fused multimodal medical image is implicitly aligned to obtain the potential alignment feature information of the fused multimodal medical image; the importance value of the potential alignment feature information of the fused multimodal medical image is obtained by the adaptive fusion method of the attention mechanism and weighted calculation is performed to obtain a balanced multimodal medical feature fusion image.

[0063] Furthermore, the balanced multimodal medical feature fusion image is denoised based on a parameterized method to set a denoising target; the balanced multimodal medical feature fusion image is input into the denoising output diffusion decoder of the hepatobiliary stage medical image synthesis network model, and denoised by a heavy parameter Gaussian sampling method to obtain a denoised multimodal medical feature fusion image; based on the denoising target, the denoised multimodal medical feature fusion image is mapped by a comparative learning method to obtain a hepatobiliary stage medical synthesis image.

[0064] In some specific embodiments, first, the embodiment of the present invention pre-trains a latent diffusion model to generate HBP images for each modality image. Compared with the general diffusion model, the latent diffusion model is more suitable for multi-modal input of medical images. It can learn information at different stages through perceptual compression operations (the encoder reduces the data dimension and the decoder returns to the original pixel space) while reducing computational costs.

[0065] The latent variable z = E(X) is adjusted according to the variance adjustment strategy β1,…,β t Diffusion is iterated over time steps t and is defined as follows:

[0066]

[0067] The forward process makes z t Comply with z t =α t z+σ t ∈, where ∈~N(0,1), α t =1-β, The diffusion decoder learns how to t Denoising is restored to z. According to the reparameterization method, the goal of denoising training can be expressed as:

[0068]

[0069] Where L D The time step in is t~U[1,T].∈ θ It is a U-shaped backbone denoising model parameterized by θ. In the HBP image synthesis process, the denoising process can be achieved by re-parameterized Gaussian sampling, and its expression is:

[0070]

[0071] Then, through contrastive learning, the encoders of each mode are aligned, and the input of any mode is projected into the same latent space. The embodiment of the present invention uses the InfoNCE loss function to constrain each encoder, which is defined as follows: Where k is the current mode, k i are the other modes in the mini-batch and τ is the temperature coefficient.

[0072] like Figure 2 As shown in the figure, the LSAT block is a transformer structure for multimodal feature fusion. This method fuses multimodal potential information by merging after linear operations, and then implicitly aligns multimodal spatiotemporal features through a multi-head attention mechanism. Then, an adaptive fusion method based on the attention mechanism is used to calculate the importance of each mode and weight the output.

[0073] Among them, LSAT first performs spatial attention operation on the multimodal latent features to implicitly arrange different modes, then learns the contribution of each mode input through the MLP layer and FC layer, and balances the modal information in the time dimension by adding weighted features.

[0074] In addition, the embodiment of the present invention also uses typical image quality evaluation indicators for performance evaluation: structural similarity (SSIM), peak signal-to-noise ratio (PSNR) and normalized root mean square error (NRMSE). Figure 3 The performance comparison between the method of the embodiment of the present invention and other HBP synthesis methods is given. The reference based on UNet and the reference based on GAN can achieve better performance, but the method proposed in the reference based on diffusion and the embodiment of the present invention can greatly improve the quality of the entire image.

[0075] also, Figure 3 The ablation study performance of the method proposed in the embodiment of the present invention is shown. It can be seen that the image generation quality of the multimodal fusion model can be improved by aligning the diffusion encoder to project the multimodal information into the same space. Specifically, after gradually increasing the importance of spatial attention and balancing the time series, the method of the embodiment of the present invention achieves the best performance.

[0076] like Figure 3 As shown, the image quality generated by UNet and GAN is worse than that generated by diffusion, especially in the area near the tumor. In particular, by aligning the multimodal feature encoder and the multi-head attention mechanism, the method of the embodiment of the present invention achieves better image generation quality. However, there are still differences from the original image in the tumor. After using the attention-based adaptive time series feature fusion, the synthesized HBP image has the same value as the real HBP image in clinical diagnosis, and the tumor in the synthesized HBP image is accurate and obvious, which is comparable to the original HBP image.

[0077] In addition, the contribution of different modes to HBP image generation is as follows Figure 4 As shown. It can be clearly seen that the delayed phase image contributes the most to the synthesis of the HBP image, followed by the portal venous phase. The contribution of the pre-enhancement image remains stable during the training process. The worst are the two arterial phases. Studies have found that after the injection of Gd-EOB-DTPA, the probability of transient severe artifacts (TSM) in the arterial phase is the highest, and the image quality is the worst. Moreover, as the scanning time increases, the probability of transient severe artifacts in other phases gradually decreases, resulting in a lower contribution of the arterial phase to the synthesis of HBP. However, the arterial phase is also essential for the generation of HBP images, so their contribution remains stable between 0.12 and 0.13.

[0078] The data table for further comparison is shown below:

[0079] Table 1 Performance comparison of different HBP synthesis methods

[0080] Methods SSIM(%)↑ PSNR(dB)↑ NRMSR↓ <![CDATA[ UNet-based ]]> 75.35±4.36 23.34±2.17 0.194±0.031 Hi-Net 76.20±5.73 24.38±1.95 0.178±0.033 SADM 81.23±3.22 25.87±1.01 0.166±0.025 Proposed 84.82±3.27 27.63±0.92 0.157±0.021

[0081] Table 2 Ablation experiment data results

[0082] CA MHA LSAT SSIM(%)↑ PSNR(dB)↑ NRMSR↓ 80.42±3.34 25.31±1.12 0.171±0.026 √ 81.47±3.47 26.17±1.15 0.163±0.033 √ √ 83.88±3.35 27.44±1.11 0.158±0.025 √ √ √ 84.82±3.27 27.63±0.92 0.157±0.021

[0083] See also Figure 5 The embodiment of the present application further provides a system for synthesizing HBP images from dynamic enhanced images based on deep learning, which can implement the above-mentioned method for synthesizing HBP images from dynamic enhanced images based on deep learning. The system includes:

[0084] An acquisition module is used to acquire multimodal medical images and perform data preprocessing to obtain preprocessed multimodal medical images, wherein the multimodal medical images include pre-enhancement medical images, early arterial medical images, late arterial medical images, portal venous medical images, and delayed medical images;

[0085] A construction module is used to introduce a multimodal implicit alignment module based on weighted fusion of spatiotemporal latent space to construct a network model for medical image synthesis in the hepatobiliary stage;

[0086] A synthesis module is used to perform hepatobiliary stage medical image synthesis processing on the preprocessed multimodal medical image based on the hepatobiliary stage medical image synthesis network model, and output a hepatobiliary stage medical synthesized image.

[0087] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0088] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for synthesizing HBP images from dynamic enhanced images based on deep learning, characterized in that: The method comprises: Acquiring a multimodal medical image and performing data preprocessing to obtain a preprocessed multimodal medical image, wherein the multimodal medical image includes a pre-enhancement medical image, an early arterial medical image, a late arterial medical image, a portal venous medical image, and a delayed medical image; A multimodal implicit alignment module based on weighted fusion of spatiotemporal latent space was introduced to construct a network model for medical image synthesis in the hepatobiliary stage; The hepatobiliary medical image synthesis network model comprises a multimodal parallel input diffusion encoder, a multimodal implicit alignment module based on spatiotemporal latent space weighted fusion, and a denoising output diffusion decoder, wherein the output end of the multimodal parallel input diffusion encoder is connected to the input end of the multimodal implicit alignment module based on spatiotemporal latent space weighted fusion, and the output end of the multimodal implicit alignment module based on spatiotemporal latent space weighted fusion is connected to the input end of the denoising output diffusion decoder; Inputting the preprocessed multimodal medical image into the hepatobiliary phase medical image synthesis network model; Based on the multimodal parallel input diffusion encoder of the hepatobiliary phase medical image synthesis network model, performing latent space coding alignment processing on the preprocessed multimodal medical image to obtain latent alignment features of the multimodal medical image; Based on the multimodal implicit alignment module of the spatiotemporal latent space weighted fusion of the hepatobiliary stage medical image synthesis network model, multimodal weighted feature fusion processing is performed on the latent alignment features of the multimodal medical image to obtain a balanced multimodal medical feature fusion image; Based on the denoising output diffusion decoder of the hepatobiliary stage medical image synthesis network model, the balanced multimodal medical feature fusion image is denoised to obtain a hepatobiliary stage medical synthesis image.

2. The method according to claim 1, characterized in that The step of acquiring a multimodal medical image and performing data preprocessing to obtain a preprocessed multimodal medical image includes: Acquire multimodal medical images; Scaling the multimodal medical image to obtain a scaled multimodal medical image; The scaled multimodal medical image is scanned and inferred by a sliding window method to obtain a preprocessed multimodal medical image.

3. The method according to claim 1, characterized in that The multimodal implicit alignment module based on the spatiotemporal latent space weighted fusion of the hepatobiliary stage medical image synthesis network model performs multimodal weight feature fusion processing on the potential alignment features of the multimodal medical image to obtain a balanced multimodal medical feature fusion image, including: Inputting the potential alignment features of the multimodal medical images into the multimodal implicit alignment module of the spatiotemporal latent space weighted fusion of the hepatobiliary medical image synthesis network model; Based on the transformer structure of the multimodal implicit alignment module of spatiotemporal latent space weighted fusion, linear operation processing is performed on the latent alignment features of the multimodal medical image to obtain latent feature information of the fused multimodal medical image; Based on the multi-head attention mechanism of the multi-modal implicit alignment module of the spatiotemporal latent space weighted fusion, implicitly align the latent feature information of the fused multi-modal medical image to obtain the latent alignment feature information of the fused multi-modal medical image; The importance value of the potential alignment feature information of the fused multimodal medical image is obtained through an adaptive fusion method of the attention mechanism and weighted calculation is performed to obtain a balanced multimodal medical feature fusion image.

4. The method according to claim 1, characterized in that: The denoising output diffusion decoder based on the hepatobiliary stage medical image synthesis network model performs denoising processing on the balanced multimodal medical feature fusion image to obtain a hepatobiliary stage medical synthesis image, including: Based on the parametric method, set the denoising target; Inputting the balanced multimodal medical feature fusion image into the denoising output diffusion decoder of the hepatobiliary stage medical image synthesis network model, performing denoising processing by a heavy parameter Gaussian sampling method, and obtaining a denoised multimodal medical feature fusion image; Based on the denoising target, the denoised multimodal medical feature fusion image is mapped through a contrastive learning method to obtain a medical composite image of the hepatobiliary stage.

5. The method according to claim 4, characterized in that The expression of the denoising target is specifically as follows: In the above formula, L D represents the denoising objective function, t represents the time step, ∈ θ represents the U-shaped backbone denoising model parameterized by θ, ∈ represents the denoising model, represents the L2 norm, z t Represents the image at time t.

6. The method according to claim 5, characterized in that The expression of the denoising process performed by the heavy parameter Gaussian sampling method is specifically as follows: In the above formula, q(z t-1 |z t ) represents the inverse denoising process, N represents normal distribution, z t-1 represents the image at time t-1, σ t represents the standard deviation at time t, β t represents the diffusion speed at time t, and I represents the covariance matrix as the unit matrix.

7. The method according to claim 1, characterized in that It also includes evaluating the hepatobiliary phase medical synthetic image through image quality evaluation indicators, and the image quality evaluation indicators include a structural similarity loss function, a peak signal-to-noise ratio loss function and a normalized root mean square error loss function.

8. A system for synthesizing HBP images from dynamic enhanced images based on deep learning, characterized in that: The system comprises: An acquisition module, used for acquiring multimodal medical images and performing data preprocessing to obtain preprocessed multimodal medical images, wherein the multimodal medical images include pre-enhancement medical images, early arterial medical images, late arterial medical images, portal venous medical images, and delayed medical images; A construction module is used to introduce a multimodal implicit alignment module based on weighted fusion of spatiotemporal latent space to construct a network model for medical image synthesis in the hepatobiliary stage; The hepatobiliary medical image synthesis network model comprises a multimodal parallel input diffusion encoder, a multimodal implicit alignment module based on spatiotemporal latent space weighted fusion, and a denoising output diffusion decoder, wherein the output end of the multimodal parallel input diffusion encoder is connected to the input end of the multimodal implicit alignment module based on spatiotemporal latent space weighted fusion, and the output end of the multimodal implicit alignment module based on spatiotemporal latent space weighted fusion is connected to the input end of the denoising output diffusion decoder; A synthesis module, used for inputting the preprocessed multimodal medical image into the hepatobiliary stage medical image synthesis network model; Based on the multimodal parallel input diffusion encoder of the hepatobiliary phase medical image synthesis network model, performing latent space coding alignment processing on the preprocessed multimodal medical image to obtain latent alignment features of the multimodal medical image; Based on the multimodal implicit alignment module of the spatiotemporal latent space weighted fusion of the hepatobiliary stage medical image synthesis network model, multimodal weighted feature fusion processing is performed on the latent alignment features of the multimodal medical image to obtain a balanced multimodal medical feature fusion image; Based on the denoising output diffusion decoder of the hepatobiliary stage medical image synthesis network model, the balanced multimodal medical feature fusion image is denoised to obtain a hepatobiliary stage medical synthesis image.

Images