A method, system, electronic device, and storage medium for synthesizing brain medical images

Abstract

The present invention relates to a method, system, electronic device and storage medium for synthesizing brain medical images. First, a multi-modal fusion network is used to fuse and add the same-level features of two source images to learn the common latent representation vector between multi-modal data. The same-level features can be high-level semantic information or low-level fine-grained features, solving the problem that existing dual-modal medical image synthesis methods cannot effectively extract and fuse multi-level features. Then, a residual bottleneck attention mechanism is used to couple the common latent representation vector with the attention-aware features, which can better retain the rich image information in the original images. Finally, a multi-modal synthesis network is used to fuse the coupled feature maps and the multi-level features of each source image, and output a brain medical synthesis image of the target modality, making full use of the complementary information between multi-modal medical images, and being able to generate a target modality image with richer semantic information, stronger fine-grained features and more delicate texture details.

Description

Technical Field

[0001] The present invention relates to the field of digital image processing, and particularly to a method, system, electronic device and storage medium for synthesizing brain medical images. Background Art

[0002] Medical image fusion refers to an image processing technology that synthesizes multiple images obtained by different imaging devices or the same imaging device at different times into one image. The medical image synthesis technology displays the structural information and functional information of the human body tissue structure through complementary imaging modes, and plays an increasingly important role in various medical diagnoses and treatments. It helps to make accurate early medical diagnoses, saves the cost of magnetic resonance imaging, and can provide complementary information between multiple modal medical images.

[0003] Existing image synthesis methods are mainly divided into two categories: registration-based methods and learning-based methods, which have successfully promoted the development of image synthesis technology. The existing registration-based methods calculate an atlas based on pre-registered images, and then use the same transformation method to map the source image to the target image. The existing literature discloses a method that combines multi-parameter atlas registration and patch-based synthesis. The multi-parameter atlas MR (Magnetic Resonance) images are registered with the target MR images based on multi-channel deformable registration, and the atlas CT images are locally averaged and weighted deformed and fused using the structural similarity index (SSIM). The weights used in the process of synthesizing CT images can also be used for the synthesis of MR images. To ensure the fidelity of the synthesized image, a patch-based method is used to refine the regions with lower fidelity to achieve the purpose of accurately synthesizing normal and abnormal tissues. Although this method can accurately synthesize abnormal and normal regions in MR or CT images at the same time, since it is mainly an atlas-based method, its performance depends highly on the registration accuracy, and the running time is long, and it is not easy to obtain a target image with high fidelity.

[0004] Therefore, to solve the above problems, scholars have proposed a learning-based method, which mainly establishes a non-linear mapping between the source image and the target image at the pixel level. The prior art discloses a supervised random forest image synthesis method named REPLICA. Aiming at the problem that T2 modality images and Flair modality images cannot be synthesized in medicine, non-linear regression is learned to predict the intensity of the alternating tissue contrast of a specific input tissue contrast. The REPLICA method has a short calculation time and good performance. However, there are still some unsolved problems. Since the predicted value of the random forest is the average of the results of all tree branches, the synthesized image usually has lower noise and is smoother than the real image. The reduction of noise is a favorable situation, but the inevitable loss of inherent resolution due to taking the average always leads to a decrease in image quality.

[0005] The features manually extracted in the above learning-based method ignore the common latent representations between multiple modalities. The prior art discloses a dual-modal medical image synthesis method based on sequential generative adversarial networks and semi-supervised learning. In this method, the modality image with lower complexity is first synthesized, and then the target image with higher complexity is generated. The proposed sequential generative adversarial network is trained end-to-end in a semi-supervised manner. In the supervised training, the reconstruction loss between the real image and the synthesized image is minimized to learn the joint distribution of the dual-modal images from the real paired images of the two modalities. To avoid overfitting, in the unsupervised training, the wasserstein distance between the real image and the synthesized image is minimized to learn the distribution of each modality of the unpaired images. Although the above method can form a mapping between the source image and the target image, the multi-level features in the neural network are often ignored, and the high-level semantic information and low-level fine-grained features of the image cannot be utilized, which may lead to image distortion and the loss of detailed texture structures.

[0006] With the development of science and technology, the application of deep learning in the field of medical image synthesis has been widely used. As a common method in the field of image synthesis, the Generative Adversarial Network (GAN) has received considerable attention in the field of image processing. However, due to the characteristics of multi-modal medical images, different modalities of medical images can provide certain complementary information, which is beneficial to generating a more complete target image. However, the existing methods ignore the multi-level features between different modalities, and the high-level semantic information and low-level fine-grained information cannot be utilized simultaneously.

[0007] Based on the above, there is an urgent need to propose a new method for synthesizing brain medical images, which can make full use of the complementary information between relevant modalities while ensuring the integrity of the generated images. Summary of the Invention

[0008] The object of the present invention is to provide a method, a system, an electronic device and a storage medium for synthesizing brain medical images, so as to make full use of the complementary information between multi-modal medical images and generate a target modal image with richer semantic information, stronger fine-grained features and more delicate texture details.

[0009] To achieve the above object, the present invention provides the following solutions:

[0010] A method for synthesizing brain medical images, comprising:

[0011] Obtain a pair of source brain medical images to be synthesized; the pair of source brain medical images to be synthesized includes two source images of different modalities that are mutually registered;

[0012] Use a modality-specific feature extraction network to extract features from each source image to obtain multi-level features of each source image;

[0013] Use a multi-modal fusion network to fuse and add the same-level features of the two source images to obtain a common latent representation vector between multi-modal data; the multi-modal fusion network includes a plurality of first hybrid attention fusion modules connected in sequence, and the first hybrid attention fusion module is used to fuse the same-level features of the two source images through a variety of fusion strategies;

[0014] Use a residual bottleneck attention mechanism to couple the common latent representation vector with attention-aware features to obtain a coupled feature map;

[0015] According to the coupled feature map and the multi-level features of each source image, use a multi-modal synthesis network to output a synthesized brain medical image of the target modality; the multi-modal synthesis network includes a plurality of second hybrid attention fusion modules connected in sequence, and the plurality of second hybrid attention fusion modules are used to fuse the same-level features of the two source images through a variety of fusion strategies and then add them to the coupled feature map.

[0016] Optionally, the first hybrid attention fusion module is used to fuse the same-level features of the two source images through a variety of fusion strategies, specifically including:

[0017] According to the same-level features of the two source images, use the formula and Determine the fusion weights through a variety of fusion strategies; in the formula, Max are element-wise addition, element-wise multiplication, and element-wise maximization respectively, are the features of the n-1th layer of the two source images, is the element-wise addition weight, is the element-wise multiplication weight, F max is the element-wise maximization weight, It is a tensor shape with 3C channels, height H, and width W;

[0018] Calculate the soft attention mechanism weight W using the formula W = σ(C3C4F); where C3 and C4 are the third and fourth convolutional layers, and σ is the Sigmod function;

[0019] According to the soft attention mechanism weight and the fusion weight, use the formula Determine the final output F of the hybrid attention fusion module MAF ; where C5 is the fifth convolutional layer.

[0020] Optionally, the residual bottleneck attention mechanism is used to couple the common latent representation vector with the attention-aware feature to obtain a coupled feature map, which specifically includes:

[0021] Use the formula m c (f) = σ(MLP(AVGPOOL(f))) + MLP(MAXPOOL(f)) to determine the channel attention component of the common latent representation vector; where m c (f) is the channel attention component of the common latent representation vector, MLP is a multi-layer perceptron, AVGPOOL is an average pooling operation, MAXPOOL is a max pooling operation, f is the common latent representation vector, and σ is the Sigmod function;

[0022] Use the formula m s (f) = σ(f 7×7 ([AVGPOOL(f); MAXPOOL(f)])) to determine the spatial attention component of the common latent representation vector; where m s (f) is the spatial attention component of the common latent representation vector, f 7×7 is a convolution operation with a convolution kernel size of 7×7;

[0023] According to the spatial attention component, the channel attention component, and the common latent representation vector, use the formula Determine the final output component f''', and form a coupled feature map;

[0024] where f' is the channel attention output, f'' is the spatial attention output, is element-wise multiplication, is element-wise addition.

[0025] Optionally, the multi-modal synthesis network further includes a convolutional layer;

[0026] The output end of the second hybrid attention fusion module at the end among multiple successively connected second hybrid attention fusion modules is connected to the input end of the convolutional layer; the convolutional layer is used to obtain a brain medical synthetic image of the target modality according to the features finally output by all the second hybrid attention fusion modules.

[0027] Optionally, before obtaining the pair of source brain medical images to be synthesized, it further includes:

[0028] Training the modality-specific feature extraction network, the multi-modal fusion network, the residual bottleneck attention mechanism, and the multi-modal synthesis network using a sample data set; the sample data set consists of sample pairs of source brain medical images and corresponding labels of brain medical synthetic images.

[0029] Optionally, the training process of the modality-specific feature extraction network is as follows:

[0030] Construct a reconstruction network; the reconstruction network includes a deconvolutional layer.

[0031] Use the reconstruction network to reconstruct the multi-level features of each source image into an image.

[0032] According to each source image and the corresponding reconstructed image, use the reconstruction loss function To train the modality-specific feature extraction network.

[0033] In the formula, is the i-th source image, is The reconstructed image of, ||*||1 is the L1 norm, is The mean of, is The mean of, is The variance of, is The variance of, is the covariance, L RS is the reconstruction loss function, L R is the L1-norm loss function, L ssim is the structural similarity loss function.

[0034] Optionally, the training process of the multi-modal synthesis network is as follows:

[0035] Establish a discriminator to discriminate between the target modality image and the true label.

[0036] Determine the loss function of the discriminator as In the formula, L Dis the loss function of the discriminator, and D(x, y) is the boolean value obtained by the discriminator for discriminating the source image and the target image label. is the maximum likelihood estimation, and D(x, G(x)) is the boolean value obtained by the discriminator for discriminating the source image and the generated target image.

[0037] Determine the loss function of the discriminator as where L D is the loss function of the discriminator, D(x, y) is the boolean value obtained by the discriminator for discriminating the source image x and the target image label y. is the maximum likelihood estimation for the source image x in the dataset p data and is the maximum likelihood estimation for the source image x and the target image label y in the dataset p data , D(x, G(x)) is the boolean value obtained by the discriminator for discriminating the source image and the generated target image.

[0038] Establish the SSIM-L1 joint loss function as

[0039]

[0040] where L SSIM-L1 is the SSIM-L1 joint loss function, λ1, λ SSIM are the first and second hyperparameters, μ x , μ y are the means of the source image x and the target image label y, σ x , σ y are the variances of the source image x and the target image label y, σ xy is the covariance of the source image x and the target image label y, G(x) is the generated target image, and c1, c2 are the first and second positive constants.

[0041] According to the SSIM-L1 joint loss function, determine the loss function of the generator in the multi-modal synthesis network as where L G is the loss function of the generator in the multi-modal synthesis network.

[0042] Use the loss function of the generator and the loss function of the discriminator in the multi-modal synthesis network to train the multi-modal synthesis network.

[0043] A brain medical image synthesis system, comprising:

[0044] A brain medical source image pair acquisition module, configured to acquire a pair of brain medical source images to be synthesized; the pair of brain medical source images to be synthesized includes two source images of different modalities that are mutually registered.

[0045] A feature extraction module for extracting features from each source image using a modality-specific feature extraction network to obtain multi-level features of each source image;

[0046] A multi-modal fusion module for fusing and adding the same-level features of two source images using a multi-modal fusion network to obtain a common latent representation vector between multi-modal data; the multi-modal fusion network includes a plurality of first hybrid attention fusion modules connected in sequence, and the first hybrid attention fusion module is used to fuse the same-level features of two source images through multiple fusion strategies;

[0047] A coupling module for coupling the common latent representation vector with attention-aware features using a residual bottleneck attention mechanism to obtain a coupled feature map;

[0048] An image synthesis module for outputting a brain medical synthesis image in a target modality according to the coupled feature map and the multi-level features of each source image using a multi-modal synthesis network; the multi-modal synthesis network includes a plurality of second hybrid attention fusion modules connected in sequence, and the plurality of second hybrid attention fusion modules are used to fuse the same-level features of two source images through multiple fusion strategies and then add them to the coupled feature map.

[0049] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, it implements the brain medical image synthesis method as described above.

[0050] A computer-readable storage medium stores a computer program, and when the computer program is executed, it implements the brain medical image synthesis method as described above.

[0051] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0052] The present invention discloses a method, system, electronic device and storage medium for synthesizing brain medical images. After extracting multi-level features of each source image, first, a multi-modal fusion network is used to fuse and add the same-level features of two source images to learn the common latent representation vector between multi-modal data. The multi-modal fusion network includes multiple first hybrid attention fusion modules connected in sequence. The first hybrid attention fusion module fuses the same-level features of two source images through multiple fusion strategies. The same-level features can be high-level semantic information or low-level fine-grained features, solving the problem that the dual-modal medical image synthesis method based on sequential generative adversarial network and semi-supervised learning cannot effectively extract and fuse multi-level features. Then, a residual bottleneck attention mechanism is used to couple the common latent representation vector with the attention-aware feature, which can better retain rich image information in the original image and reduce the degradation degree of the synthesized image. Finally, a multi-modal synthesis network is used to output the brain medical synthesis image of the target modality. The multi-modal synthesis network includes multiple second hybrid attention fusion modules connected in sequence. The second hybrid attention fusion module fuses the coupled feature map and the multi-level features of each source image, making full use of the complementary information between multi-modal medical images and capable of generating a target modality image with richer semantic information, stronger fine-grained features and more delicate texture details. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required to be used in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0054] Figure 1 It is a schematic flow chart of the method for synthesizing brain medical images provided by the embodiment of the present invention;

[0055] Figure 2 It is a schematic principle diagram of the method for synthesizing brain medical images provided by the embodiment of the present invention;

[0056] Figure 3 It is a schematic principle diagram of a hybrid attention fusion module provided by the embodiment of the present invention;

[0057] Figure 4 It is a schematic diagram of the synthesis result of other brain medical images under the algorithm of the present invention provided by the embodiment of the present invention;

[0058] Figure 5 It is a schematic diagram of the difference in the synthesis result of other brain medical images under the algorithm of the present invention provided by the embodiment of the present invention; Figure 5 In (a) is a schematic diagram of the difference in the synthesis result of Pix2Pix (T1);Figure 5 In (b) is a schematic diagram of the difference in the synthesis result of Pix2Pix (T2); Figure 5 In (c) is a schematic diagram of the difference in the synthesis result of Hi-Net; Figure 5 In (d) is a schematic diagram of the difference in the synthesis result of the present invention. Detailed implementation manners

[0059] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0060] The purpose of the present invention is to provide a method, system, electronic device and storage medium for synthesizing brain medical images, so as to make full use of the complementary information between multi-modal medical images and generate a target modal image with richer semantic information, stronger fine-grained features and more delicate texture details.

[0061] To make the above objects, features and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0062] Figure 1 is a schematic flowchart of a method for synthesizing brain medical images provided by an embodiment of the present invention, Figure 2 is a schematic principle diagram of a method for synthesizing brain medical images provided by an embodiment of the present invention. As Figure 1 and Figure 2 shown, a method for synthesizing brain medical images provided by an embodiment of the present invention specifically includes the following steps:

[0063] Step S1, obtain a pair of source brain medical images to be synthesized. The pair of source brain medical images to be synthesized includes two source images of different modalities that are mutually registered.

[0064] Figure 2 The two different source images on the far left in

[0065] constitute a pair of source brain medical images to be synthesized.

[0066] The multi-level features include high-level semantic information and low-level fine-grained features.

[0067] Taking Figure 2Taking the modality-specific feature extraction network shown as an example, the modality-specific feature extraction network sequentially includes convolutional blocks with 32 and 32 channels, a pooling layer, convolutional blocks with 64 and 64 channels, a pooling layer, convolutional blocks with 128 and 128 channels, a pooling layer, a convolutional block with 128 channels, a pooling layer, and convolutional blocks with 128 and 64 channels from left to right. The three pooling layers can obtain features at different levels.

[0068] The multi-level feature extraction process is as follows: Downsample the two source images respectively, extract the hierarchical features of the two source images simultaneously, and then perform upsampling to extract the hierarchical features of the two source images.

[0069] The training process of the modality-specific feature extraction network is as follows: Construct a reconstruction network; the reconstruction network includes a deconvolution layer; use the reconstruction network to reconstruct the multi-level features of each source image into an image; according to each source image and the corresponding reconstructed image, use the reconstruction loss function Train the modality-specific feature extraction network; where is the i-th source image, is 's reconstructed image, ||*||1 is the L1 norm, is 's mean, is 's mean, is 's variance, is 's variance, is the covariance, L RS is the reconstruction loss function, L R is the L1-norm loss function, L ssim is the structural similarity loss function.

[0070] Step S3, use the multi-modal fusion network to fuse and add the same-level features of the two source images to obtain the common latent representation vector between the multi-modal data.

[0071] The multi-modal fusion network includes multiple sequentially connected first hybrid attention fusion modules, and the first hybrid attention fusion module is used to fuse the same-level features of the two source images through multiple fusion strategies. As Figure 3 shown, the fusion process inside each hybrid attention fusion module is as follows: According to the same-level features of the two source images, use the formulas and to determine the fusion weights through multiple fusion strategies; calculate the soft attention mechanism weight W using the formula W = σ(C3C4F); according to the soft attention mechanism weight and the fusion weights, use the formula to determine the final output F of the hybrid attention fusion moduleMAF ; where, Max represents element-wise addition, element-wise multiplication, and element-wise maximization respectively, are the features of the (n - 1)-th layer of two source images, is the weight for element-wise addition, is the weight for element-wise multiplication, F max is the weight for element-wise maximization, is; C3 and C4 are the third and fourth convolutional layers, σ is the Sigmod function; C5 is the fifth convolutional layer.

[0072] Figure 2 It is shown that the multi-modal fusion network consists of 3 hybrid attention fusion modules (abbreviated as MAF). Figure 2 The two source images in Figure 2 are respectively named source image A and source image B.

[0073] The same-level features input to the leftmost MAF are: the hierarchical features obtained from the pooling layer between the convolutional blocks with channel numbers 32 and 32 and the convolutional blocks with channel numbers 64 and 64 corresponding to source image A, and the hierarchical features obtained from the pooling layer between the convolutional blocks with channel numbers 32 and 32 and the convolutional blocks with channel numbers 64 and 64 corresponding to source image B. These two hierarchical features are the same-level features, and after being fused through multiple fusion strategies, they are input to the middle MAF.

[0074] The same-level features input to the middle MAF are: the pooling layer between the convolutional blocks with channel numbers 64 and 64 and the convolutional blocks with channel numbers 128 and 128 corresponding to source image A, and the pooling layer between the convolutional blocks with channel numbers 64 and 64 and the convolutional blocks with channel numbers 128 and 128 corresponding to source image B. These two hierarchical features are the same-level features, and after being fused through multiple fusion strategies, they are added to the fused features input to the leftmost MAF, and after addition, they are input to the rightmost MAF.

[0075] The same-level features input to the rightmost MAF are: the pooling layer between the convolutional blocks with channel numbers 128 and 128 and the convolutional block with channel number 128 corresponding to source image A, and the pooling layer between the convolutional blocks with channel numbers 128 and 128 and the convolutional block with channel number 128 corresponding to source image B. These two hierarchical features are the same-level features, and after being fused through multiple fusion strategies, they are added to the features input to the middle MAF, and after addition, the common latent representation vector between the multi-modal data can be obtained.

[0076] Step S4, use the residual bottleneck attention mechanism to couple the common latent representation vector with the attention-aware feature to obtain a coupled feature map.

[0077] Using the residual bottleneck attention mechanism, through parallel processing of the channel attention mechanism and the spatial attention mechanism, and finally using the residual connection method to splice with the common latent representation vector.

[0078] In one example, the residual bottleneck attention mechanism is used to couple the common latent representation vector with the attention-aware feature to obtain a coupled feature map, which specifically includes:

[0079] Using the formula to determine the channel attention component of the common latent representation vector; where m c (f) is the channel attention component of the common latent representation vector, MLP is a multi-layer perceptron, AVGPOOL is an average pooling operation, MAXPOOL is a max pooling operation, f is the common latent representation vector, and σ is the Sigmod function;

[0080] Using the formula to determine the spatial attention component of the common latent representation vector; where m s (f) is the spatial attention component of the common latent representation vector, f 7×7 is a convolution operation with a convolution kernel size of 7×7;

[0081] According to the spatial attention component, the channel attention component, and the common latent representation vector, using the formula to determine the final output component f''', forming a coupled feature map;

[0082] where f' is the channel attention output, f'' is the spatial attention output, is element-wise multiplication, is element-wise addition.

[0083] The residual bottleneck attention mechanism is Figure 2 the Res-BAM in

[0084] Step S5, according to the coupled feature map and the multi-level features of each source image, use the multi-modal synthesis network to output the brain medical synthesis image of the target modality.

[0085] The multi-modal synthesis network includes a plurality of second hybrid attention fusion modules connected in sequence. The plurality of second hybrid attention fusion modules are used to fuse the same-level features of two source images through a variety of fusion strategies and then add them to the coupled feature map. The fusion process inside each second hybrid attention fusion module is the same as the fusion process inside the first hybrid attention fusion module, and the fusion steps inside the first hybrid attention fusion module can be referred to.

[0086] The multi-modal synthesis network densely connects the common latent feature representation vector and hierarchical features with multiple hybrid attention fusion modules, and serves as the generator in the generative adversarial network to generate target-modal images.

[0087] Figure 2 The shown multi-modal synthesis network also includes 3 hybrid attention fusion modules. Figure 2 The working process of the shown multi-modal synthesis network is as follows:

[0088] The hierarchical features input to the leftmost MAF are: the hierarchical features obtained from the pooling layer between the convolutional blocks with 32 and 32 channels and the convolutional blocks with 64 and 64 channels corresponding to source image A, and the hierarchical features obtained from the pooling layer between the convolutional blocks with 32 and 32 channels and the convolutional blocks with 64 and 64 channels corresponding to source image B. These two hierarchical features are of the same level. After being fused through various fusion strategies, they are added to the coupled feature map output by the residual bottleneck attention mechanism, and the result is input to the middle MAF.

[0089] The hierarchical features input to the middle MAF are: the pooling layer between the convolutional blocks with 64 and 64 channels and the convolutional blocks with 128 and 128 channels corresponding to source image A, and the pooling layer between the convolutional blocks with 64 and 64 channels and the convolutional blocks with 128 and 128 channels corresponding to source image B. These two hierarchical features are of the same level. After being fused through various fusion strategies, they are added to the features input to the leftmost MAF, and the result is input to the rightmost MAF.

[0090] The hierarchical features input to the rightmost MAF are: the pooling layer between the convolutional blocks with 128 and 128 channels and the convolutional block with 128 channels corresponding to source image A, and the pooling layer between the convolutional blocks with 128 and 128 channels and the convolutional block with 128 channels corresponding to source image B. These two hierarchical features are of the same level. After being fused through various fusion strategies, they are added to the features input to the middle MAF.

[0091] The multi-modal synthesis network also includes a convolutional layer. The output end of the second hybrid attention fusion module at the end of multiple sequentially connected second hybrid attention fusion modules is connected to the input end of the convolutional layer. The convolutional layer is used to obtain the brain medical synthesis image of the target modality based on the features finally output by all the second hybrid attention fusion modules. Taking Figure 2 the shown multi-modal synthesis network as an example, the convolutional layer includes a convolutional block with 32 and 32 channels and a convolutional layer with 1 channel.

[0092] During training, according to the generator in the generative adversarial network, an SSIM-L1 joint loss function is designed. The discriminator is used to discriminate between the target modality image and the true label to enhance the structural similarity and contrast of the generated image and reduce noise interference.

[0093] Using the formula:

[0094]

[0095] A generator loss function is formed to generate the target modality image:

[0096]

[0097] where μ x , μ y are the means of the source image x and the target image label y, σ x , σ y are the variances of the source image x and the target image label y, σ xy is the covariance of the source image x and the target image label y, ||*||1 is the L1 norm, and λ1, λ SSIM are hyperparameters. L G is the generator loss function, is the maximum likelihood estimation of the source image x in the dataset p data .

[0098] Discriminator loss function:

[0099]

[0100] where the discriminator is used to discriminate between the target modality image and the true label to enhance the structural similarity and contrast of the generated image and reduce noise interference.

[0101] The modality-specific feature extraction network, multi-modal fusion network, residual bottleneck attention mechanism, and multi-modal synthesis network are trained simultaneously.

[0102] The present invention utilizes the advantages of multiple hybrid attention fusion modules to determine the fusion weights for different modality images using multiple fusion strategies, then calculates the attention module weights to determine the final output feature vector; it uses a residual bottleneck attention module to obtain complementary information between multiple modality images, serially obtains the channel attention component and the spatial attention component through the channel attention mechanism and the spatial attention mechanism, and uses the residual connection method to splice the spatial attention output component with the original feature map to retain the important structural information in the original feature map. The present invention designs and proposes a method based on the SSIM-L1 joint loss function, defines a loss function that takes into account both the structural similarity SSIM and the L1 norm, and during the training process, uses hyperparameters for optimization, improving the structural similarity and contrast of the generated target modality images, especially in the lesion area, and achieving clearer texture structures and high-frequency details.

[0103] A brain medical image synthesis method provided by the present invention improves the anti-noise ability of the synthesized image, enhances the similarity and contrast of the synthesized image, and fully reflects the detailed information in the image.

[0104] To verify the performance of the brain medical image synthesis method based on the hybrid attention fusion module generative adversarial network of the present invention, multiple synthesized images such as Figure 4 are selected. The difference between the synthesized images of multiple methods and the target image is extracted to obtain a difference map as shown in Figure 5 . Among them, Figure 4 Real, Pix2Pix(T1), Pix2Pix(T2), CycleGAN(T1), CycleGAN(T2), and Hi-Net respectively represent the real image, the conditional generative adversarial network (cGAN) with the T1 modality as the input, the conditional generative adversarial network (cGAN) with the T2 modality as the input, the cycle generative adversarial network (CycleGAN) with the T1 modality as the input, the cycle generative adversarial network (CycleGAN) with the T2 modality as the input, the hybrid fusion network (Hi-Net), and the results synthesized by the algorithm of the present invention. To better analyze and compare the synthesis results, objective evaluation criteria such as peak signal-to-noise ratio (PSNR), normalized mean square error (NMSE), and structural similarity (SSIM) are calculated, and the obtained results are shown in Table 1. The synthesis results of the method of the present invention have increased in all objective evaluation indicators, which also effectively shows from an objective perspective that the fusion performance of the method of the present invention is superior to other various methods. As can be seen from Table 1, the algorithm proposed by the present invention is superior to other algorithms in terms of structural similarity and mean square error, and the synthesized image is closer to the real image and contains more effective information.

[0105] Table 1 Comparison of synthesis results

[0106] Methods PSNR NMSE SSIM <![CDATA[Pix2Pix(T1→Flair)]]> 23.4287±0.8203 0.0062±0.0014 0.8377±0.0334 <![CDATA[Pix2Pix (T2→Flair)]]> 22.5352±0.9580 0.0074±0.0018 0.8276±0.0346 <![CDATA[CycleGAN(T1→Flair)]]> 12.3739±0.8014 0.0986±0.0094 0.6377±0.0544 <![CDATA[CycleGAN (T2→Flair)]]> 12.3945±0.8203 0.0986±0.0095 0.6361±0.0551 Hi-Net 23.6031±0.8316 0.0059±0.0010 0.8541±0.0298 The present invention 25.1025±0.6371 0.0040±0.0005 0.8843±0.0252

[0107] The present invention uses a synthesis strategy of multiple hybrid attention fusion modules and a residual bottleneck attention mechanism. The hybrid attention module simultaneously uses three adaptive weighting strategies of addition, multiplication, and maximization, and combines with the self-attention mechanism, which can adaptively extract inter-layer fusion features from two source images A and B, that is, high-level semantic information and low-level fine-grained features. Furthermore, it solves the problem that the dual-modal medical image synthesis method based on sequential generative adversarial network and semi-supervised learning cannot effectively extract multi-level fusion features.

[0108] In current research, the attention mechanism has been widely applied, and good synthesis effects have been achieved whether it is a parallel-channel spatial attention mechanism or a serial-channel spatial attention mechanism. However, the rich structural information in the original image is often ignored. The present invention proposes a residual bottleneck attention mechanism. Based on the serial-channel attention mechanism and the channel attention mechanism, the original feature map is coupled with the output of the spatial attention mechanism in a residual connection manner, which can better retain the rich image information in the original image and reduce the degradation degree of the synthesized image. In addition, an SSIM-L1 joint loss function is proposed, and hyperparameters are used for tuning during network training. According to the feature information of the original image and the synthesized image, the values of the hyperparameters are adaptively adjusted to improve the structural similarity and contrast of the synthesized image, and finally a synthesized image with richer detail information and more complete basic information is obtained.

[0109] The embodiment of the present invention also provides a brain medical image synthesis system, including:

[0110] A brain medical source image pair acquisition module, configured to acquire a pair of brain medical source images to be synthesized; the pair of brain medical source images to be synthesized includes two source images of different modalities that are mutually registered;

[0111] A feature extraction module, configured to perform feature extraction on each source image by using a modality-specific feature extraction network to obtain multi-level features of each source image;

[0112] A multi-modal fusion module, configured to fuse and add the same-level features of the two source images by using a multi-modal fusion network to obtain a common latent representation vector between multi-modal data; the multi-modal fusion network includes a plurality of sequentially connected first hybrid attention fusion modules, and the first hybrid attention fusion module is configured to fuse the same-level features of the two source images through multiple fusion strategies;

[0113] A coupling module, configured to couple the common latent representation vector with the attention-aware feature by using a residual bottleneck attention mechanism to obtain a coupled feature map;

[0114] An image synthesis module, configured to output a brain medical synthesis image in a target modality according to the coupled feature map and the multi-level features of each source image by using a multi-modal synthesis network; the multi-modal synthesis network includes a plurality of second hybrid attention fusion modules connected in sequence, and the plurality of second hybrid attention fusion modules are configured to add the coupled feature map after fusing the same-level features of two source images through multiple fusion strategies.

[0115] The brain medical image synthesis system provided by the embodiments of the present invention has a similar working principle and beneficial effects to the brain medical image synthesis method described in the above embodiments, so details are not described herein again. For specific content, reference can be made to the introduction of the method embodiments above.

[0116] In addition, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the brain medical image synthesis method as described above is implemented.

[0117] When the computer program in the above-mentioned memory is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disc that can store program codes.

[0118] Further, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed, the brain medical image synthesis method as described above is implemented.

[0119] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple. For related parts, reference can be made to the description of the method part.

[0120] In this text, specific examples are used to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation on the present invention.

Claims

1. A method for synthesizing brain medical images, characterized in that, Including: Obtain a pair of brain medical source images to be synthesized; the pair of brain medical source images to be synthesized includes two source images of different modalities that are mutually registered; Use a modality-specific feature extraction network to extract features from each source image to obtain multi-level features of each source image; Use a multi-modal fusion network to fuse and add the same-level features of the two source images to obtain a common latent representation vector between the multi-modal data; the multi-modal fusion network includes a plurality of first hybrid attention fusion modules connected in sequence, and the first hybrid attention fusion module is used to fuse the same-level features of the two source images through a variety of fusion strategies; Use a residual bottleneck attention mechanism to couple the common latent representation vector with the attention-aware feature to obtain a coupled feature map; According to the coupled feature map and the multi-level features of each source image, use a multi-modal synthesis network to output a brain medical synthesis image of the target modality; the multi-modal synthesis network includes a plurality of second hybrid attention fusion modules connected in sequence, and the plurality of second hybrid attention fusion modules are used to fuse the same-level features of the two source images through a variety of fusion strategies and then add them to the coupled feature map.

2. The brain medical image synthesis method according to claim 1, wherein The first hybrid attention fusion module is used to fuse the same-level features of the two source images through a variety of fusion strategies, specifically including: According to the same-level features of two source images, the fusion weights are determined by using the formulas and through multiple fusion strategies; in the formula, Max represents addition between elements, multiplication between elements, and maximization between elements respectively, is the feature of the (n-1)-th layer of the two source images, is the weight for addition between elements, is the weight for multiplication between elements, F max is the weight for maximization between elements, is the tensor shape of the number of channels 3C, height H, and width W; Calculate the soft attention mechanism weight W using the formula W = σ(C3C4F); where C3 and C4 are the third and fourth convolutional layers, and σ is the Sigmod function; According to the soft attention mechanism weights and the fusion weights, use the formula to determine the final output F of the hybrid attention fusion module MAF ; where C5 is the fifth convolutional layer.

3. The brain medical image synthesis method according to claim 1, wherein The process of using the residual bottleneck attention mechanism to couple the common latent representation vector with the attention-aware feature to obtain a coupled feature map specifically includes: Using the formula m c (f) = σ(MLP(AVGPOOL(f))) + MLP(MAXPOOL(f)) to determine the channel attention component of the common latent representation vector; where m c (f) is the channel attention component of the common latent representation vector, MLP is a multi-layer perceptron, AVGPOOL is an average pooling operation, MAXPOOL is a max pooling operation, f is the common latent representation vector, and σ is the Sigmod function; Using the formula m s (f) = σ(f 7×7 ([AVGPOOL(f); MAXPOOL(f)])) to determine the spatial attention component of the common latent representation vector; where m s (f) is the spatial attention component of the common latent representation vector, and f 7×7 is a convolution operation with a convolution kernel size of 7×7; According to the spatial attention component, the channel attention component, and the common latent representation vector, use the formula to determine the final output component f”', and form a coupled feature map; Among them, f' is the output of channel attention, and f'' is the output of spatial attention, is element-wise multiplication, is element-wise addition.

4. The brain medical image synthesis method according to claim 1, characterized in that The multi-modal synthesis network further includes a convolutional layer; The output end of the second hybrid attention fusion module at the end of the plurality of second hybrid attention fusion modules connected in sequence is connected to the input end of the convolutional layer; the convolutional layer is used to obtain a brain medical synthesis image of the target modality according to the features finally output by all the second hybrid attention fusion modules.

5. The brain medical image synthesis method according to claim 1, characterized in that Before the step of obtaining the pair of brain medical source images to be synthesized, it further includes: Use a sample data set to train a modality-specific feature extraction network, a multi-modal fusion network, a residual bottleneck attention mechanism, and a multi-modal synthesis network; the sample data set consists of brain medical source image pair samples and corresponding brain medical synthesis image labels.

6. The brain medical image synthesis method according to claim 5, characterized in that, The training process of the modality-specific feature extraction network is: Construct a reconstruction network; the reconstruction network includes a deconvolutional layer; Use the reconstruction network to reconstruct the multi-level features of each source image into an image; Using the reconstruction loss function according to each source image and the corresponding reconstructed image train the modality-specific feature extraction network; wherein, is the i-th source image, is 's reconstructed image, ||*||1 is the L1 norm, is 's mean value, is 's mean value, is 's variance, is 's variance, is the covariance, L RS is the reconstruction loss function, L R is the L1-norm loss function, L ssim is the structural similarity loss function.

7. The brain medical image synthesis method according to claim 5, wherein The training process of the multi-modal synthesis network is: Establish a discriminator to discriminate between the target modality image and the true label; Determine the loss function of the discriminator as where L D is the loss function of the discriminator, D(x, y) is the boolean value obtained by the discriminator for discriminating the source image x and the target image label y, is the maximum likelihood estimation of the source image x in the dataset p data , is the maximum likelihood estimation of the source image x and the target image label y in the dataset p data , and D(x, G(x)) is the boolean value obtained by the discriminator for discriminating the source image and the generated target image; Establish an SSIM-L1 joint loss function as where L SSIM-L1 is the SSIM-L1 combined loss function, λ1, λ SSIM are the first and second hyperparameters, μ x , μ y are the means of the source image x and the target image label y, σ x , σ y are the variances of the source image x and the target image label y, σ xy is the covariance of the source image x and the target image label y, G(x) is the generated target image, and c1, c2 are the first and second positive constants; According to the SSIM-L1 combined loss function, the loss function of the generator in the multi-modal synthesis network is determined as where L G is the loss function of the generator in the multi-modal synthesis network; Use the loss function of the generator and the loss function of the discriminator in the multi-modal synthesis network to train the multi-modal synthesis network.

8. A brain medical image synthesis system, characterized in that, Including: A brain medical source image pair acquisition module for obtaining a pair of brain medical source images to be synthesized; the pair of brain medical source images to be synthesized includes two source images of different modalities that are mutually registered; A feature extraction module, which is used to extract features from each source image by using a modality-specific feature extraction network to obtain multi-level features of each source image; A multi-modal fusion module, which is used to fuse and add the same-level features of two source images by using a multi-modal fusion network to obtain a common latent representation vector between multi-modal data; the multi-modal fusion network includes a plurality of first hybrid attention fusion modules connected in sequence, and the first hybrid attention fusion module is used to fuse the same-level features of two source images through a variety of fusion strategies; A coupling module, which is used to couple the common latent representation vector with the attention-aware feature by using a residual bottleneck attention mechanism to obtain a coupled feature map; An image synthesis module, which is used to output a brain medical synthesis image of the target modality according to the coupled feature map and the multi-level features of each source image by using a multi-modal synthesis network; the multi-modal synthesis network includes a plurality of second hybrid attention fusion modules connected in sequence, and the plurality of second hybrid attention fusion modules are used to fuse the same-level features of two source images through a variety of fusion strategies and then add them to the coupled feature map.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the brain medical image synthesis method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored thereon, and when the computer program is executed, it implements the brain medical image synthesis method according to any one of claims 1 to 7.

Images