A reference-free multimodal medical fusion image quality assessment method based on DwG2NPAN

Through the DwG2NPAN method, the diffused GAN and attention mechanism are used to solve the problems of image details loss and noise distortion in multimodal medical image fusion, providing an objective quality evaluation standard, and improving image quality and generator performance.

CN118261899BActive Publication Date: 2024-11-22XUZHOU MEDICAL UNIVERSITY
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202410464575.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-17
Publication Date
2024-11-22
Estimated Expiration
2044-04-17

AI Technical Summary

Technical Problem

The existing multimodal medical image fusion technology can easily lead to image details loss, block effects and noise distortion during the fusion process, and lack objective quality evaluation standards, making it difficult for doctors to accurately evaluate the quality of the fusion image.

Method used

A reference-free multimodal medical fusion image quality evaluation method based on DwG2NPAN was designed, and a differential-perceived attention network (DwG2NPAN) guided by diffusion generative adversarial network (GAN) is used to generate high-quality attention network through diffusion model and attention mechanism. Fusion images provide objective standards for medical image quality evaluation.

Benefits of technology

Improve the objectivity and reliability of the evaluation of medical image fusion quality, enhance the generator's ability to capture medical image features, optimize the stability and generated image quality of the GAN network, and provide more accurate multimodal medical fusion image quality Evaluate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118261899B_ABST
    Figure CN118261899B_ABST
Patent Text Reader

Abstract

The present invention proposes a reference-free multimodal medical fusion image quality evaluation method based on DwG2NPAN, which relates to the field of medical image fusion. The method realizes accurate evaluation of the quality of medical fusion images by simulating the perceptual differences of the human visual system, and aims to solve the problem of lack of objective and reliable reference standards in the current medical fusion image quality evaluation process. The present invention mainly consists of a difference perception module guided by a diffusion model-assisted generative adversarial network and an attention-based quality assessment network. By constructing an innovative model named DwG2NPAN, the advantages of GAN in image generation are fully utilized, and the characteristics of the diffusion model are combined. With low-quality fusion images as input, high-quality fusion images are generated, providing an objective and reliable reference standard for the quality evaluation of medical fusion images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of medical image fusion, and in particular to a reference-free multimodal medical fusion image quality evaluation method based on DwG2NPAN. Background Art

[0002] In the fusion process of multimodal medical image fusion technology, multi-scale conversion, block feature extraction and fusion strategy often cause the loss of details of the fused image, resulting in distortions such as blur and block effect; in addition, the fusion rule of taking the largest and weighted average is likely to cause local blur, noise and deformation of the target in the fused image, and it is impossible to guarantee the ideal quality of medical fusion images. Due to the lack of unified objective quality evaluation standards in clinical practice, the selection of different fusion images still requires doctors to rely on their own experience and visual inspection. The existing objective fusion quality evaluation method evaluates the overall image, underestimates the impact of the local significant distortion of the lesion on the visual quality, and cannot accurately evaluate the quality of medical image fusion. Therefore, according to the characteristics of medical images themselves, it is of great significance to study objective quality evaluation methods that meet the visual characteristics of doctors to evaluate the pros and cons of fusion algorithms, guide the design of fusion algorithms, and improve the quality of fusion images. Summary of the invention

[0003] The present invention designs a reference-free multimodal medical fusion image quality assessment method based on DwG2NPAN, creatively proposes a diffusion-based generative adversarial network (GAN)-guided difference perception attention network (Diffusion with GAN-Guided Nuance Perceptual Attention Network, DwG2NPAN), which is specially designed for multimodal medical fusion image quality assessment. The present invention mainly consists of a diffusion module (Diffusion Module)-assisted generative adversarial network (GAN)-guided difference perception module and an attention-based quality assessment network (AQA). By constructing an innovative model named DwG2NPAN, the advantages of GAN in image generation are fully utilized, combined with the characteristics of the diffusion model, and low-quality fusion images are used as input to generate high-quality fusion images, providing an objective and reliable reference standard for the quality evaluation of medical fusion images.

[0004] A reference-free multimodal medical fusion image quality assessment method based on DwG2NPAN comprises the following steps:

[0005] Image information of each modality is collected, reference images are selected for each group of fused images, and a difference-aware attention network DwG2NPAN guided by a diffuse generative adversarial network (GAN) is constructed. The original medical image is input, and adaptive feature extraction and fusion are performed based on the difference-aware attention network DwG2NPAN to generate a high-quality fused image, and the comprehensive quality evaluation score of the original medical image is output.

[0006] Preferably, the fused image with the highest mean opinion score (MOS) of each image is selected as the reference image. If the MOS scores of multiple fused images are simultaneously the highest, one of them is randomly selected.

[0007] Furthermore, DwG2NPAN combines the diffusion model to optimize the adversarial network GAN to form a diffusion-based generative adversarial network DwG, which generates medical fusion images based on DwG and reduces the gap between the output of the initial generator and the target image.

[0008] Furthermore, the diffusion model consists of two processes: forward diffusion and reverse diffusion.

[0009] In the forward diffusion stage, Gaussian noise is gradually added to the fused image I org In this paper, a series of images with increasing noise levels are generated. org1 , ..., I orgT-1 , I orgT , where T is the total number of diffusion steps, these images with different noise levels will be used as input to the generator G;

[0010] In the reverse diffusion stage, we start from the high-noise image and gradually remove the noise to generate the same noisy image I as that generated in the forward diffusion process. org , I org1 , ..., I orgT-1 The corresponding new image I' org , I' org1 , ..., I' orgT-1 ;

[0011] The I in the forward propagation of the diffusion model orgt-1 with I orgt Pair them and input them as true image pairs to the GAN discriminator D. Correspondingly, I' generated by back propagation orgt-1 with I orgt As a fake image pair, it is input into the discriminator D of GAN to learn the difference between real images and generated images.

[0012] Furthermore, in the diffusion-based generative adversarial network DwG, the generator G generates new image data, and the discriminator D judges the authenticity of the generated image.

[0013] The objective function of the generator G is expressed as:

[0014]

[0015] Among them, L G represents the generator objective function, V G (G, D) represents the generator value function, which is used to measure the realism of the image generated by the generator G and the degree of misjudgment of the discriminator. N is the total number of generated samples, and I GT represents the reference image, P dataO Representative I orgt The data distribution, Representative I orgt The expectation of θ is a hyperparameter. During the training process of G, the generator objective function is minimized.

[0016] The objective function of the discriminator D is:

[0017]

[0018] Among them, L D represents the objective function of the discriminator, V D (G, D) represents the discriminator value function, which is used to evaluate the accuracy of the discriminator D in correctly identifying real images and generated images, P dataGT isI GT The data distribution, isI GT , and t represents the time step in the diffusion process.

[0019] Preferably, the DwG generator G adds a unique feature warehouse UFW module to identify image spatial features at multiple scales, and continuously fuses them in overlapping structures to extract effective spatial features, thereby enhancing the generator's ability to capture medical image features.

[0020] Furthermore, the unique feature warehouse UFW module includes a maximum pooling layer, an upsampling layer and a residual module: the maximum pooling layer includes a filter of size 2×2 and a moving step of 2, which reduces the amount of computation and controls overfitting by reducing the spatial dimension of the data, while retaining the edge and texture information of the medical image; the upsampling layer is composed of four layers of convolutional neural network, each layer of the convolutional neural network consists of an upsampling operation, a convolutional layer with a kernel size of 3×3, a BN layer and an activation layer, which are used to amplify the feature map and restore the image to a higher resolution; the residual module includes three convolutional layers and jump connections, the kernel of each convolutional layer is 3×3, and each layer is followed by batch normalization and LeakyReLU activation function, and the jump connection allows the input to be directly added to the output after convolution, thereby improving the information transfer of the neural network.

[0021] Furthermore, based on AQA to evaluate the quality of multimodal medical fusion images, the loss function of AQA can be expressed as:

[0022]

[0023] in, is the weight parameter, L AQA represents the loss function of AQA, L CAM represents the loss function of the CAM module, L QA Represents the objective function of fusion image quality evaluation.

[0024] Preferably, the objective function of the weighted optimization discriminator D is introduced:

[0025]

[0026] Among them, L D represents the discriminator value function with weight optimization, and W represents a specific weight, which is used to increase the model's attention to those medical fusion images that have little impact on the diagnosis results. The formula is as follows:

[0027]

[0028] Among them, I hq represents a high-quality fused image, represents the nth high-quality fused image used for evaluation, represents the evaluation score of the nth high-quality fused image, W n Represents the specific weight of the nth high-quality fused image.

[0029] Preferably, the total loss function is expressed as:

[0030]

[0031] Among them, Loss represents the total loss function of the DwG2NPAN network, min G max D V(G,D) represents the optimal result of the generator and discriminator under adversarial training, minL G represents the optimal result of the generator under adversarial training, maxL' D represents the optimal result of the discriminator under adversarial training, and γ represents L AQA Weight in the overall loss function.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention designs a reference-free multimodal medical fusion image quality assessment method based on DwG2NPAN, creatively proposes a diffusion-based generative adversarial network (GAN)-guided difference perception attention network (Diffusion with GAN-Guided Nuance Perceptual Attention Network, DwG2NPAN), which is specially designed for multimodal medical fusion image quality assessment. The present invention mainly consists of a diffusion module (Diffusion Module)-assisted generative adversarial network (GAN)-guided difference perception module and an attention-based quality assessment network (AQA). By constructing an innovative model named DwG2NPAN, the advantages of GAN in image generation are fully utilized, combined with the characteristics of the diffusion model, and low-quality fusion images are used as input to generate high-quality fusion images, providing an objective and reliable reference standard for the quality evaluation of medical fusion images.

[0034] 1. The GAN network of the present invention has the problem of asynchronous and unbalanced training of the generator and the discriminator during the training process, which leads to poor stability and affects the quality of the generated medical reference images. To address this problem, this paper innovatively introduces a diffusion model as an auxiliary mechanism. Through the adaptive diffusion process, the real image and the generated image can be effectively converted, thereby improving the stability of the model and optimizing the quality of the generated image.

[0035] 2. The present invention proposes a unique feature warehouse (UFW) module, which can identify image spatial features from multiple scales and continuously fuse them in overlapping structures to achieve effective spatial feature extraction, enhance the generator's ability to capture medical image features, and further improve the performance of the generator in the GAN network.

[0036] 3. The present invention proposes a loss function specifically for the quality perception of multimodal medical fusion images, provides a new solution for the generation and quality evaluation of medical fusion images, and more accurately evaluates the quality of multimodal medical fusion images. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is the overall framework of the DwG2NPAN model of the present invention;

[0038] Figure 2 It is the DWG network structure of the present invention;

[0039] Figure 3 It is the UFW module structure of the present invention;

[0040] Figure 4 This is the AQA network structure of the present invention. DETAILED DESCRIPTION

[0041] The following is a detailed description of a reference-free multimodal medical fusion image quality evaluation method based on DwG2NPAN of the present invention in conjunction with the accompanying drawings and specific implementation methods.

[0042] The present invention proposes a Diffusion with GAN-Guided Nuance Perceptual Attention Network (DwG2NPAN) which is specially designed for multimodal medical fusion image quality assessment. It is mainly composed of a Diffusion Module assisted by a Generative Adversarial Network (GAN)-guided Difference Perceptual Attention Network and an Attention-based Quality Assessment Network (AQA), such as Figure 1 As shown in the figure. The medical image is first generated into a high-quality image through the DWG network. In this process, the multi-scale features of the image are extracted through the UFW module and sent to the AQA network. The input data of the AQA network is the difference features between the original image and the generated high-quality image and the features extracted by the UFW module. The output of the network is the comprehensive quality evaluation score of the original image, which provides a reliable reference for medical diagnosis. In addition, through the application of the CAM module, the distribution of network attention is further supervised, and the interpretability of the model is enhanced through the feature map generated by it.

[0043] This paper proposes a method for generating medical fusion images based on a diffusion model combined with a generative adversarial network (GAN) (Diffusion with Generative Adversarial Network, DwG), aiming to solve the problem of lack of objective reference standards in the current field of medical fusion images. By constructing an innovative model named DwG2NPAN, the advantages of GAN in image generation are fully utilized, combined with the characteristics of the diffusion model, and low-quality fusion images are used as input to generate high-quality fusion images, providing an objective and reliable reference standard for the quality evaluation of medical fusion images.

[0044] The traditional GAN ​​network has the problem of asynchronous and unbalanced training of the generator and the discriminator during the training process, which leads to poor stability and affects the quality of the generated medical reference images. To address this problem, the present invention innovatively introduces a diffusion model as an auxiliary mechanism. Through the adaptive diffusion process, the real image and the generated image are effectively converted, thereby improving the stability of the model and optimizing the quality of the generated image. In addition, in order to further improve the performance of the generator in the GAN network, the present invention proposes a unique feature warehouse (UFW) module, which can identify image spatial features from multiple scales and continuously fuse them in overlapping structures to achieve effective spatial feature extraction and enhance the generator's ability to capture medical image features. At the same time, in order to more accurately evaluate the quality of multimodal medical fusion images, the present invention also proposes a loss function specifically for the quality perception of multimodal medical fusion images, providing a new solution for the generation and quality evaluation of medical fusion images. The details of the overall framework will be described in the following steps.

[0045] Step 1: Collect image information of each modality and complete image preprocessing.

[0046] Step 1.1: Establish a multimodal medical image fusion quality evaluation database

[0047] For 120 pairs of color images and 9 pairs of grayscale images, 10 mainstream fusion algorithms were used to generate 1290 images with a resolution of 128 × 128 pixels. The mean opinion score (MOS) of each image was obtained from radiologists, ranging from 1 to 5, where 5 means excellent image quality and does not affect the diagnosis; 4 means good image quality and can be diagnosed; 3 means poor image quality and affects diagnosis; 2 means poor image quality and obviously affects diagnosis; 1 means extremely poor image quality and cannot be diagnosed.

[0048] Step 1.2: Select a reference image

[0049] In order to select the reference image for each set of fused images (i.e., 10 images generated by a pair of images through 10 fusion algorithms), the present invention uses MOS to determine the image quality. The fused image with the highest MOS score is used as the reference image. If multiple fused images have the highest MOS score, one of them can be randomly selected. This ensures the robustness of the reference image selection process and fully considers the subjective quality evaluation of radiologists.

[0050] The DwG2NPAN model proposed in this invention is composed of DwG, AQA and UFW modules. Specifically:

[0051] Step 2, design DwG. Using DwG as the basis of the DwG2NPAN model, high-quality fused images are generated through low-quality fused images, providing an objective standard for the quality evaluation of fused images. In the traditional GAN ​​structure, the generator (Generator, G) is responsible for generating new image data, and the discriminator (Discriminator, D) is used to judge the authenticity of the generated image. However, the training process of GAN is extremely unstable. In order to solve this problem, the present invention proposes a DwG model, which uses the balance mechanism of the back diffusion process to effectively bridge the gap between the output of the initial generator and the target image, significantly improves the stability of GAN network training, and further optimizes the quality of generated images. In addition, the diffusion model introduces randomness in its generation process, which helps the GAN network generate more diverse outputs and enriches the generated medical reference image library. In the design of the generator, it consists of upsampling and downsampling operations, and the proposed UFW module is embedded between the two to enhance the generator's ability to capture medical image features. The structure of DwG is as follows Figure 2 shown.

[0052] The diffusion model is constructed by transforming the input image I org The two processes of forward diffusion and backward diffusion gradually generate a series of fused images with different noise levels. orgt As input, we aim to generate a fused image I with the best quality hq ,Right now D aims to distinguish between real fused images and generated fused images.

[0053] Step 2.1: Construct a diffusion model. Figure 2 As shown in Figure 1, the diffusion model consists of two core processes: forward diffusion and backward diffusion, which provide rich input data for GAN and alleviate the instability of GAN network training caused by the lack of feedback.

[0054] 2.1.1 Forward Diffusion: In this stage, Gaussian noise is gradually added to the clear fused image to generate a series of images I with increasing noise levels. org1 , ..., I orgT-1 , I orgT , where T is the total number of diffusion steps. Considering the balance between image generation speed and computational complexity, T is set to 8. These images with different noise levels will be used as input to the generator G.

[0055] 2.1.2 Backward diffusion: Backward diffusion is the process of gradually denoising an image. Starting from a high-noise image, the noise is gradually removed to generate a noisy image I generated in the forward diffusion process. org , I org1 , ..., IorgT-1 The corresponding new image I' org , I' org1 , ..., I' orgT-1 .

[0056] Step 2.2: Design the generator G. Figure 2 As shown in Figure 1, G is a convolutional neural network consisting of downsampling and upsampling operations. Since medical fusion images require the network to focus on more detailed features, the convolution kernel size of the convolution operation in the generator is all set to 3×3.

[0057] 2.2.1 Downsampling operation: It consists of three sequence networks with the same structure. This sequence network consists of a convolutional layer, an activation function, a batch normalization (BN) and a residual module. For the activation function, a leaky linear rectifier (LeakyReLU) function is used. In order to make the gradient back propagate more efficiently, the present invention adds a BN layer after the activation function, and uses a residual structure in the downsampling process, which consists of three convolutional layers and a jump connection. The residual structure allows the gradient to propagate more smoothly during training, which not only improves the training efficiency of the model, but also helps to extract richer and more accurate feature information.

[0058] 2.2.2UFW module: In view of the key role of effective spatial feature extraction technology in the quality assessment of medical fusion images, the present invention proposes the UFW module. This module can efficiently extract the texture and shape features of medical images. The UFW module consists of a maximum pooling layer, an upsampling layer, and a residual module. By performing two downsampling and two upsampling operations on the original feature map, the extraction and fusion of different required features are realized. The detailed architecture of UFW is shown in Figure 3As shown in the figure. The maximum pooling layer includes a filter of size 2×2 and a moving step of 2. It reduces the amount of calculation and controls overfitting by reducing the spatial dimension of the data, while retaining the edge and texture information of the medical image. The upsampling layer consists of four layers of convolutional neural networks. Each layer of the convolutional neural network consists of an upsampling operation, a convolutional layer with a kernel size of 3×3, a BN layer, and an activation layer, which is used to enlarge the feature map and restore the image to a higher resolution. The residual module includes three convolutional layers and jump connections. The kernel of each convolutional layer is 3×3, and each layer is followed by batch normalization and LeakyReLU activation function. The jump connection allows the input to be directly added to the output after convolution, improving the information transfer of the neural network. The unique feature warehouse UFW module adopts a plug-and-play design and can be easily embedded in the existing network architecture in a serial manner to effectively process the input features while ensuring that the input and output sizes are consistent. This module can be seamlessly integrated between specific layers of the network, thereby enhancing the feature extraction and processing capabilities of various different models. In addition, the number of UFW modules can be selectively added according to different application requirements. This module not only enhances the network's ability to capture details and efficiently extracts texture and shape features of medical images, but also helps to improve the performance and generalization ability of the overall network.

[0059] 2.2.3 Upsampling operation: It consists of four layers of convolutional neural network. Each layer of the convolutional neural network consists of an upsampling operation, a convolutional layer with a kernel size of 3×3, a BN layer and an activation layer. As for the activation function, the present invention uses the LeakyReLU activation function in the first three layers and the Sigmoid activation function in the last layer.

[0060] 2.2.4 Design loss function: The generator objective function is expressed as:

[0061]

[0062] Where N is the total number of generated samples. GT P represents the reference image, i.e. the ground truth (GT). dataO Representative I orgt The data distribution, Representative I orgt θ is a hyperparameter and is set to 0.5. During the training of G, the objective function is minimized.

[0063] Step 2.3: Design the discriminator D. The architecture of the discriminator is a simple four-layer convolutional neural network. Its uniqueness lies in the construction of the input, specifically:

[0064] 2.3.1 I in the forward propagation of the diffusion model orgt-1 with Iorgt Pair them and input them into D as true image pairs. Correspondingly, I' generated by back propagation orgt-1 with I orgt As fake image pairs, they are input into D. This pairing helps D better learn the difference between real and generated images.

[0065] The convolution operation kernel size of each layer of 2.3.2D is 3×3, the stride is 2, and the padding is 1. LeakyReLU is then used as the activation function, followed by BN. For each convolution operation, the size of the feature map is reduced to one-fourth of the input, such as Figure 2 shown.

[0066] 2.3.3Finally, the present invention adds an independent convolutional layer according to the sequential structure for classification operation.

[0067] 2.3.4 Design loss function: The objective function of the discriminator can be expressed as:

[0068]

[0069] Where P dataG isI GT The data distribution, isI GT expectations.

[0070] Step 3: Establish AQA. AQA is established based on the DWG network. The nature of this network makes it easy to modify its network architecture according to needs. Due to the guidance of DWG, AQA does not need to extract the feature representation of the fused image from scratch. Therefore, it is not necessary to adopt a complex network structure. Figure 4 .

[0071] Step 3.1: Calculate the fused image I org and high quality fused image I hq The present invention uses the fused image with higher MOS as the GT of the generator, thereby limiting its fitting trend. org and higher quality fused image I hq The subtle difference between sub =|I hq -I org |. To indicate the total number of evaluations and to indicate the nth evaluation image, the definition can be modified as:

[0072]

[0073] To ensure the consistency of input, and First, perform convolution operations separately, and then complete the splicing operation.

[0074] Step 3.2: Concatenate feature maps. The feature representation extracted in DWG can guide AQA to obtain more convincing quality evaluation results. Therefore, the present invention introduces the feature map F output by the last UFW module last , concatenated before inputting the fully connected layer to obtain the feature map F conca ,as follows:

[0075]

[0076] where vgg(·) represents the operation of AQA before the fully connected layer.

[0077] So far, basic evaluation of image quality can be achieved by the above methods.

[0078] Step 3.3: Introduce the attention mechanism.

[0079] 3.3.1 Introduction of Class Activation Mapping (CAM): By using the weights of the last fully connected layer, the present invention introduces CAM. With the help of the quality score of AQA and the weights of the fully connected layer, CAM has the ability to supervise the distribution of network attention. In addition, the feature map F generated by CAM CAM It can also make up for the uninterpretability of the model.

[0080] 3.3.2 Design loss function: CAM’s GT is F GT , the objective function is:

[0081]

[0082] Among them, L CAM represents the loss function of the CAM module, and |·|1 represents the L1 norm.

[0083] The prediction score of the fused image is denoted as Q pre , whose GT is Q t , then the objective function can be expressed as:

[0084] L QA =-[Q t log(σ(Q pre ))+(1-Q t )log(1-σ(Q pre ))] (6)

[0085] Where σ(·) represents the Sigmoid function, that is, Q pre Mapped to the interval (0,1), it is expressed as:

[0086]

[0087] Therefore, the loss function of AQA can be expressed as:

[0088]

[0089] L QA represents the objective function of fusion image quality evaluation, is the weight parameter, which is set to 1 after experimental verification.

[0090] Step 4: Design a perceptual loss function. The ultimate goal of the DwG2NPAN model proposed in this invention is to accurately evaluate the quality of the fused image, rather than to obtain the fused image. It can be clearly seen from the calculation formula that the evaluation result depends largely on the generated image I hq .if If it is directly used as the input of AQA without feedback to DWG, the training process of the quality evaluation network will be unstable and difficult to converge. Therefore, the present invention designs a perceptual loss function (as shown in Formula 11) to address the above problems. This function makes full use of the real and false image pairs generated by the diffusion model as input data, and through a unique weight optimization strategy, it combines the dual requirements of the generation model and the quality evaluation network, effectively enhancing the stability of network training. The specific function is explained as follows:

[0091] Step 4.1: Weight optimization strategy: In medical practice, when the mean subjective score (MOS) is greater than 3, the medical fusion image is generally considered to have no negative impact on the diagnosis result. Based on this key indicator, the present invention defines specific weights to ensure that the model pays more attention to images with less negative impact on the diagnosis result during training, as shown in formula (9):

[0092]

[0093] in, represents the nth high-quality fused image used for evaluation, represents the evaluation score of the nth high-quality fused image, W n Represents the specific weight of the nth high-quality fused image.

[0094] The loss function (2) of the discriminator is further optimized by introducing weights as follows:

[0095]

[0096]

[0097] After experimental verification, γ was finally set to 0.01, which improved the accuracy of evaluating the quality of medical fusion images while ensuring the stability of the model.

[0098] 2 Experiments

[0099] In order to evaluate the effectiveness of the proposed method, the present invention evaluates the performance of the model based on the following experiments, including experimental settings, evaluation methods and experimental results.

[0100] Step 1: Experimental setup and evaluation method

[0101] Step 1: Experimental environment and settings.

[0102] During the network training process, after experiments, the three hyperparameters θ, γ is finally set to 0.5, 1, and 0.01 respectively. The Adam optimizer is used with an initial learning rate of 0.0002. In addition, the present invention implements a dynamic learning rate adjustment strategy to enhance the model convergence during training. Specifically, the present invention uses a decay factor of 0.95 to reduce the learning rate after every 20 batches.

[0103] In order to evaluate the effectiveness of the proposed model, the present invention adopts a five-fold cross validation method. The performance of the model is evaluated after each training batch, and the best performing model is selected as the final model among 1500 training batches. In the validation phase, the performance of the model on the test set is evaluated. Each evaluation is repeated 1000 times, and the average of the evaluation indicators obtained is taken.

[0104] Step 2: Evaluate the solution.

[0105] In order to comprehensively evaluate the performance of the model, the present invention adopts four classic evaluation indicators: Spearman's Rank Correlation Coefficient (SRCC), Kendall's Rank Correlation Coefficient (KRCC), Pearson Product-Moment Correlation Coefficient (PLCC), and Root-Mean-Square Error (RMSE). Among them, SRCC and KRCC are the two most commonly used rank correlation indicators, which can be used to measure the monotonicity between the model prediction results and the MOS score. The higher the value, the better the correlation between the two sets of data, and the smaller the value, the worse the correlation between the two sets of data.

[0106] Step 2: Evaluation results of G2NPAN

[0107] Step 1: Compare the performance of other models. In order to prove the effectiveness of the proposed method, the present invention compares the following methods (for methods without specific names, the name of the first author is used to refer to them). Including Blind Multiple Pseudo Reference Images-Based Method (BMPRI), In-Depth Analysis of Tsallis Entropy-Based Method (TEIA), Mutual Information-Based Optimization Method (Hossny), Objective Evaluation of Fusion Performance (OEFP), Ratio of Spatial Frequency Error-Based Method (rSFe), Perceptual Quality Assessment Method (Tang). The results are listed in Table 1:

[0108] Conclusion: From Table 1, we can see that the results of PLCC, SRCC, KRCC and RMSE are 0.9044, 0.9007, 0.8502 and 0.1029 respectively, which means that the method proposed in this paper achieves the best performance. This means that the objective evaluation results obtained by G2NPAN are closest to the subjective MOS results given by doctors.

[0109] Secondly, although the BMPRI method introduces pseudo-reference images to provide reference information for quality evaluation, it is mainly aimed at distorted images of natural scenes. Therefore, it performs poorly in the field of medical fusion images. The method proposed in the present invention generates reference information based on high-quality fusion images, and designs a quality assessment method based on the specificity of medical images to achieve the best performance. As can be seen from Table 1, BMPRI is also better than some quality evaluation indicators designed specifically for fusion images, which once again illustrates the importance of reference information for quality evaluation of reference-free images.

[0110] Finally, rSFe, Tang and the proposed method consider the differences between medical fusion images and natural fusion images, so their performance is better than that of TEIA, Hossny and OEFP.

[0111] From the above analysis, it can be seen that the DwG2NPAN method proposed in this invention has advantages in objective quality assessment of medical fusion images.

[0112] Table 1 Performance comparison with other quality evaluation methods

[0113]

[0114] Step 2: Ablation experiment of the model. The present invention conducts ablation experiments from different perspectives to demonstrate the superiority of the proposed method.

[0115] 2.1 Performance of Backbone Networks. In order to verify the universality of the proposed AQA, the present invention uses different backbone networks to build models and test model performance. The results are listed in Table 2.

[0116] Conclusion: On the one hand, from these results, we can note that VGG11 performs even better than VGG16 and VGG19. This seems to go against the idea that the deeper the network, the better the model performance. However, a more detailed analysis should be performed for different task types. In fact, VGG19 and VGG16 have more convolutional layers than VGG11, which allows the network to learn more semantic information (such as deep features). However, the model evaluates multimodal medical fusion images and does not need to identify what the image represents, such as what disease or which organ, etc., but what is in the image. Therefore, quality assessment tasks may require more structure than image semantic information (such as shallow features). Compared with VGG19, VGG11 has improved in PLCC, SRCC, KRCC, and RMSE.

[0117] On the other hand, in order to demonstrate the practicality of CAM and pre-trained models, the present invention makes adjustments based on the VGG11 network. From the experimental results, it can be seen that both CAM and the introduced pre-trained model enhance the performance of the quality prediction network.

[0118] Table 2 Ablation experiment results

[0119]

[0120] 2.2 Performance of key modules. In order to evaluate the performance of each module in the DwG2NPAN model, the present invention conducted an ablation experiment. The method proposed in the present invention integrates the differences between fused images of different qualities and modifies the loss function according to the scoring criteria for medical images in the clinical environment. Therefore, it is necessary to fully explore the performance of each key module. The present invention uses the VGG11 network as the backbone network, and names the DwG-guided quality assessment models with and without the UFW module as DwG2N (Diffusion with GAN-Guided Network, DwG2N) and DwG2NN (Diffusion with GAN-Guided Nuance Network, DwG2NN), respectively. In addition, the present invention redesigns the perceptual loss function of DwG, and the corresponding network is named DwG2NPN (Diffusion with GAN-Guided Nuance Perceptual Network, G2NPN). Finally, CAM is added to the DwG2NPN model to supervise the quality prediction results of the fused image, and the network is named DwG2NPAN (Diffusion with GAN-Guided Nuance Perceptual Attention Network, G2NPAN), which is the model proposed by the present invention. The present invention uses the same training method to train DwG2N, DwG2NN and DwG2NPN, and Table 3 shows the corresponding prediction results.

[0121] Conclusion: First, under the reference information provided by DwG, the DwG2N model achieves the greatest performance improvement over VGG11. The DwG2N model generates fused images of better quality, which is equivalent to providing reference images for IQA, and has the most significant performance improvement. However, the amount of difference information in the reference information may not be enough. Since UFW can extract subtle differences in spatial features by learning multi-scale fused image features multiple times. Therefore, the DwG2NN model further improves the model performance.

[0122] Secondly, since DwG has the ability to identify the quality of fused images, that is, perception ability, the performance of the DwG2NPN model is greatly improved, especially in SRCC and KRCC. Finally, by introducing CAM, the DwG2NPAN proposed in this paper achieves the best performance in medical fusion image quality assessment.

[0123] In summary, the present invention finds that, whether from the bar graph analysis or from the data analysis, the method proposed by the present invention can effectively improve the quality evaluation results. Except for RMSE, the improvement of the other three indicators is more than 10%. Similarly, the model with reference information provided by DwG is better than all the methods in Table 2.

[0124] Table 3 Ablation experiment results of each key module in DwG2NPAN

[0125]

[0126] Step 3: Dataset Experiment

[0127] Medical image analysis has always faced the challenge of small data samples, which are not enough to provide sufficient training samples for learning-based models. In order to study the relationship between the training sample size and the performance of the proposed method, the training sample size is gradually increased from 20% to 80%, while the remaining images are used as tests.

[0128] Conclusion: Obviously, as the number of training samples increases, the performance of the model proposed in the present invention gradually increases. When the number of training samples decreases, the model performance does not drop sharply. This proves that the DwG2NPAN quality evaluation method proposed in the present invention has good robustness.

[0129] The above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A reference-free multimodal medical fusion image quality assessment method based on DwG2NPAN, characterized in that: The steps include: Collect image information of each modality, select reference images for each group of fused images, and construct a difference-aware attention network DwG2NPAN guided by a diffuse generative adversarial network (GAN). Input the original medical image, perform adaptive feature extraction and fusion based on the difference-aware attention network DwG2NPAN, generate a high-quality fused image, and output the comprehensive quality evaluation score of the original medical image. Among them, the difference-aware attention network DwG2NPAN is composed of a medical fusion image generation network DwG based on a combination of a diffusion model and a generative adversarial network GAN, an attention-based quality assessment network AQA, and a unique feature warehouse UFW module; DwG is a generative adversarial network (GAN) assisted by a diffusion model. The diffusion model introduces randomness in its generation process, helping the GAN network to generate more diverse outputs, and generating high-quality fused images with low-quality fused images as input. The medical image is first generated with high quality through the DwG network. During this process, the multi-scale features of the image are extracted through the UFW module and sent to the AQA network. The design of the DwG generator consists of upsampling and downsampling operations, and the UFW module is embedded between the two. The UFW module consists of a maximum pooling layer, an upsampling layer, and a residual module. By performing two downsampling and two upsampling operations on the original feature map, the extraction and fusion of different required features are realized. The input data of the AQA network is the difference features between the original image and the generated high-quality image, as well as the features extracted by the UFW module, and the output is the comprehensive quality evaluation score of the original image.

2. The method for evaluating the quality of a non-reference multimodal medical fusion image based on DwG2NPAN according to claim 1, characterized in that: The fused image with the highest mean opinion score (MOS) of each image is selected as the reference image. If the MOS scores of multiple fused images are simultaneously the highest, one of them is randomly selected.

3. The method for evaluating the quality of a non-reference multimodal medical fusion image based on DwG2NPAN according to claim 1, characterized in that: DwG2NPAN combines the diffusion model to optimize the adversarial network GAN to form a diffusion-based generative adversarial network DwG, which generates medical fusion images based on DwG and reduces the gap between the output of the initial generator and the target image.

4. The method for evaluating the quality of a non-reference multimodal medical fusion image based on DwG2NPAN according to claim 3 is characterized in that: The diffusion model consists of two processes: forward diffusion and reverse diffusion. In the forward diffusion stage, Gaussian noise is gradually added to the fused image I org In this paper, a series of images with increasing noise levels are generated. org1 , ..., I orgT-1 , I orgT , where T is the total number of diffusion steps, these images with different noise levels will be used as input to the generator G; In the reverse diffusion stage, starting from the high-noise image, the noise is gradually removed to generate the same noise as that generated in the forward diffusion process. The new image I′ corresponding to the acoustic image org , I′ org1 , ..., I′ orgT-1 ; The I in the forward propagation of the diffusion model orgt-1 with I orgt Pair them and input them as true image pairs to the GAN discriminator D. Correspondingly, the back-propagation generated I o ' rgt-1 with I orgt As a fake image pair, it is input into the discriminator D of GAN to learn the difference between real images and generated images.

5. The method for evaluating the quality of a non-reference multimodal medical fusion image based on DwG2NPAN according to claim 4 is characterized in that: In the diffusion-based generative adversarial network DwG, the generator G generates new image data, and the discriminator D judges the authenticity of the generated image. The objective function of the generator G is expressed as: Among them, L G represents the generator objective function, V G (G, D) represents the generator value function, which is used to measure the realism of the image generated by the generator G and the degree of misjudgment of the discriminator. N is the total number of generated samples, and I GT represents the reference image, P dataO Representative I orgt The data distribution, Representative I orgt The expectation of θ is a hyperparameter. During the training process of G, the generator objective function is minimized. The objective function of the discriminator D is: Among them, L D represents the objective function of the discriminator, V D (G, D) represents the discriminator value function, which is used to evaluate the accuracy of the discriminator D in correctly identifying real images and generated images, P dataGT isI GT The data distribution, isI GT , and t represents the time step in the diffusion process.

6. The method for evaluating the quality of a non-reference multimodal medical fusion image based on DwG2NPAN according to claim 5, characterized in that: In the DwG generator G, a unique feature warehouse UFW module is added to identify image spatial features at multiple scales, and continuously fuse them in overlapping structures to extract effective spatial features, thereby enhancing the generator's ability to capture medical image features.

7. The method for evaluating the quality of a non-reference multimodal medical fusion image based on DwG2NPAN according to claim 6, characterized in that: The unique feature warehouse UFW module includes a maximum pooling layer, an upsampling layer, and a residual module: the maximum pooling layer includes a filter of size 2×2 and a moving step of 2, which reduces the amount of calculation and controls overfitting by reducing the spatial dimension of the data, while retaining the edge and texture information of the medical image; the upsampling layer consists of a four-layer convolutional neural network, each layer of which consists of an upsampling operation, a convolutional layer with a kernel size of 3×3, a BN layer, and an activation layer, which is used to amplify the feature map and restore the image to a higher resolution; The residual module consists of three convolutional layers and skip connections. The kernel of each convolutional layer is 3×3, and each layer is followed by batch normalization and LeakyReLU activation function. The skip connection allows the input to be directly added to the output after convolution, improving the information transfer of the neural network.

8. The method for evaluating the quality of a non-reference multimodal medical fusion image based on DwG2NPAN according to claim 7, characterized in that: Based on AQA to evaluate the quality of multimodal medical fusion images, the loss function of AQA can be expressed as: in, is the weight parameter, L AQA represents the loss function of AQA, L CAM represents the loss function of the CAM module, L QA Represents the objective function of fusion image quality evaluation.

9. The method for evaluating the quality of a non-reference multimodal medical fusion image based on DwG2NPAN according to claim 8, characterized in that: Introduce the objective function of the weight optimization discriminator D: Among them, L' D represents the discriminator value function with weight optimization, and W represents a specific weight, which is used to increase the model's attention to those medical fusion images that have little impact on the diagnosis results. The formula is as follows: Among them, I hq represents a high-quality fused image, represents the nth high-quality fused image used for evaluation, represents the evaluation score of the nth high-quality fused image, W n Represents the specific weight of the nth high-quality fused image.

10. The method for evaluating the quality of a non-reference multimodal medical fusion image based on DwG2NPAN according to claim 9, characterized in that: The total loss function is expressed as: Among them, Loss represents the total loss function of the DwG2NPAN network, min G max D V(G,D) represents the optimal result of the generator and discriminator under adversarial training, minL G represents the optimal result of the generator under adversarial training, maxL' D represents the optimal result of the discriminator under adversarial training, and γ represents L AQA Weight in the overall loss function.

Citation Information

Patent Citations

  • Multimodal medical image fusion model and method based on knowledge distillation and interpretability

    CN115880554A

  • Image generation method based on diffusion model and generative adversarial network

    CN116563399A

  • Quality perception network and attention-like Siamese network collaborative medical fusion image quality evaluation method

    CN117726614A