A multimodal medical image fusion method based on feature aggregation and decomposition

Abstract

The present invention discloses a multimodal medical image fusion method based on feature aggregation and decomposition, comprising: constructing a feature aggregation network and a decomposition feedback network; splicing a source image and an initial decomposition image to form a training set; inputting the spliced ​​image into the feature aggregation network to output a fused image; inputting the fused image into the decomposition feedback network to obtain two decomposed images; calculating a total loss based on the source image, the fused image, and the decomposition image, and updating the feature aggregation network and the decomposition feedback network based on the total loss; if the iterative training end condition is met, outputting the feature aggregation network and the decomposition feedback network for multimodal medical image fusion; otherwise, splicing the source image and the decomposition image to form a new training set for continued iterative training. The present invention solves the problem of insufficiently detailed image feature extraction and lack of information compensation, and sets the fused image as an intermediate result, achieving better fusion performance by optimizing the end result.

Description

Technical Field

[0001] The present invention belongs to the technical field of image fusion, and in particular relates to a multimodal medical image fusion method based on feature aggregation and decomposition. Background Art

[0002] Multimodal medical image fusion is a subdivision of image fusion tasks. Unlike natural images, medical image datasets are much smaller, so it is not possible to use large datasets to pre-train models. In other words, the means of extracting medical image features are relatively scarce, so the extraction of medical image features requires more detailed analysis. However, existing methods for feature extraction of medical images still have many shortcomings. For example, multimodal medical image fusion based on convolution operations, although convolution operations have excellent computational performance and local context extraction capabilities, this limits the ability to model long-range image dependencies. In other words, different areas of the image cannot be connected. The result is a lack of image information, which leads to poor fusion performance.

[0003] To address this shortcoming, some methods use the Transformer to fuse multimodal medical images. The Transformer boasts powerful global context awareness, effectively modeling long-range image dependencies and focusing on more important features. However, it also has some drawbacks. First, it overemphasizes certain key features, neglecting local features. Second, the Transformer is a model with a large number of parameters, and using it exclusively for multimodal medical image fusion would result in an excessively high computational burden.

[0004] Combining the advantages of convolution and Transformer, serial CNN-Transformer-based methods can address this problem to a certain extent. However, this serial feature extraction method focuses more on the connection between local and global features, which inevitably leads to information loss. Furthermore, because convolution inevitably causes information loss in images, existing methods aim to minimize this loss, but rarely consider information compensation. Secondly, most methods follow a process of feature extraction-feature fusion-image reconstruction, with the fused image being the final destination. Improvements to fusion methods are limited to this process, without going beyond the established thinking. Summary of the Invention

[0005] The purpose of the present invention is to provide a multimodal medical image fusion method based on feature aggregation and decomposition, to solve the problems of insufficiently detailed image feature extraction and lack of information compensation, and to set the fused image as an intermediate result, so as to achieve better fusion performance by optimizing the endpoint result.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A multimodal medical image fusion method based on feature aggregation and decomposition, comprising:

[0008] Constructing a feature aggregation network and a decomposition feedback network, wherein the feature aggregation network includes an encoder, a global feature extraction module, a local feature extraction module and a decoder;

[0009] Take the source image and the initial decomposition image and stitch them together to form a training set;

[0010] The spliced ​​image is input into the feature aggregation network, which extracts shallow features through an encoder, and inputs the shallow features into a global feature extraction module and a local feature extraction module respectively to obtain global features and local features. The global features and local features are superimposed and then input into a decoder, which reconstructs and outputs a fused image;

[0011] Inputting the fused image into the decomposition feedback network to obtain two decomposed images;

[0012] The total loss is calculated based on the source image, the fused image and the decomposed image, and the feature aggregation network and the decomposition feedback network are updated according to the total loss. If the conditions for the end of iterative training are met, the feature aggregation network and the decomposition feedback network are output for multimodal medical image fusion; otherwise, the source image and the decomposed image are spliced ​​to form a new training set to continue iterative training.

[0013] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution. They are merely further supplements or optimizations. Under the premise that there are no technical or logical contradictions, each optional method can be combined separately for the above-mentioned overall solution, or multiple optional methods can be combined.

[0014] Preferably, the encoder consists of an adaptive convolution layer, a batch normalization layer and a ReLU activation function.

[0015] Preferably, the global feature extraction module includes three paths, and the multi-scale features output by the three paths are fused at the pixel level to obtain the global feature;

[0016] The first path includes an adaptive convolution layer, a Swin Transformer block, and an adaptive convolution layer connected in sequence. The second path includes an adaptive convolution layer, an adaptive convolution layer, a Swin Transformer block, and an adaptive convolution layer connected in sequence. The third path includes an adaptive convolution layer, an adaptive convolution layer, an adaptive convolution layer, a Swin Transformer block, and an adaptive convolution layer connected in sequence.

[0017] Preferably, the local feature extraction module includes three 3×3 convolutional layers and one 1×1 convolutional layer connected in sequence;

[0018] The input of the first 3×3 convolution layer is the shallow features and the source image, the input of the second 3×3 convolution layer is the shallow features, the source image and the output of the first 3×3 convolution layer, the input of the third 3×3 convolution layer is the shallow features, the source image, the output of the first 3×3 convolution layer and the output of the second 3×3 convolution layer, and the input of the 1×1 convolution layer is the shallow features, the output of the first 3×3 convolution layer, the output of the second 3×3 convolution layer and the output of the third 3×3 convolution layer.

[0019] Preferably, the source image and the initial decomposed image are spliced ​​together to form a training set, including:

[0020] Acquire PET-MRI image pairs as source images, where PET is an RGB color image and MRI is a grayscale image;

[0021] Convert the RGB color image to YUV color space and take the Y channel of the converted image Pairing images with grayscale images;

[0022] The initial decomposition image is taken as a copy of the source image, that is, the initial decomposition image is the paired Image and grayscale image, channel cascade the source image and the initial decomposition image to obtain the four-channel input feature sample I MRI is a grayscale image.

[0023] Preferably, the decomposition feedback network includes a 1×1 convolutional layer and two branches connected to the 1×1 convolutional layer, the two branches do not share parameters, wherein the first branch includes three 3×3 convolutional layers connected in sequence, and the second branch includes three 3×3 convolutional layers connected in sequence.

[0024] Preferably, the calculating the total loss based on the source image, the fused image and the decomposed image comprises:

[0025] A feature aggregation loss is calculated according to the source image and the fused image, a decomposition feedback loss is calculated according to the source image and the decomposition image, and the feature aggregation loss and the decomposition feedback loss are summed to obtain a total loss.

[0026] Preferably, the calculating feature aggregation loss according to the source image and the fused image comprises:

[0027] The feature aggregation loss L FA By the structural similarity loss L SSIM and regional mutual information loss L RMI composition:

[0028] L FA =L SSIM +L RMI

[0029] The structural similarity loss L SSIM It is a structural constraint, and its mathematical expression is:

[0030] L SSIM =β1·(1-SSIM(I f ,I1))+β2·(1-SSIM(I f ,I2))

[0031] Where β1 and β2 are trade-off parameters, I f represents the fused image, I1 and I2 represent the source images, and SSIM(.) represents the structural similarity index operation, which is specifically defined as:

[0032]

[0033] Where, I s Indicates the source image, specifically I1 or I2, and are the pixel averages of the source image and the fused image, and are the pixel variances of the source image and the fused image, is the pixel covariance of the source image and the fused image, C1 and C2 are constants;

[0034] The region mutual information loss L RMI is a regional constraint, and its mathematical expression is:

[0035] L RMI =γ1·RMI(I f ,I1)+γ2·RMI(I f ,I2)

[0036] Where γ1 and γ2 are trade-off parameters, and RMI(·) is the regional mutual information operation for calculating the fused image and the source image, which is specifically defined as:

[0037]

[0038] Where λ is the weight parameter, λ∈[0,1], represents the calculation of the standard cross entropy loss between the fused image and the source image, B represents the batch size, It means finding the lower bound of the mutual information between the fused image and the source image corresponding to the batch index b in the mini-batch.

[0039] Preferably, the calculating the decomposition feedback loss according to the source image and the decomposition image comprises:

[0040] The decomposition feedback loss L DF Including strength loss L int and gradient loss L grad , mathematically expressed as:

[0041] L DF =L int +L grad

[0042] The strength loss L int is defined as:

[0043]

[0044] Where H and W represent the height and width of the source image or decomposed image, i and j represent the i-th row and j-th column, Represents the decomposed image I de1 The pixel value of the i-th row and j-th column in , Represents the decomposed image I de2 The pixel value of the i-th row and j-th column in , represents the pixel value of the i-th row and j-th column in the source image I1, Represents the pixel value of the i-th row and j-th column in the source image I2;

[0045] The gradient loss L grad is defined as:

[0046]

[0047] Where, Represents the gradient operation.

[0048] The multimodal medical image fusion method based on feature aggregation and decomposition provided by the present invention has the following beneficial effects compared with the existing technology:

[0049] The present invention proposes a decomposition feedback network, which is composed of a two-stage network with a feature aggregation network for performing image fusion operations. Traditional image fusion methods mostly adopt the process of feature extraction-feature fusion-image reconstruction, focusing on improving image fusion technology, such as improving feature extraction and feature fusion methods. The feature aggregation network of the present invention is an improvement on the feature extraction part. In addition, in order to further improve the fusion performance, the present invention introduces a decomposition feedback network and uses reverse thinking to make the fused image an intermediate process, and improves the intermediate process by optimizing the end result.

[0050] The present invention inputs the fused image into a decomposition feedback network, decomposes it into two decomposed images, and through the constraints of the loss function, makes the generated decomposed image approach the source image, that is, obtains a better end result. A better end result means that the intermediate process of the present invention is also optimized, that is, the fused image of the present invention can contain more scene information from the source image, achieving better fusion performance. In addition, the present invention feeds the decomposed image back to the input. In the early stage of training, the rough decomposed image can be used as noise to optimize the feature aggregation network, making it more robust. When the generated decomposed image is optimized to a certain extent, it can serve as further information supplement, allowing the feature aggregation network to extract more features. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 Flowchart of the multimodal medical image fusion method based on feature aggregation and decomposition of the present invention;

[0052] Figure 2 This is a data transfer diagram of the multimodal medical image fusion method based on feature aggregation and decomposition of the present invention;

[0053] Figure 3 This is a structural diagram of the feature aggregation network of the present invention;

[0054] Figure 4 This is a structural diagram of the feedback network decomposed according to the present invention. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0057] The present invention overcomes the shortcomings of existing methods and proposes a multimodal medical image fusion method based on feature aggregation and decomposition. The present invention includes a feature aggregation network and a decomposition feedback network. In the method of the present invention, the fused image is not regarded as the end point. The present invention sets up two networks to make the fused image an intermediate result, urging the decomposition feedback network to produce a better end point result to achieve a better intermediate result, and the end point result of the decomposition feedback network is fed back to the input of the model to achieve further compensation of information. Therefore, the present invention can solve the problem of insufficiently detailed image feature extraction and lack of information compensation, and innovatively sets the fused image as an intermediate result, and achieves better fusion performance by optimizing the end point result.

[0058] The fusion method of this embodiment includes two networks: a feature aggregation network and a decomposition feedback network. The feature aggregation network follows the process of feature extraction-feature fusion-image reconstruction, and adds a local fine feature extraction module to improve the feature extraction process, thereby achieving better fusion performance. The emergence of the decomposition feedback network means that the fused image is no longer the end point of the model. The fused image becomes the input of the decomposition feedback network, and the improvement of the intermediate result (fused image) is achieved by making the decomposition feedback network generate a better result (decomposed image).

[0059] like Figure 1-2 As shown, a multimodal medical image fusion method based on feature aggregation and decomposition in this embodiment includes the following steps:

[0060] Step S1: Construct a feature aggregation network and a decomposition feedback network.

[0061] like Figure 3 As shown, the feature aggregation network of this embodiment includes an encoder, a global feature extraction module, a local feature extraction module and a decoder.

[0062] Among them, the encoder consists of an adaptive convolution layer (AC, modulatable convolution kernel), a batch normalization layer (BN) and a ReLU activation function.

[0063] The global feature extraction module (Global Feature Block) includes three paths, and the multi-scale features output by the three paths are fused at the pixel level to obtain global features. The first path includes an adaptive convolution layer, a Swin Transformer block (adaptive Transformer block, SWTR), and an adaptive convolution layer connected in sequence. The second path includes an adaptive convolution layer, an adaptive convolution layer, a Swin Transformer block, and an adaptive convolution layer connected in sequence. The third path includes an adaptive convolution layer, an adaptive convolution layer, an adaptive convolution layer, a Swin Transformer block, and an adaptive convolution layer connected in sequence. Each adaptive convolution layer is followed by a batch normalization layer (BN) and a ReLU activation function.

[0064] Among them, the local feature extraction module (Local Feature Block) includes three 3×3 convolutional layers and one 1×1 convolutional layer connected in sequence; the input of the first 3×3 convolutional layer is the shallow features and the source image (the two are input after channel concatenation), the input of the second 3×3 convolutional layer is the shallow features, the source image, and the output of the first 3×3 convolutional layer (the three are input after channel concatenation), the input of the third 3×3 convolutional layer is the shallow features, the source image, the output of the first 3×3 convolutional layer, and the output of the second 3×3 convolutional layer (the four are input after channel concatenation), and the input of the 1×1 convolutional layer is the shallow features, the output of the first 3×3 convolutional layer, the output of the second 3×3 convolutional layer, and the output of the third 3×3 convolutional layer (the four are input after channel concatenation). And each convolutional layer is connected to a batch normalization layer (BN) and a ReLU activation function.

[0065] Among them, the decoder consists of an adaptive convolution layer and a Tanh activation function.

[0066] Among them, such as Figure 4 As shown in the figure, the decomposition feedback network consists of a 1×1 convolutional layer and two branches connected to the 1×1 convolutional layer. The two branches do not share parameters. The first branch consists of three sequentially connected 3×3 convolutional layers, and the second branch consists of three sequentially connected 3×3 convolutional layers. Except for the last 3×3 convolutional layer in each branch, which is connected to a Tanh activation function, all other convolutional layers are connected to a ReLU activation function.

[0067] The network structures with the same name in this application are the same, but the network parameters are not necessarily the same. For example, in the adaptive convolution layer, the input is sent to the context gating layer, the convolution kernel is converted into a modulated convolution kernel, and then a convolution operation is performed on the input. For example, the Swin Transformer block also has a conventional unified structure and is not described in detail in this embodiment.

[0068] Step S2: Take the source image and the initial decomposition image and stitch them together to form a training set.

[0069] The training set uses PET-MRI image pairs with an image size of 256×256. PET is an RGB color image and MRI is a grayscale image. Because the implementation is a fusion of single-channel images and single-channel images, the RGB color image is first converted to YUV color space before the image is input to the network. The Y channel of the converted image is taken. and MRI images I MRI Pairing, then performing overlapping cropping operations on the paired images with a stride of 20, cropping the images into 120×120 patches to achieve the purpose of augmenting the dataset, and finally normalizing the images to [0,1] for ease of training.

[0070] The processed source image needs to be spliced ​​with the decomposed image. The decomposed image in this embodiment comes from the decomposition feedback network. When it is first executed, the decomposed image takes the initial decomposition image as a copy of the source image.

[0071] Since the network only accepts one input, the source image and I MRI Perform channel cascade and receive the decomposed image I from the decomposition feedback network de1 and I de2 To compensate for the information, the source image after cascading Then with I de1 and I de2 Perform channel cascade to form four-channel input feature samples The input feature sample for the first execution is

[0072] Step S3: input the spliced ​​image into the feature aggregation network.

[0073] The encoder consists of an adaptive convolution layer, a batch normalization layer, and a ReLU activation function to extract the input The shallow features F s .Right now:

[0074]

[0075] where E(·) represents the encoder.

[0076] In order to further extract the deep semantic information of the input, the shallow feature F s They are respectively input into the local feature extraction module and the global feature extraction module to extract deep semantic information from different angles.

[0077] In the local feature extraction module, the shallow feature F s First, three 3×3 convolutions are performed to extract deep semantic features, and the convolution layer structure is densely connected, which can achieve the result of feature reuse and reduce the information loss caused by convolution to a certain extent. Then, this application innovatively combines the source image after image splicing Introducing it before the three convolutional layers has two advantages: First, it can reduce the loss of image information during the feature extraction stage of the encoder by converting the source image Directly introducing it into the local feature extraction module of the deep semantic feature extraction stage can avoid information loss caused by convolution and achieve the purpose of information compensation; secondly, allowing the convolution layer to directly contact the source image is conducive to its learning, making it easier to learn the fine features of the image. Finally, the features extracted by each convolution layer are input into a 1×1 convolution layer to achieve feature aggregation while reducing the number of channels, and finally obtain the local feature F L .Right now:

[0078] F L =LFB(F s ) (2)

[0079] Where LFB(·) represents the local feature extraction module. The local feature extraction module proposed in the present invention, combined with the Transformer-based feature extraction module, realizes a more comprehensive extraction of image features. The receptive field of traditional CNN-based methods is limited, and they can only focus on local areas of the image at the same time. The advantage of this is that they can focus on more detailed features in the local areas of the image, that is, they have excellent local feature extraction capabilities. However, precisely because of the limited receptive field, traditional CNN-based methods cannot establish connections with other areas of the image, that is, they cannot model long-distance dependencies, resulting in unrealistic fused images. On the contrary, the Transformer-based method has a strong global context attention capability and can model long-distance dependencies well, but it pays insufficient attention to local details.

[0080] Therefore, the present invention proposes a local feature extraction module, which, combined with a feature extraction module implemented using a Transformer, forms a parallel feature extraction process. This allows for global image modeling and extraction of semantic information, while also focusing on local regions of the image to supplement detailed features. Furthermore, the present invention introduces the source image into each convolutional layer within the local feature extraction module. This not only compensates for information lost by the convolution operation, but also allows the convolutional layer to more easily learn the image's features, thereby extracting more detailed features.

[0081] There are three paths in the global feature extraction module. The three paths have progressive hierarchical structures, aiming to extract features of different scales of the input. The first path consists of two adaptive convolutional layers (which can modulate the convolution kernel to extract global features) and an adaptive Transformer block. The second path consists of three adaptive convolutional layers and an adaptive Transformer block. The third path consists of four adaptive convolutional layers and an adaptive Transformer block. At the end of each of the three paths, an adaptive convolutional layer is assigned to aggregate features, and the remaining structures are used to extract shallow features F. s Deep features of different scales. Finally, the multi-scale features of the three paths are summed element by element to obtain the global feature F G .Right now:

[0082] F G =GFB(F s ) (3)

[0083] Where GFB(·) represents the global feature extraction module.

[0084] The global features and local features are summed element by element and the fused image is reconstructed through the decoder. The decoder consists of an adaptive convolution and a Tanh activation function. The global feature F is obtained. G and local features F L After that, this embodiment adds them element by element and inputs them into the decoder to reconstruct the fused image The formula is:

[0085]

[0086] Where D(·) represents the decoder. Since this embodiment uses color space conversion in the source image, it is necessary to convert the fused image Perform inverse color space conversion and convert Convert back to RGB space and finally get the fused image I f .

[0087] Step S4: input the fused image into the decomposition feedback network to obtain two decomposed images.

[0088] In the decomposition feedback network, the fusion image I f First, shallow features are extracted from the fused image through a shared 1×1 convolutional layer. Then, in order to generate decomposed images of different modalities, the extracted features are divided into two branches for processing. The two branches have the same structure, both containing three 3×3 convolutional layers, but the parameters are not shared between the two branches. Finally, two decomposed images of different modalities are generated. de1 and I de2 . Decompose image I de1 and I de2 The decomposed image is then fed back to the model input, i.e., step S2, to concatenate the decomposed image with the source image to achieve further information compensation. The introduction of the local feature extraction module and the decomposition feedback network in this embodiment enables better preservation of image features and achieves higher fusion performance.

[0089] Step S5: Calculate the total loss based on the source image, fused image, and decomposed image, and update the feature aggregation network and decomposition feedback network according to the total loss. If the iterative training end conditions are met, output the feature aggregation network and decomposition feedback network for multimodal medical image fusion; otherwise, splice the source image and the decomposed image to form a new training set to continue iterative training.

[0090] Specifically, the feature aggregation loss is used to evaluate the fused image obtained in step S3 and the source image in step S2. The decomposition feedback loss is used to evaluate the decomposed image in step S4 and the source image in step S2. Finally, the two losses are combined to train the network model. The total loss function is:

[0091] L total =L FA +αL DF (5)

[0092] Among them, L total is the total loss, L FA is the feature aggregation loss, L DF For decomposition feedback loss, α is a parameter that weighs feature aggregation loss and decomposition feedback loss. In this embodiment, it is set to α=1.

[0093] Feature aggregation loss L FA By the structural similarity loss L SSIM and regional mutual information loss L RMI composition:

[0094] L FA =L SSIM +L RMI (6)

[0095] Feature aggregation loss is used to guide the feature aggregation network to generate a fusion image with rich information. The structural similarity loss LSSIM It is a structural constraint, which is achieved through the structural similarity index SSIM. It can guide the fused image and the source image to maintain the same structural distribution, ensuring that the fused image contains sufficient structural details. Its mathematical expression is:

[0096] L SSIM =β1·(1-SSIM(I f ,I1))+β2·(1-SSIM(I f ,I2)) (7)

[0097] Where β1 and β2 are weighting parameters used to determine the importance of the similarity between the fused image and the source image. In this embodiment, β1 = β2 = 1 is set because the two source images are equally important. f Represents the fused image. I1 and I2 represent the source images, specifically I1 is the PET image I2 is MRI image I MRI SSIM(.) represents the operation of calculating the structural similarity index. The structural similarity index is a common evaluation indicator in image fusion. Its value range is [0,1], which reflects the similarity between the fused image and the source image. Its specific definition is:

[0098]

[0099] Among them I s Indicates the source image, specifically I1 or I2, and are the pixel averages of the source image and the fused image, and are the pixel variances of the source image and the fused image, is the pixel covariance of the source image and the fused image, C1 and C2 are constants used to avoid the denominator from being zero.

[0100] Regional mutual information loss L RMI It is a region-level constraint that limits the information transfer of the source image from region to region, avoiding the introduction of some undesirable artifacts, thereby further ensuring the preservation of information in the fused image. Its mathematical expression is:

[0101] L RMI =γ1·RMI(I f ,I1)+γ2·RMI(I f ,I2) (9)

[0102] Similarly, γ1 and γ2 are trade-off parameters. In this embodiment, γ1 = 1 and γ2 = 2.5 are set. RMI(·) is an operation to calculate the regional mutual information between the fused image and the source image. Regional mutual information is a variant of mutual information. It consists of cross entropy and mutual information lower bound, which can be defined as:

[0103]

[0104] Wherein λ is a weight parameter, λ∈[0,1], and in this embodiment, λ=0.5 is set. represents the calculation of the standard cross entropy loss between the fused image and the source image, B represents the batch size, It means finding the lower bound of the mutual information between the fused image and the source image corresponding to the batch index b in the mini-batch.

[0105] Decomposition feedback loss function L DF Including the strength loss function L int And the gradient loss function L grad The mathematical expression is:

[0106] L DF =L int +L grad (11)

[0107] Strength loss term L int The decomposed image can be guided to keep the distribution of pixel intensity as consistent as possible with the source image, making the decomposed image more natural. It is defined as:

[0108]

[0109] Where H and W represent the image height and width of the source image or decomposed image (they have the same height and width), i and j represent the i-th row and j-th column, Represents the decomposed image I de1 The pixel value of the i-th row and j-th column in , Represents the decomposed image I de2 The pixel value of the i-th row and j-th column in , represents the pixel value of the i-th row and j-th column in the source image I1, Represents the pixel value of the i-th row and j-th column in the source image I2.

[0110] Gradient loss term L grad First, the gradient of the decomposed image and the source image is calculated, and then the loss is used to constrain the gradient distribution of the decomposed image and the source image to be consistent, that is, to keep the texture detail distribution of the decomposed image and the source image consistent. It is defined as:

[0111]

[0112] in represents a gradient operation. In this embodiment, the Laplacian operator is used to calculate the gradient map of the image.

[0113] Get the feature aggregation loss L FA and decomposed feedback loss L DFAfter obtaining the values ​​of , the parameters of the feature aggregation network and the decomposition feedback network are updated according to their performance to realize the training of the feature aggregation network and the decomposition feedback network.

[0114] In the testing or inference application phases of this embodiment, the source images used are all PET-MRI image pairs, specifically, registered image pairs. Decomposed images are also generated during these phases, but they are no longer used as input to the feature aggregation network. Instead, the output image from the feature aggregation network is directly inversely converted to RGB space to produce the fused image.

[0115] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0116] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A multimodal medical image fusion method based on feature aggregation and decomposition, characterized in that: The multimodal medical image fusion method based on feature aggregation and decomposition includes: Constructing a feature aggregation network and a decomposition feedback network, wherein the feature aggregation network includes an encoder, a global feature extraction module, a local feature extraction module and a decoder; Take the source image and the initial decomposition image and stitch them together to form a training set; The spliced ​​image is input into the feature aggregation network, which extracts shallow features through an encoder, and inputs the shallow features into a global feature extraction module and a local feature extraction module respectively to obtain global features and local features. The global features and local features are superimposed and then input into a decoder, which reconstructs and outputs a fused image; Inputting the fused image into the decomposition feedback network to obtain two decomposed images; The total loss is calculated based on the source image, the fused image, and the decomposed image, and the feature aggregation network and the decomposition feedback network are updated according to the total loss. If the iterative training end condition is met, the feature aggregation network and the decomposition feedback network are output for multimodal medical image fusion; otherwise, the source image and the decomposed image are spliced ​​together to form a new training set and iterative training is continued; The global feature extraction module includes three paths, and the multi-scale features output by the three paths are fused at the pixel level to obtain the global feature; The first path includes an adaptive convolution layer, a Swin Transformer block, and an adaptive convolution layer connected in sequence. The second path includes an adaptive convolution layer, an adaptive convolution layer, a Swin Transformer block, and an adaptive convolution layer connected in sequence. The third path includes an adaptive convolution layer, an adaptive convolution layer, an adaptive convolution layer, a Swin Transformer block, and an adaptive convolution layer connected in sequence. The local feature extraction module includes three 3×3 convolutional layers and one 1×1 convolutional layer connected in sequence; The input of the first 3×3 convolution layer is the shallow features and the source image, the input of the second 3×3 convolution layer is the shallow features, the source image and the output of the first 3×3 convolution layer, the input of the third 3×3 convolution layer is the shallow features, the source image, the output of the first 3×3 convolution layer and the output of the second 3×3 convolution layer, and the input of the 1×1 convolution layer is the shallow features, the output of the first 3×3 convolution layer, the output of the second 3×3 convolution layer and the output of the third 3×3 convolution layer.

2. The multimodal medical image fusion method based on feature aggregation and decomposition according to claim 1, characterized in that: The encoder consists of an adaptive convolution layer, a batch normalization layer, and a ReLU activation function.

3. The multimodal medical image fusion method based on feature aggregation and decomposition according to claim 1, wherein: The source image and the initial decomposition image are spliced ​​together to form a training set, including: Acquire PET-MRI image pairs as source images, where PET is an RGB color image and MRI is a grayscale image; Convert the RGB color image to YUV color space and take the Y channel of the converted image Pairing images with grayscale images; The initial decomposition image is taken as a copy of the source image, that is, the initial decomposition image is the paired Image and grayscale image, channel cascade the source image and the initial decomposition image to obtain the four-channel input feature sample I MRI is a grayscale image.

4. The multimodal medical image fusion method based on feature aggregation and decomposition according to claim 1, wherein: The decomposition feedback network includes a 1×1 convolutional layer and two branches connected to the 1×1 convolutional layer, the two branches do not share parameters, wherein the first branch includes three 3×3 convolutional layers connected in sequence, and the second branch includes three 3×3 convolutional layers connected in sequence.

5. The multimodal medical image fusion method based on feature aggregation and decomposition according to claim 1, wherein: The total loss is calculated based on the source image, the fused image and the decomposed image, including: A feature aggregation loss is calculated according to the source image and the fused image, a decomposition feedback loss is calculated according to the source image and the decomposition image, and the feature aggregation loss and the decomposition feedback loss are summed to obtain a total loss.

6. The multimodal medical image fusion method based on feature aggregation and decomposition according to claim 5, characterized in that: The calculating feature aggregation loss according to the source image and the fused image includes: The feature aggregation loss L FA By the structural similarity loss L SSIM and regional mutual information loss L RMI composition: L FA =L SSIM +L RMI The structural similarity loss L SSIM It is a structural constraint, and its mathematical expression is: L SSIM =β1·(1-SSIM(I f ,I1))+β2·(1-SSIM(I f ,I2)) Where β1 and β2 are trade-off parameters, I f represents the fused image, I1 and I2 represent the source images, and SSIM(.) represents the structural similarity index operation, which is specifically defined as: Where, I s Indicates the source image, specifically I1 or I2, and are the pixel averages of the source image and the fused image, and are the pixel variances of the source image and the fused image, is the pixel covariance of the source image and the fused image, C1 and C2 are constants; The region mutual information loss L RMI is a regional constraint, and its mathematical expression is: L RMI =γ1·RMI(I f ,I1)+γ2·RMI(I f ,I2) Where γ1 and γ2 are trade-off parameters, and RMI(·) is the regional mutual information operation for calculating the fused image and the source image, which is specifically defined as: Where λ is the weight parameter, λ∈[0,1], represents the calculation of the standard cross entropy loss between the fused image and the source image, B represents the batch size, It means finding the lower bound of the mutual information between the fused image and the source image corresponding to the batch index b in the mini-batch.

7. The multimodal medical image fusion method based on feature aggregation and decomposition according to claim 5, characterized in that: The calculating the decomposition feedback loss according to the source image and the decomposition image includes: The decomposition feedback loss L DF Including strength loss L int and gradient loss L grad , the mathematical expression is: L DF =L int +L grad The strength loss L int is defined as: Where H and W represent the height and width of the source image or decomposed image, i and j represent the i-th row and j-th column, Represents the decomposed image I de1 The pixel value of the i-th row and j-th column in , Represents the decomposed image I de2 The pixel value of the i-th row and j-th column in , represents the pixel value of the i-th row and j-th column in the source image I1, Represents the pixel value of the i-th row and j-th column in the source image I2; The gradient loss L grad is defined as: Where, Represents the gradient operation.

Images