A multimodal medical image fusion method based on cross neural network and ECA-S

Abstract

The present invention discloses a multimodal medical image fusion method based on a cross neural network and ECA‑S, belonging to the field of medical image analysis. The method uses two single-modal medical images as input in a fusion network model, cross-extracts image feature information through a structural function module and a functional structure module, can realize the interaction between structural information and functional information, thereby well extracting texture detail information of medical images, and adopts the ECA‑S attention mechanism to fuse the extracted feature information by continuously adjusting the weights of structural information and functional information. In the decomposition network model, the decomposition network composed of a residual network forces the fused image to contain richer information. The present invention realizes multimodal medical image fusion by combining a cross network, an improved attention mechanism and a decomposition network. The texture details and contrast processing of the fused image are more detailed, the algorithm complexity is lower, and the optimization of existing methods and the upgrading of existing technologies are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to a multimodal medical image fusion method based on a cross neural network and ECA-S. Background Art

[0002] Multimodal medical image fusion plays an important role in clinical analysis and treatment planning, and can assist doctors in quickly diagnosing and treating diseases. Since single-modal imaging is single, it is not conducive to doctors observing the condition. In order to obtain richer information from single-modal medical images, multimodal medical image fusion is widely used in clinical analysis to achieve accurate description of the lesion site.

[0003] In the process of traditional medical image fusion, texture details are easily lost and the contrast is low. "Artifacts" are easily generated at the edge details of the image, resulting in missing image contours and blurred textures after fusion. In order to solve these problems, a deep learning image fusion algorithm is proposed to improve the fusion quality.

[0004] The core of image fusion technology lies in optimizing algorithms and improving the quality of image fusion. At present, image fusion is divided into three levels: feature level, decision level and pixel level. Although feature level fusion reduces the complexity of the algorithm when processing images, the fused image is prone to lose some detail information. Decision level fusion has good real-time and adaptability, but the experimental data increases errors and loses the most image information. Pixel level fusion is the process of information fusion of pixel points in the image, which is also the mainstream of current research methods.

[0005] The present invention provides a multimodal medical image fusion method based on a cross network and ECA-S, which can significantly enhance texture details, edge contour clarity and fused image contrast, and the fused medical image is more in line with the visual characteristics of the human eye. Summary of the invention

[0006] The purpose of the present invention is to provide a multimodal medical image fusion method based on cross neural network and ECA-S in order to solve the above problems.

[0007] To achieve the above object, the technical solution adopted by the present invention is: a multimodal medical image fusion method based on a cross neural network and ECA-S, comprising the following steps:

[0008] S1: Use GRFEB and RGFEB in the fusion network model to extract shallow feature information of the image.

[0009] Among them, GRFEB consists of a low-level feature extraction block with a convolution kernel of 5×5 and Mish as the activation function layer, and three structure-guided function blocks (denoted as GR blocks), and RGFEB also consists of a low-level feature extraction block and three function-guided structure blocks (denoted as RG blocks);

[0010] Multiple cross connections are designed on the GR block and the RG block to achieve more complete information interaction between GRFEB and RGFEB.

[0011] S2: The outputs of the third GR block and the third RG block in the cross network module are cascaded in the channel dimension and fed into a convolution with a kernel size of 1×1 and LReLU as the activation function for feature extraction.

[0012] S3: The extracted feature information is used to focus on salient information from both spatial and channel aspects through the ECA-S attention mechanism;

[0013] S4: The cross feature and the convolution feature with a kernel of 1×1 are fused through the ECA-S attention mechanism to obtain a fused image;

[0014] S5: The fusion result is passed through the decomposition network model again. The decomposition network composed of residual networks forces the fused image to contain richer information.

[0015] Furthermore, the input of the GR block includes the output of the previous blocks in GRFEB and RGFEB, i.e. and Where n∈[1, 3] represents the block index in the feature extraction branch. In order to fully extract features from MRI images, a two-stage residual structure is used;

[0016] Furthermore, the ECA-S attention mechanism is composed of an effective channel attention mechanism (ECA) and a spatial attention mechanism (SAM) connected in parallel, so that both attention modules can directly learn the original input feature information, without having to pay attention to the order of spatial attention and channel attention, forming a new ECA-S attention mechanism model.

[0017] Furthermore, a loss function is designed in the network model, where the loss function is composed of the fusion loss L sf and decomposition loss L dc Composition, formula:

[0018] L=L sf +L dc (4)

[0019] Furthermore, the fusion loss is also composed of the structural loss term L stru and the functional loss term L func The loss function can be expressed as formula (5):

[0020] L sf =βL stru +L func (5)

[0021] β is used to balance the structural term and the functional term.

[0022] Furthermore, the structural loss function L stru The rich texture features of the fused image are increased. The structural loss is composed of an adaptive decision block. The adaptive decision block filters out the influence of noise on the image through a Gaussian low-pass filter, minimizes the influence of noise on the decision block, and uses the Scharr operator to obtain the structure graph. The structure graph contains a lot of texture detail information and clear contour information. Then, a decision graph is generated according to the size of the structure.

[0023]

[0024]

[0025] S 2i,j =1-S 1i,j (8)

[0026] Where i and j represent the weight map of the pixel in the i-th row and j-th column, H and W represent the height and width of the image, I1 and I2 are the source images, and I fusion is the fused image, |·| represents the absolute value function, is the Laplace operator, L(·) represents the Gaussian low-pass filter function, min(·) represents the minimum function, and sign(·) is the sign function.

[0027] Furthermore, the functional loss L func The contrast of the fused image is affected. The functional loss is achieved by improving the Frobenius norm. The improved Frobenius norm is different from the Euclidean distance. It is the square root of the average sum of the squares of the elements at each position in the matrix. It measures the distance between the pixels of the fused image and the pixels of the source image, effectively adjusting the weights between the fused image and the two source images, forcing the network to select more effective information.

[0028] The formula for functional loss can be expressed as formula (9):

[0029]

[0030] Where i and j represent the weight map of the pixel in the i-th row and j-th column, H and W represent the height and width of the image, I1 and I2 are the source images, and Ifused is the fused image, and α is the balancing weight.

[0031] Further, decomposing the loss term L dc The decomposition result of the fused image is required to be as similar as possible to the source image, which can be expressed as:

[0032]

[0033] Among them, I 1_de and I 2_de is the decomposition result of the fused image, I1 and I2 are the source images.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] Compared with the currently popular multimodal medical image fusion algorithm, the performance of this algorithm has been greatly improved. In the fusion stage, not only the interaction of multimodal medical image structural information and functional information is completed through the cross network, but also an improved attention mechanism is used to focus on important information through high weights and ignore irrelevant information with low weights, while continuously adjusting the weights to select important information.

[0036] The improved attention mechanism is used to fuse the shallow and deep feature information extracted from the original image to obtain a fused image. The salient information of the image is retained in both channels and space, useless information is suppressed, and all salient information of the useful fusion layer is ensured to be fully utilized.

[0037] In the decomposition stage, in order to force the fused image to contain richer information as much as possible, the decomposition process from the fused image to the source image is considered. Since the quality of the decomposed image directly depends on the fusion result, the fusion result can contain more detailed information.

[0038] The combined application of cross networks, improved attention mechanisms and decomposition networks has enabled multimodal medical image fusion, and further demonstrated the unique advantages of cross networks, improved attention mechanisms and decomposition network models in image processing. Texture details and contrast processing are more detailed, the algorithm complexity is lower, and it also has great advantages in application in other fields, achieving the optimization of existing methods and the upgrading of existing technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of the multimodal medical image fusion method based on cross network and ECA-S of the present invention;

[0040] Figure 2 The GR Block network structure of the present invention;

[0041] Figure 3 The RG Block network structure of the present invention;

[0042] Figure 4 The ECA-S attention mechanism network structure of the present invention;

[0043] Figure 5 Decomposition network process of the present invention;

[0044] Figure 6 is the adaptive decision block of the present invention;

[0045] Figure 7 This is a diagram of the fusion results of different MRI-PET methods for mild Alzheimer's disease according to an embodiment of the present invention;

[0046] Figure 8 This is a diagram showing the fusion results of different MRI-SPECT methods for metastatic bronchial carcinoma according to an embodiment of the present invention;

[0047] Fig. 9 This is a diagram of the fusion results of different methods of MRI-CT for meningioma in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0049] The technical solution of the present invention is based on the multimodal medical image fusion method process of the cross network and ECA-S as follows: Figure 1 As shown in the figure. In the fusion network model, two single-modality medical images are used as input, image feature information is extracted through the structure function module and the function structure module, and the ECA-S attention mechanism fuses the final image. In the decomposition network model, the decomposition network composed of residual networks forces the fused image to contain richer information.

[0050] The specific implementation steps are as follows;

[0051] (1) The aligned images are passed through GRFEB and RGFEB to extract shallow feature information of the image. GRFEB consists of a low-level feature extraction block with a convolution kernel of 5×5 and Mish as the activation function layer, and three structure-guided functional blocks (denoted as GR blocks). Similarly, RGFEB consists of a low-level feature extraction block and three function-guided structural blocks (denoted as RG blocks). Multiple cross connections are designed on the GR block and the RG block to achieve more complete information interaction between GRFEB and RGFEB. Specifically, the input of each GR block includes not only the output of the previous block in GRFEB, but also the output of the corresponding previous block in RGFEB. The same is true for the RG block. Table 1 lists the specific parameters of the proposed fusion model, where the convolutional layers are numbered from left to right and from bottom to top in each block or module.

[0052] Table 1 Specific parameters of the fusion-decomposition model

[0053]

[0054]

[0055] The network architecture of the GR module is as follows: Figure 2 The input of the GR block includes the output of the previous blocks in GRFEB and RGFEB, i.e. and Where n∈[1, 3] represents the block index in the feature extraction branch. In order to fully extract features from MRI images, a two-stage residual structure is used. Specifically, It is fed into the first-level residual structure, which consists of two 3×3 convolutional layers and an LReLU layer. Added to the output to form a residual connection. Processed by the same residual structure to join the features extracted from PET or SPECT images. The outputs of these two residual structures are cascaded in the channel, followed by a 1×1 convolutional layer for feature fusion and channel dimension reduction. The second stage residual structure is used to enhance the structural information in the MRI image. Original input Add the output of the 1×1 convolutional layer to get the final output The RG module also uses a two-level residual structure, such as Figure 3 The difference is that the second level residual connection is performed on the output of the previous block in RGFEB (i.e., ) to enhance the functional information in PET or SPECT images.

[0056] (2) The outputs of the third GR block and the third RG block in the cross network module are cascaded in the channel dimension and fed into a convolution with a kernel size of 1×1 and LReLU as the activation function for feature extraction.

[0057] (3) The extracted feature information is used to focus on significant information from both spatial and channel aspects through the ECA-S attention mechanism. The ECA-S attention mechanism is connected in parallel by the Efficient channel attention mechanism (ECA) and the Spatial Attention Mechanisms (SAM), so that both attention modules can directly learn the original input feature information, without paying attention to the order of spatial attention and channel attention, forming a new ECA-S attention mechanism model. Figure 4As shown in Figure 1. The parallel connection branch of ECA-S is composed of a spatial attention mechanism. First, the input feature map is max-pooled (Maxpool) and average-pooled (avgpool) and stacked in the channel dimension. Second, a 1×1 convolution kernel is used to extract the features of the stacked feature maps. Finally, the extracted feature information is normalized by the spatial weight of the sigmoid activation function, and then the input feature map and the weight are multiplied to obtain M. S The other branch is composed of the ECA attention mechanism. First, global average pooling (GlobalAvgpooling) is performed on the input feature layer. Secondly, the pooled feature strips are convolved and the weight of each feature point is generated through the sigmoid activation function. Finally, the weight is combined with the original feature layer to obtain M ECA . The calculated M S With M ECA Add together to get M ECA_S The entire calculation process can be expressed as formula (1)-(3):

[0058] M S (F) = σ(f 1×1 ([AvgPool(I);MaxPool(I)])) (1)

[0059] M ECA (F) = σ(f k×k (GlobalAvgPooling(I))) (2)

[0060]

[0061] Where I represents the input feature map, Avgpool represents average pooling, Maxpool represents maximum pooling, GlobalAvgpooling represents global average pooling, and f k×k represents the convolution operation with a kernel size of k×k, σ represents the sigmoid activation function, Represents element-by-element addition.

[0062] (4) The cross features are fused with the convolution features with a kernel of 1×1 through the ECA-S attention mechanism to obtain a fused image.

[0063] (5) The fusion result is further decomposed to make the fusion result contain more texture details. Figure 5As shown in the figure, each block first passes through a 1×1 convolution layer, then a 3×3 convolution layer, forming a multi-scale structure to fully extract spatial information, and finally outputs feature information through the LReLU activation function. In order to fully decompose the feature information of the fused image, the input of the decomposition network is divided into two parts, one part of the input is the image fused through the ECA-S attention mechanism, and the other part is the output of the third layer of GR Block and RG Block. The purpose is to consider that the ECA-S attention mechanism extracts the significant information of the image, while some unimportant information is easily ignored. Therefore, the third layer of GR Block and RG Block is directly output as part of the decomposition network to make the information extraction more sufficient during the decomposition process.

[0064] (6) Design of loss function

[0065] The network framework of this paper is divided into two parts. The fusion network extracts features to generate a single fused image. The decomposition network decomposes and reconstructs the structural and functional information, and the decomposition network is dedicated to decomposing the result of approximating the source image from the fusion result. Correspondingly, the loss function also consists of two parts: the fusion loss L sf and decomposition loss L dc .

[0066] L=L sf +L dc (4)

[0067] (7) Fusion loss

[0068] The information contained in the image can be divided into structural and functional information, where the functional information represents the overall brightness and the structural information represents the texture structure. Based on this, the medical image fusion problem is transformed into the problem of extracting and reconstructing the structural and functional information in the image, and the fusion loss is also composed of the structural loss term L stru and the functional loss term L func The loss function can be expressed as formula (5):

[0069] L sf =βL stru +L func (5)

[0070] Among them, β is used to balance the structural term and the functional term.

[0071] The present invention designs an adaptive structural loss function L struTo increase the rich texture features of the fused image. First, the adaptive weight block first uses a Gaussian low-pass filter to filter out the impact of noise on the image. Minimize the impact of noise on the weight block as much as possible, then use the Scharr operator to obtain the gradient map. The structure map contains a lot of texture detail information and clear contour information. Finally, a decision map is generated according to the size of the structure. Its principle diagram is as follows Figure 6 shown.

[0072]

[0073]

[0074] S 2i,j =1-S 1i,j (8)

[0075] Where i and j represent the weight map of the pixel in the i-th row and j-th column, H and W represent the height and width of the image, I1 and I2 are the source images, and I fusion is the fused image, |·| represents the absolute value function, is the Laplace operator, L(·) represents the Gaussian low-pass filter function, min(·) represents the minimum function, and sign(·) is the sign function.

[0076] Function loss L func The functional loss is achieved by improving the Frobenius norm. The improved Frobenius norm is different from the Euclidean distance. It is the square root of the average sum of the squares of the elements at each position in the matrix. It measures the distance between the pixels of the fused image and the pixels of the source image, effectively adjusts the weights between the fused image and the two source images, and forces the network to select more effective information. The functional loss formula can be expressed as formula (9):

[0077]

[0078] Where i and j represent the weight map of the pixel in the i-th row and j-th column, H and W represent the height and width of the image, I1 and I2 are the source images, and I fused is the fused image, and α is the balancing weight.

[0079] (8) Decomposition of loss term L dc The decomposition result of the fused image is required to be as similar as possible to the source image, which can be expressed as:

[0080]

[0081] Among them, I 1_de and I 2_deis the decomposition result of the fused image, I1 and I2 are the source images. Since the difference between the decomposed image and the input image determines the effect of the fusion result, the decomposition loss function will try to make the fusion result obtain more image information, so as to achieve the best fusion quality.

[0082] This method is compared with seven image fusion methods that have been popular in recent years. The compared algorithms are: MLEPF-based fusion method, NSST-based fusion method, LEGFF-based fusion method, MATR-based fusion method, CNP-based fusion method, SDNet-based fusion method, and CFL-based fusion method.

[0083] The multimodal medical image fusion effect evaluation method of the present invention includes subjective visual evaluation and objective index evaluation. In order to fully reflect that the fusion algorithm framework of this paper has good fusion performance, spatial frequency (SF), average gradient (AG), fusion quality (Q AB / F ), correlation coefficient (CC), entropy (EN) and mutual information (MI) and other objective indicators are used to evaluate the results of the fused image. AG and EN are indicators to measure the clarity and amount of information of the image. The larger the value, the richer and more comprehensive the information extracted from the input image. MI calculates how much information from the source image is included in the fused image by measuring the similarity between the two images. The correlation between the two images is represented by CC. The closer the value is to 1, the closer the feature information between the two images is. The measurement of image grayscale is represented by SF. The larger the SF value, the closer the grayscale value of the fusion result is to the grayscale value of the input image. Q AB / F It is an estimation of the degree of information expression in the fusion result image. The larger the value, the greater the amount of information and the better the image quality. The experimental data of disease characteristics of three groups of different source images are shown in Tables 1, 2 and 3.

[0084] Table 2 Objective evaluation indexes of MRI-PET image fusion for “mild Alzheimer’s disease”

[0085]

[0086] Table 3 Objective evaluation indexes of MRI-SPECT image fusion for “metastatic bronchial carcinoma”

[0087]

[0088] Table 4 Objective evaluation indexes of MRI-CT image fusion for “meningioma”

[0089]

[0090] It can be concluded from the results of the seven index values ​​in Tables 2, 3 and 4 that the AG, EN, SF, MI, Q AB / F The objective evaluation indicators of and CC are all improved by 22.87%, 19.64%, 23.02%, 12.70%, 6.79% and 30.35%, indicating that the fusion method performs better in processing texture details and edge contours, and has high contrast and low complexity. This is because the structural information and functional information are interacted through the cross network, the attention mechanism focuses on the significant information, and the decomposition network fusion makes the fused image have richer detail information.

[0091] Figures 7 to 9 The figure shows the fusion comparison of different methods for different diseases, where sub-images (a) and (b) are source images; sub-image (c) is a medical image fusion algorithm based on MLEPF; (d) is a medical image fusion algorithm based on NSST; (e) is a medical image fusion algorithm based on LEGFF; (f) is a medical image fusion algorithm based on MATR; (g) is a medical image fusion algorithm based on CNP; (h) is a medical image fusion algorithm based on SDNet; (i) is a medical image fusion algorithm based on CFL; (j) is the result of the multimodal medical image fusion algorithm of the present invention. In summary, the fused image obtained by the method of the present invention is significantly better than other fusion algorithms in terms of subjective perception and objective evaluation, which shows that the experimental effect of the method of the present invention is better.

[0092] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0093] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A multimodal medical image fusion method based on cross neural network and ECA-S, characterized in that: The specific steps are: S1: Use GRFEB and RGFEB in the fusion network model to extract shallow feature information of the image. Among them, GRFEB includes a low-level feature extraction block with a convolution kernel of 5×5 and Mish as the activation function layer, and three structure-guided function blocks, which are represented as GR blocks. RGFEB also consists of a low-level feature extraction block and three function-guided structure blocks, which are represented as RG blocks. Multiple cross-connections are designed on the GR block and the RG block to achieve more complete information interaction between GRFEB and RGFEB; S2: The outputs of the third GR block and the third RG block in the cross network module are cascaded in the channel dimension and fed into a convolution with a kernel size of 1×1 and LReLU as the activation function for feature extraction; S3: The extracted feature information is used to focus on salient information from both spatial and channel aspects through the ECA-S attention mechanism; S4: The cross feature and the convolution feature with a kernel of 1×1 are fused through the ECA-S attention mechanism to obtain a fused image; S5: The fusion result is passed through the decomposition network model again. The decomposition network composed of residual networks forces the fused image to contain richer information.

2. The multimodal medical image fusion method based on cross neural network and ECA-S according to claim 1, characterized in that: The input of the GR block includes the output of the previous blocks in GRFEB and RGFEB, i.e. and , where n∈[1, 3] represents the block index in the feature extraction branch. In order to fully extract features from MRI images, a two-stage residual structure is used.

3. The multimodal medical image fusion method based on cross neural network and ECA-S according to claim 1, characterized in that: The ECA-S attention mechanism is composed of the Efficient channel attention mechanism (ECA) and the spatial attention mechanism (SAM) connected in parallel, so that both attention modules can directly learn the original input feature information, without having to pay attention to the order of spatial attention and channel attention, forming a new ECA-S attention mechanism model.

4. The multimodal medical image fusion method based on cross neural network and ECA-S according to claim 1, characterized in that: A loss function is designed in the network model, where the loss function is composed of fusion loss and decomposition loss Composition, formula: (4)。 5. The multimodal medical image fusion method based on cross neural network and ECA-S according to claim 4, characterized in that: The fusion loss is also composed of the structural loss term and the functional loss term The loss function can be expressed as formula (5): L sf =βL stru +L func (5) Used to balance structural and functional items.

6. The multimodal medical image fusion method based on cross neural network and ECA-S according to claim 5, characterized in that: Structural loss function Increase the rich texture features of the fused image. The structural loss is composed of an adaptive decision block. The adaptive decision block uses a Gaussian low-pass filter to filter out the impact of noise on the image, minimize the impact of noise on the decision block, and use the Scharr operator to obtain the structure diagram. The structure diagram contains a lot of texture detail information and clear contour information. Then, a decision diagram is generated according to the size of the structure: (6) (7) (8) In the formula, i and j Indicates that the pixel is i Row and j Column weight graph, H and W Represents the height and width of the image, and is the source image, To fuse the images, · represents the absolute value function, (·) is the Laplace operator, L(·) represents the Gaussian low-pass filter function, min(·) represents the minimum function, and sign(·) is the sign function.

7. The multimodal medical image fusion method based on cross neural network and ECA-S according to claim 5, characterized in that: Loss of functionality Affects the contrast of the fused image; the functional loss is achieved by improving the Frobenius norm. The improved Frobenius norm is different from the Euclidean distance. It is the square root of the average sum of the squares of the elements at each position in the matrix. Its function is to measure the distance between the pixels of the fused image and the pixels of the source image, effectively adjust the weights between the fused image and the two source images, and force the network to select more effective information. The formula for functional loss can be expressed as formula (9): (9) In the formula, i and j Indicates that the pixel is i Row and j Column weight graph, H and W Represents the height and width of the image, and is the source image, To fuse the images, To balance the weight.

8. The multimodal medical image fusion method based on cross neural network and ECA-S according to claim 4, characterized in that: Decomposing loss terms The decomposition result of the fused image is required to be as similar as possible to the source image, which can be expressed as: (10) in, and is the decomposition result of the fused image, and is the source image.

Images