A zero-learning medical image fusion method based on Siamese convolutional neural network
Preprocessing and feature extraction of medical images through pre-trained twin convolutional neural networks solves the problems of image decomposition and feature extraction in multimodal medical image fusion, and achieves efficient and low-resource image fusion effect.
Patent Information
- Application Number
- CN202210478942.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-04
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing multimodal medical image fusion method has problems such as poor spatial overlapping feature separation, inaccurate manual extraction of features, noise sensitivity and high hardware resource consumption in the process of image decomposition and feature extraction.
Pre-trained twin convolutional neural network is used to pre-process and feature extraction of medical images, and the kernel norm is calculated through sliding window, upsampling and exponential operations are generated to generate significant feature maps, and image fusion is achieved by combining weighted average fusion algorithm and guide filter.
Automatic multimodal medical image significant feature extraction is realized, reducing the demand for hardware resources, improving fusion accuracy and efficiency, and reducing dependence on specific training data sets and high-performance hardware.
Smart Images

Figure CN114821259B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical image processing, and more specifically to a zero-learning medical image fusion method based on a twin convolutional neural network. Background Art
[0002] In the field of medical image processing technology, due to the diversity of imaging mechanisms and the limitations of imaging equipment, medical images of different modalities focus on different categories of organ / tissue information. For example, computed tomography (CT) can accurately detect densely structured bones and implanted objects; magnetic resonance imaging (MRI) provides high-resolution anatomical information for soft tissues, but is not as sensitive as CT for bone diagnosis; positron emission tomography (PET) images contain rich information about tumor function and metabolism; single photon emission computed tomography (SPECT) images can reflect the blood flow information of tissues and organs, but the resolution of PET and SPECT images is low. A single medical imaging device can only obtain certain characteristics when interpreting and judging organ and tissue information, and cannot make an accurate and reliable diagnosis of the patient's condition. Therefore, in the process of medical image processing, the comprehensive use of multimodal medical image information is an important way to enhance the application value of multimodal medical images.
[0003] Traditional multimodal medical image fusion methods usually include three steps: decomposition, fusion and reconstruction. The design of image decomposition and fusion rules are two main factors affecting the fusion quality. Some commonly used image decomposition methods often cannot effectively separate spatial overlapping features, and gradient reversal is prone to occur at the edges of the image, thereby introducing artifacts or false information into the fused image. The design of fusion rules is often to manually extract image features, generate weight values, and perform weighted or selective fusion. Manually extracted features often cannot accurately express the significant features of the image, and have poor robustness to noise, mismatch and other conditions of the source image, resulting in a decrease in the brightness and contrast of the fused image.
[0004] In recent years, deep learning methods have been applied to the field of multimodal medical image fusion due to their powerful feature extraction and data representation capabilities. The deep learning network itself can be regarded as a feature extractor, and the intermediate mapping representation can be used to reconstruct the salient features of the fused image. Compared with traditional methods, the fusion method based on deep learning can achieve better fusion performance, but the deep learning network is trained based on image data sets, and therefore requires a large amount of specific types of image data, which is not available in practical applications. In addition, the convolutional neural network (CNN) occupies a large amount of memory during the training and fusion process, and is time-consuming and energy-consuming, which is not conducive to the engineering application of multimodal medical image fusion. Therefore, this field urgently needs a multimodal medical image fusion method that can overcome the above defects. Summary of the invention
[0005] The present invention provides a zero-learning medical image fusion method based on a twin convolutional neural network to solve the problems in the prior art that image decomposition cannot effectively separate spatial overlapping features, manually extracted features cannot accurately express the significant features of the source image, resulting in a decrease in the brightness and contrast of the fused image, and is sensitive to noise and mismatch of the source image, and the deep learning network requires a large amount of learning training data and occupies a large amount of hardware resources, which is time-consuming and energy-consuming.
[0006] The technical solution adopted by the present invention comprises the following steps:
[0007] Step A: For the source image φ k Perform preprocessing to generate the input source image f k , and then normalize it to generate a normalized source image It is used as the input of the pre-trained three-layer twin convolutional neural network to obtain the feature maps of the three convolutional layers, expressed as and the initial salient feature map of the fully connected layer, expressed as k∈{1,2} is the image index, d∈{1,2,3} is the convolution layer index, M∈{64,128,256} is the number of feature maps of the convolution layer, 1:M means from the feature Figure 1 To feature map M;
[0008] Step B: In the feature map of each convolutional layer, select a 5×5×M window in a sliding window manner, arrange it into a 25×M matrix, calculate its nuclear norm, and get a pixel value. When the sliding window traverses the entire feature map, the initial significant feature map of each convolutional layer is obtained, which is expressed as
[0009] Step C: Initial salient feature map of convolutional layer 3 and the initial salient feature map of the fully connected layer Perform upsampling operations to generate the salient feature map of convolution layer 3 And the salient feature map of the fully connected layer Its size is the same as the normalized source image The size is consistent, so Get the salient feature map of convolutional layer 1 make Get the salient feature map of convolutional layer 2 In this way, the salient feature maps of each layer are uniformly represented as l∈{1,2,3,4} is the layer index of the salient feature map;
[0010] Step D: salient feature maps of each layer Perform exponential operations and apply the Softmax operator to generate the initial weight graphs for each layer
[0011] Step E: Normalize the source image As the guide image of the guide filter, the initial weight map of each layer Perform guided filtering to obtain the guided filter output image Apply the Softmax operator to generate the weight graph of each layer
[0012] Step F: Using the weight maps of each layer The weighted average fusion algorithm is used to obtain the fused images of each layer. The pixel maximization algorithm is used to obtain the final fused image f F .
[0013] Step A of the present invention comprises:
[0014] Sub-step A1: pre-processing the source image to generate an input source image, and then performing normalization processing to generate a normalized source image;
[0015] The source image is φ k , if φ k is a grayscale image, then let f k =φ k , get the input source image f k , normalize it and get the normalized source image The calculation formula is:
[0016]
[0017] Where k∈{1,2} is the image index, f k and They represent the kth input source image before normalization and the kth normalized source image after normalization, respectively. kmin and f kmax Respectively represent the kth input source image f k The minimum and maximum values of pixels in ;
[0018] If the source image φ k In the figure, φ1 is a grayscale image and φ2 is an RGB color image. φ2 needs to be preprocessed and converted to YUV space. The conversion formula from RGB to YUV image space is:
[0019]
[0020] Where φ 2R ,φ 2G ,φ 2B Respectively represent the red, green, and blue channel images of the source image φ2 in the RGB color space, Represent the converted images respectively Three component images of brightness, color and saturation in YUV color space;
[0021] Let f1 = φ1, At this time, the input source image f k In the figure, f1 and f2 are both grayscale images, which are normalized to obtain the normalized source image At the same time and The components are also normalized to obtain a normalized image and The calculation formula is:
[0022]
[0023] Where k∈{1,2} is the image index, f k and They represent the kth input source image before normalization and the kth normalized source image after normalization, respectively. kmin and f kmax Respectively represent the kth input source image f k The minimum and maximum values of pixels in U min Indicates the minimum pixel value in the U component image, U min Indicates the maximum pixel value in the U component image, V min Indicates the minimum pixel value in the V component image, V max Indicates the minimum pixel value in the V component image;
[0024] Sub-step A2: Construct a twin convolutional neural network model, where the input to output is: convolutional layer 1, convolutional layer 2, pooling layer, convolutional layer 3 and fully connected layer;
[0025] The twin convolutional neural network includes two network branches with the same weight parameters. Each branch contains 3 convolutional layers, 1 maximum pooling layer and 1 fully connected layer. The order is convolutional layer 1>convolutional layer 2>maximum pooling layer>convolutional layer 3>fully connected layer.
[0026] The convolution kernel size and stride of the convolution layer in the two network branches are 3×3 and 1 respectively, the kernel size and stride of the maximum pooling layer are 2×2 and 2 respectively, and the weight vector of the fully connected layer is 1×32768; the normalized source image input in the two branches In the CNN, two 16×16 image blocks are obtained by two sliding windows with a step size of 2 and sent to the twin convolutional neural network. The number of feature maps generated by convolution layer 1, convolution layer 2 and convolution layer 3 on each network branch are 64, 128 and 256 respectively, and the sizes are 16×16, 16×16 and 8×8 respectively. The fully connected layer connects a total of 512 feature maps of size 8×8 generated by convolution layer 3 in the two network branches to form a 32768×1 matrix, and then performs linear weighted operations with two weight vectors of the same size and different elements to obtain two different neurons. In the two network branches, when the two sliding windows traverse the two normalized source images respectively, all the neurons corresponding to the two sliding windows will be obtained. Finally, two initial significant feature maps representing the clarity of the normalized source image are generated in the fully connected layer. Since there is a max pooling layer after convolution layer 2, the size of the 64 feature maps in convolution layer 1 and the 128 feature maps in convolution layer 2 are the same as the normalized source image. The size is consistent, the size of the 256 feature maps in the convolution layer 3 and the initial salient feature map in the fully connected layer is one quarter of the normalized source image;
[0027] Normalize the source image After the twin convolutional neural network model is operated, M feature maps will be generated in the convolution layer, k∈{1,2} is the input normalized source image index, M∈{64,128,256} is the number of convolutional layer feature maps, and the feature maps of the three convolutional layers are uniformly expressed as d∈{1,2,3} is the convolutional layer index, and the initial salient feature map generated by the fully connected layer is expressed as 1: M means from the feature Figure 1 to feature map M.
[0028] Step B of the present invention comprises:
[0029] The nuclear norm of a matrix is the sum of the singular values of the matrix. In , a 5×5×M window is selected by window sliding mode, and it is arranged into a 25×M matrix. The sum of the singular values of the matrix is calculated, which is the nuclear norm of the window, and a pixel value in the initial significant feature map is obtained. When the sliding window traverses the entire feature map, the initial significant feature map of the convolution layer is obtained. The initial significant feature map of each convolution layer is expressed as
[0030] Step C of the present invention comprises:
[0031] The initial salient feature maps of convolutional layer 3 and fully connected layer are represented as and The bicubic interpolation algorithm is used to perform upsampling operations on it, and the significant feature maps of the convolution layer 3 and the fully connected layer are obtained, which are represented as and The calculation formula is:
[0032]
[0033] Where s∈{3,4} is the index value of the initial salient feature map of convolutional layer 3 and fully connected layer, is the point to be interpolated, Points to be interpolated The pixel value at for Nearby 4×4 neighborhood points, m,n∈{0,1,2,3}, For point The pixel value at and is the coefficient value, and its calculation formula is as follows:
[0034]
[0035]
[0036] In the formula, τ is -0.5
[0037] After bicubic interpolation, and The size of the normalized source image The size is consistent, so Get the salient feature map of convolutional layer 1 make Get the salient feature map of convolutional layer 2 In this way, the salient feature maps of each layer are expressed as l∈{1,2,3,4} is the layer index of the salient feature map.
[0038] Step D of the present invention comprises:
[0039] Perform exponential operations on the salient feature maps of each layer and apply the Softmax operator to generate the initial weight map The calculation formula is:
[0040]
[0041] Where e (·) is the exponential operation with base e, is the initial weight map, is the salient feature map of each layer, and l∈{1,2,3,4} is the layer index of the salient feature map.
[0042] Step E of the present invention comprises:
[0043] The initial weight map As the input image of the guided filter, normalize the source image As the guidance image of the guided filter, the output image of the guided filter is According to the representation method of pixel output, the output calculation formula of the guided filter is:
[0044]
[0045] In the formula and The output images of the guided filters are and the normalized source image The pixel value at point θ in the middle, θ is the window ω centered at point z z The pixel point inside, ω z The size of is (2λ+1)×(2λ+1), where λ is the window size radius, and are δ in all overlapping windows containing point θ and point z respectively. z and h z The average value of is calculated as
[0046]
[0047] Where ω θ is a window containing points θ and z, and its size is still (2λ+1)×(2λ+1), δ z and h z The calculation formulas for the values are:
[0048]
[0049] Where μ z and is the normalized source image In ω z The mean and variance in |ω| is ω z The number of pixels within is the initial weight map In ω z The mean of , ε is the regularization parameter to prevent the denominator from being zero. The above calculation process traverses the entire image and can obtain the output image of the guided filter Apply the Softmax operator to generate the weight graph:
[0050]
[0051] In the formula is the weight graph of the lth layer.
[0052] Step F of the present invention comprises:
[0053] If the source image φ k For grayscale images, use the weight map generated by each layer The weighted average fusion algorithm is used to obtain the fused images of each layer. Then the pixel maximization algorithm is used to obtain the final fused image f F , the weighted average fusion calculation formula of each layer is:
[0054]
[0055] In the formula is the image after fusion at the lth layer, is the weight map of the lth layer, is the normalized source image;
[0056] The calculation formula of the pixel maximum algorithm is:
[0057]
[0058] In the formula, max{·} is the maximum value operation, f F is the final fused image, They are the fused images of the first, second, third and fourth layers respectively;
[0059] If the source image φ k In the figure, φ1 is a grayscale image, φ2 is an RGB color image, and the weight map generated by each layer is used. The weighted average fusion algorithm is used to obtain the fused images of each layer. Then the pixel maximization algorithm is used to obtain the grayscale fusion image f Y , the weighted average fusion calculation formula is:
[0060]
[0061] In the formula is the image after fusion at the lth layer, is the weight map of the lth layer, is the normalized source image;
[0062] The calculation formula of the pixel maximum algorithm is:
[0063]
[0064] In the formula, max{·} is the maximum value operation, f Y is the grayscale fusion image, They are the fused images of the first, second, third and fourth layers respectively;
[0065] The grayscale fused image fY Assign to Right now and and Together they form a YUV space image, which is then converted to RGB color space to form the final color fused image f F , the conversion formula from YUV space to RGB color space is:
[0066]
[0067] Where f FR 、f FG and f FB They are red, green and blue three-channel images in RGB color space, and They are the brightness, color and saturation components of the YUV space, respectively. FR 、f FG and f FB The three-channel images form a fused image f in RGB color space F .
[0068] The present invention has the following beneficial effects:
[0069] (1) The pre-trained twin convolutional neural network method is used to automatically extract significant features of multimodal medical images, without the need for image transformation and manual design of image feature extraction methods, which saves time and effort and facilitates the engineering application of multimodal medical image fusion.
[0070] (2) The pre-trained twin convolutional neural network method is used to automatically extract the significant features of multimodal medical images, which can express images more comprehensively and deeply, and realize the representation of image semantics at multiple abstract levels. It does not require training learning data sets and complex hardware configuration, eliminating the previous deep learning fusion method's over-reliance on training data sets and advanced hardware platforms, which is conducive to the construction of a high-precision and low-hardware complexity image fusion platform, and improves the fusion accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0071] Figure 1 is a flow chart of the present invention;
[0072] Figure 2 It is a twin convolutional neural network model diagram of the present invention;
[0073] Figure 3 Schematic diagram of generating initial significant feature maps of each convolutional layer of the present invention;
[0074] Figure 4is a test data set and fusion result of an embodiment of the present invention, wherein A1 is a CT image; A2 is an MRI image; A3 is a fusion image of CT and MRI of a comparison method; A4 is a fusion image of CT and MRI of a comparison method; A5 is a fusion image of CT and MRI of the method of the present invention; B1 is an MRI image; B2 is a PET image; B3 is a fusion image of MRI and PET of a comparison method; B4 is a fusion image of MRI and PET of a comparison method; B5 is a fusion image of MRI and PET of the method of the present invention; C1 is an MRI image; C2 is a SPECT image; C3 is a fusion image of MRI and SPECT of a comparison method; C4 is a fusion image of MRI and SPECT of a comparison method; C5 is a fusion image of MRI and SPECT of the method of the present invention. DETAILED DESCRIPTION
[0075] See also Figure 1 , 2 , including the following steps:
[0076] Step A: For the source image φ k Perform preprocessing to generate the input source image f k , and then normalize it to generate a normalized source image It is used as the input of the pre-trained three-layer twin convolutional neural network to obtain the feature maps of the three convolutional layers, expressed as and the initial salient feature map of the fully connected layer, expressed as k∈{1,2} is the image index, d∈{1,2,3} is the convolution layer index, M∈{64,128,256} is the number of feature maps of the convolution layer, 1:M means from the feature Figure 1 To feature map M;
[0077] Step B: In the feature map of each convolutional layer, select a 5×5×M window in a sliding window manner, arrange it into a 25×M matrix, calculate its nuclear norm, and get a pixel value. When the sliding window traverses the entire feature map, the initial significant feature map of each convolutional layer is obtained, which is expressed as
[0078] Step C: Initial salient feature map of convolutional layer 3 and the initial salient feature map of the fully connected layer Perform upsampling operations to generate the salient feature map of convolution layer 3 And the salient feature map of the fully connected layer Its size is the same as the normalized source image The size is consistent, so Get the salient feature map of convolutional layer 1 make Get the salient feature map of convolutional layer 2 In this way, the salient feature maps of each layer are uniformly represented as l∈{1,2,3,4} is the layer index of the salient feature map;
[0079] Step D: salient feature maps of each layer Perform exponential operations and apply the Softmax operator to generate the initial weight graphs for each layer
[0080] Step E: Normalize the source image As the guide image of the guide filter, the initial weight map of each layer Perform guided filtering to obtain the guided filter output image Apply the Softmax operator to generate the weight graph of each layer
[0081] Step F: Using the weight maps of each layer The weighted average fusion algorithm is used to obtain the fused images of each layer. The pixel maximization algorithm is used to obtain the final fused image f F .
[0082] Step A of the present invention comprises:
[0083] Sub-step A1: pre-processing the source image to generate an input source image, and then performing normalization processing to generate a normalized source image;
[0084] The source image is φ k , if φ k is a grayscale image, then let f k =φ k , get the input source image f k , normalize it and get the normalized source image The calculation formula is:
[0085]
[0086] Where k∈{1,2} is the image index, f k and They represent the kth input source image before normalization and the kth normalized source image after normalization, respectively. kmin and f kmax Respectively represent the kth input source image f k The minimum and maximum values of pixels in ;
[0087] If the source image φ k In the figure, φ1 is a grayscale image and φ2 is an RGB color image. φ2 needs to be preprocessed and converted to YUV space. The conversion formula from RGB to YUV image space is:
[0088]
[0089] Where φ 2R ,φ 2G ,φ 2B Respectively represent the red, green, and blue channel images of the source image φ2 in the RGB color space, Represent the converted images respectively Three component images of brightness, color and saturation in YUV color space;
[0090] Let f1 = φ1, At this time, the input source image f k In the figure, f1 and f2 are both grayscale images, which are normalized to obtain the normalized source image At the same time and The components are also normalized to obtain a normalized image and The calculation formula is:
[0091]
[0092] Where k∈{1,2} is the image index, f k and They represent the kth input source image before normalization and the kth normalized source image after normalization, respectively. kmin and f kmax Respectively represent the kth input source image f k The minimum and maximum values of pixels in U min Indicates the minimum pixel value in the U component image, U min Indicates the maximum pixel value in the U component image, V min Indicates the minimum pixel value in the V component image, V max Indicates the minimum pixel value in the V component image;
[0093] Sub-step A2: Construct a twin convolutional neural network model, where the input to output is: convolutional layer 1, convolutional layer 2, pooling layer, convolutional layer 3 and fully connected layer;
[0094] The twin convolutional neural network includes two network branches with the same weight parameters. Each branch contains 3 convolutional layers, 1 maximum pooling layer and 1 fully connected layer. The order is convolutional layer 1>convolutional layer 2>maximum pooling layer>convolutional layer 3>fully connected layer.
[0095] The convolution kernel size and stride of the convolution layer in the two network branches are 3×3 and 1 respectively, the kernel size and stride of the maximum pooling layer are 2×2 and 2 respectively, and the weight vector of the fully connected layer is 1×32768; the normalized source image input in the two branches In the CNN, two 16×16 image blocks are obtained by two sliding windows with a step size of 2 and sent to the twin convolutional neural network. The number of feature maps generated by convolution layer 1, convolution layer 2 and convolution layer 3 on each network branch are 64, 128 and 256 respectively, and the sizes are 16×16, 16×16 and 8×8 respectively. The fully connected layer connects a total of 512 feature maps of size 8×8 generated by convolution layer 3 in the two network branches to form a 32768×1 matrix, and then performs linear weighted operations with two weight vectors of the same size and different elements to obtain two different neurons. In the two network branches, when the two sliding windows traverse the two normalized source images respectively, all the neurons corresponding to the two sliding windows will be obtained. Finally, two initial significant feature maps representing the clarity of the normalized source image are generated in the fully connected layer. Since there is a max pooling layer after convolution layer 2, the size of the 64 feature maps in convolution layer 1 and the 128 feature maps in convolution layer 2 are the same as the normalized source image. The size is consistent, the size of the 256 feature maps in the convolution layer 3 and the initial salient feature map in the fully connected layer is one quarter of the normalized source image;
[0096] Normalize the source image After the twin convolutional neural network model is operated, M feature maps will be generated in the convolution layer, k∈{1,2} is the input normalized source image index, M∈{64,128,256} is the number of convolutional layer feature maps, and the feature maps of the three convolutional layers are uniformly expressed as d∈{1,2,3} is the convolutional layer index, and the initial salient feature map generated by the fully connected layer is expressed as 1: M means from the feature Figure 1 to feature map M.
[0097] Step B of the present invention comprises:
[0098] The nuclear norm of a matrix is the sum of the singular values of the matrix. In , a 5×5×M window is selected by window sliding mode, and it is arranged into a 25×M matrix. The sum of the singular values of the matrix is calculated, which is the nuclear norm of the window, and a pixel value in the initial significant feature map is obtained. When the sliding window traverses the entire feature map, the initial significant feature map of the convolution layer is obtained. The initial significant feature map of each convolution layer is expressed as The process can be seen in Figure 3 shown.
[0099] Step C of the present invention comprises:
[0100] The initial salient feature maps of convolutional layer 3 and fully connected layer are represented as and The bicubic interpolation algorithm is used to perform upsampling operations on it, and the significant feature maps of the convolution layer 3 and the fully connected layer are obtained, which are represented as and The calculation formula is:
[0101]
[0102] Where s∈{3,4} is the index value of the initial salient feature map of convolutional layer 3 and fully connected layer, is the point to be interpolated, Points to be interpolated The pixel value at for Nearby 4×4 neighborhood points, m,n∈{0,1,2,3}, For point The pixel value at and is the coefficient value, and its calculation formula is as follows:
[0103]
[0104]
[0105] In the formula, τ is -0.5
[0106] After bicubic interpolation, and The size of the normalized source image The size is consistent, so Get the salient feature map of convolutional layer 1 make Get the salient feature map of convolutional layer 2 In this way, the salient feature maps of each layer are expressed as l∈{1,2,3,4} is the layer index of the salient feature map.
[0107] Step D of the present invention comprises:
[0108] Perform exponential operations on the salient feature maps of each layer and apply the Softmax operator to generate the initial weight map The calculation formula is:
[0109]
[0110] Where e (·) is the exponential operation with base e, is the initial weight map, is the salient feature map of each layer, and l∈{1,2,3,4} is the layer index of the salient feature map.
[0111] Step E of the present invention comprises:
[0112] The initial weight map As the input image of the guided filter, normalize the source image As the guidance image of the guided filter, the output image of the guided filter is According to the representation method of pixel output, the output calculation formula of the guided filter is:
[0113]
[0114] In the formula and The output images of the guided filters are and the normalized source image The pixel value at point θ in the middle, θ is the window ω centered at point z z The pixel point inside, ω z The size of is (2λ+1)×(2λ+1), where λ is the window size radius, and are δ in all overlapping windows containing point θ and point z respectively. z and h z The average value of is calculated as
[0115]
[0116] Where ω θ is a window containing points θ and z, and its size is still (2λ+1)×(2λ+1), δ z and h z The calculation formulas for the values are:
[0117]
[0118] Where μ z and is the normalized source image In ω z The mean and variance in |ω| is ω z The number of pixels within is the initial weight map In ω z The mean of , ε is the regularization parameter to prevent the denominator from being zero. The above calculation process traverses the entire image and can obtain the output image of the guided filter Apply the Softmax operator to generate the weight graph:
[0119]
[0120] In the formula is the weight graph of the lth layer.
[0121] Step F of the present invention comprises:
[0122] If the source image φ k For grayscale images, use the weight map generated by each layer The weighted average fusion algorithm is used to obtain the fused images of each layer. Then the pixel maximization algorithm is used to obtain the final fused image f F , the weighted average fusion calculation formula of each layer is:
[0123]
[0124] In the formula is the image after fusion at the lth layer, is the weight map of the lth layer, is the normalized source image;
[0125] The calculation formula of the pixel maximum algorithm is:
[0126]
[0127] In the formula, max{·} is the maximum value operation, f F is the final fused image, They are the fused images of the first, second, third and fourth layers respectively;
[0128] If the source image φ k In the figure, φ1 is a grayscale image, φ2 is an RGB color image, and the weight map generated by each layer is used. The weighted average fusion algorithm is used to obtain the fused images of each layer. Then the pixel maximization algorithm is used to obtain the grayscale fusion image f Y , the weighted average fusion calculation formula is:
[0129]
[0130] In the formula is the image after fusion at the lth layer, is the weight map of the lth layer, is the normalized source image;
[0131] The calculation formula of the pixel maximum algorithm is:
[0132]
[0133] In the formula, max{·} is the maximum value operation, fY is the grayscale fusion image, They are the fused images of the first, second, third and fourth layers respectively;
[0134] The grayscale fused image f Y Assign to Right now and and Together they form a YUV space image, which is then converted to RGB color space to form the final color fused image f F , the conversion formula from YUV space to RGB color space is:
[0135]
[0136] Where f FR 、f FG and f FB They are red, green and blue three-channel images in RGB color space, and They are the brightness, color and saturation components of the YUV space, respectively. FR 、f FG and f FB The three-channel images form a fused image f in RGB color space F .
[0137] In order to solve the problems in the traditional multimodal medical image fusion process, image decomposition cannot optimally separate overlapping features, the degree of automation of image significant feature extraction is low, and it is difficult to optimally realize the weighted or selective fusion of multimodal medical images. At the same time, in order to solve the problems that the deep learning fusion method requires a specific type of image training set and a high-configuration hardware processing platform, which cannot meet the actual application needs. The present invention provides a zero-learning medical image fusion method based on a twin convolutional neural network, which does not require a training data set and high-performance hardware processing equipment. It uses pre-trained twin convolutional neural network technology to automatically extract multimodal medical image features and multi-level abstraction of features, so as to overcome the drawbacks of manual feature extraction in traditional multimodal medical image fusion, so as to enhance the practicality of multimodal medical image fusion and ultimately improve the fusion accuracy of multimodal medical images.
[0138] The present invention provides a zero-learning medical image fusion method based on a twin convolutional neural network. The method utilizes a twin convolutional neural network to automatically extract features from each modality image, generates a significant feature map based on the extracted features, and implements weight allocation through joint guided filtering to effectively fuse medical images of different modalities.
[0139] Convolutional neural network is a trainable multi-stage feedforward artificial neural network. It is a typical deep learning model that can learn the hierarchical feature representation mechanism of signal / image data with different levels of abstraction. Each stage contains a certain number of feature maps corresponding to the feature abstraction level. Each unit or coefficient in the feature map is called a neuron. The feature maps of different stages are connected by performing linear convolution, nonlinear activation and spatial pooling operations on neurons. The twin convolutional neural network contains two convolution branches with exactly the same weights. Each branch consists of three convolution layers and a maximum pooling layer. In order to reduce memory consumption and improve computational efficiency, a pre-trained lightweight twin convolutional network model is used to realize image semantic representation at multiple levels of abstraction, thereby ultimately improving the fusion accuracy of multimodal medical images.
[0140] In order to verify the effectiveness of the multimodal medical image fusion method proposed in the present invention, the scheme of the present invention was tested on multiple real test data sets. Here are the test results on a set of typical test data sets: the test data set is the Harvard whole brain atlas data set (http: / / www.med.harvard.edu / AANLIB / ), which is a public benchmark database established by two scholars Keitha. Johnson and J. Alex Becker at Harvard Medical School mainly for evaluating the performance of multimodal medical image fusion methods. The whole brain atlas data set mainly describes normal and abnormal brain structures, including four imaging types: CT, MRI, PET and SPECT.
[0141] In the invention test example, three groups of brain images in the whole brain atlas dataset are used as experimental data, each group of images is a medical image from a different imaging device. Figure 4 (A1)(A2) are a set of brain slice images of fatal stroke. Figure 4 (A1) is a brain slice image under CT modality; Figure 4 (A2) is a brain slice image under MRI modality; Figure 4 (B1)(B2) is a set of brain slice images of mild Alzheimer's disease. Figure 4 (B1) is a brain slice image under MRI mode. Figure 4 (B2) is a brain slice image under PET modality; Figure 4 (C1)(C2) are a set of brain slice images of metastatic bronchial carcinoma. Figure 4 (C1) is a brain slice image under MRI mode. Figure 4(C2) is a convex image of a brain slice under SPECT modality. In addition, the fusion results obtained by the method of the present invention are compared with the fusion results obtained by the two existing methods [Xu, Zhiping. Medical image fusion using multi-level local extrema [J]. Information Fusion, 2014, 19: 38-48.] (comparison method one) [Liang X, Hu P, Zhang L, et al. MCFNet: Multi-Layer Concatenation Fusion Network for Medical Images Fusion [J]. IEEE Sensors Journal, 2019, 19 (16): 7107-7119.] (comparison method two). The corresponding test results are shown in Figure 2. Figure 4 As shown in (A3)(A4)(A5)(B3)(B4)(B5)(C3)(C4)(C5).
[0142] Comparison method 1: Using the multi-scale local extreme value decomposition method, the source image is decomposed into an approximate layer image and a detail layer image. The approximate layer image adopts the local energy selective fusion rule, and the detail layer image adopts the local area contrast selective fusion rule;
[0143] Comparison method 2: Using the mean square error function as the loss function, training a multi-layer cascade fusion network, using a convolutional neural network to extract the features of the input image, then cascading the feature map, using a convolution operation to encode the cascade feature map, and finally using a deconvolution operation to obtain a fused image. The parameters of the guide filter used in the method of the present invention are window radius λ=1 and regularization parameter ε=0.0001. Figure 4 (A3)(A4)(A5) are respectively comparative method 1, comparative method 2 and the method of the present invention. Figure 4 (A1)(A2) shows the fusion results of a set of CT and MRI images; Figure 4 (B3)(B4)(B5) are respectively comparative method 1, comparative method 2 and the method of the present invention. Figure 4 (B1)(B2) shows the fusion results of a set of MRI and PET images; Figure 4 (C3)(C4)(C5) are respectively the comparison method 1, the comparison method 2 and the method of the present invention. Figure 4 (C1)(C2) shows the fusion results of a set of MRI and SPECT images;
[0144] In the fusion results of CT and MRI images, compared with the fusion results of method 1, the fused image loses a lot of energy, the overall brightness and contrast are low, and there are slight artifacts, which is not conducive to visual observation. Figure 4 (A3); Compared with the fusion result of method 2, the brightness and contrast of the fused image are improved, but the edge is not detailed and the soft tissue information is missing. Figure 4 (A4); In the fusion result of the method of the present invention, the structural information of the fused image is clear and complete, the complementary information of the CT and MRI images is well preserved, and the edge detail information is preserved without generating artifacts and false information, such as Figure 4 (A5).
[0145] In the fusion results of MRI and PET images, compared with the fusion results of method 1, the overall brightness of the fused image is lower, with slight color distortion and structural information loss, such as Figure 4 (B3); Compared with the fusion result of method 2, the fused image has serious color distortion and serious loss of edge detail information, such as Figure 4 (B4), in the fusion result of the method of the present invention, the fused image has rich color information, is clear and natural, has high brightness and contrast, and the structure and intensity information are well preserved, with good visual experience. Figure 4 (B5).
[0146] In the fusion results of MRI and SPECT images, the fusion results of comparison method 1 and comparison method 2 both have color distortion problems in the fused images, and some important functional information contained in the SPECT source image is lost, such as Figure 4 (C3)(C4); In the fusion result of the method of the present invention, the fused image is very close to the SPECT source image, the image brightness and contrast are moderate, and the structural details are clear, such as Figure 4 (C5).
[0147] In order to better evaluate the performance of the fusion method, in addition to the above-mentioned subjective evaluation, objective evaluation indicators are used to objectively analyze the fusion results of different methods. The objective evaluation indicators used in the present invention include structural information similarity (SSIM), human visual system perception quality evaluation coefficient (Q HVS ), feature mutual information (FMIw), edge fusion quality factor (Q E ) and visual information fidelity (VIFF). SSIM evaluates the quality of the fused image by using the similarity between the fused image and the input source image in terms of brightness, contrast, and structure; Q HVSIt is based on the human eye visual characteristic model and evaluates the quality of the fused image by calculating the degree of preservation of the perceived contrast between the fused image and the input source image. FMIw evaluates the quality of the fused image by calculating the mutual information between the feature information of the fused image and the input source image. E The quality of the fused image is evaluated by the structural information similarity of the weighted edge image with significant features. VIFF is an image quality evaluation index that combines the natural image statistical model, the image distortion model, and the human visual system model. The larger the value of the above evaluation index, the better the quality of the fused image, and vice versa. The objective evaluation index data are shown in Table 1, Table 2, and Table 3.
[0148] Table 1 shows the objective evaluation index data of CT and MRI image fusion results
[0149]
[0150] Table 2 shows the objective evaluation index data of MRI and PET image fusion results
[0151]
[0152] Table 3 shows the objective evaluation index data of MRI and SPECT image fusion results
[0153]
[0154] It can be seen from Table 1, Table 2 and Table 3 that the result images obtained by the fusion method of the present invention are superior to those of the comparison method 1 and the comparison method 2 in terms of objective index evaluation, which shows that the method of the present invention can stably obtain high-quality fusion results.
[0155] So far, the embodiments of the present invention have been described in detail in conjunction with the accompanying drawings. Based on the above description, those skilled in the art should have a clear understanding of the zero-learning medical image fusion method based on the twin convolutional neural network of the present invention.
[0156] The zero-learning medical image fusion method based on twin convolutional neural networks of the present invention uses a pre-trained twin convolutional neural network to automatically extract multimodal medical image features. It does not require manual feature selection, does not require specific training learning data sets and advanced hardware equipment, reduces memory requirements, improves computing efficiency, facilitates the engineering application of medical image fusion methods, and achieves excellent results in the field of medical image fusion.
Claims
1. A zero-learning medical image fusion method based on twin convolutional neural network, characterized in that: The following steps are involved: Step A: For the source image φ k Perform preprocessing to generate the input source image f k , and then normalize it to generate a normalized source image It is used as the input of the pre-trained three-layer twin convolutional neural network to obtain the feature maps of the three convolutional layers, expressed as and the initial salient feature map of the fully connected layer, expressed as k∈{1,2} is the image index, d∈{1,2,3} is the convolution layer index, M∈{64,128,256} is the number of feature maps of the convolution layer, 1:M means from feature map 1 to feature map M; Step B: In the feature map of each convolutional layer, select a 5×5×M window in a sliding window manner, arrange it into a 25×M matrix, calculate its nuclear norm, and get a pixel value. When the sliding window traverses the entire feature map, the initial significant feature map of each convolutional layer is obtained, which is expressed as Step C: Initial salient feature map of convolutional layer 3 and the initial salient feature map of the fully connected layer Perform upsampling operations to generate the salient feature map of convolution layer 3 And the salient feature map of the fully connected layer Its size is the same as the normalized source image The size is consistent, so Get the salient feature map of convolutional layer 1 make Get the salient feature map of convolutional layer 2 In this way, the salient feature maps of each layer are uniformly represented as l∈{1,2,3,4} is the layer index of the salient feature map; Step D: salient feature maps of each layer Perform exponential operations and apply the Softmax operator to generate the initial weight graphs for each layer Step E: Normalize the source image As the guide image of the guide filter, the initial weight map of each layer Perform guided filtering to obtain the guided filter output image Apply the Softmax operator to generate the weight graph of each layer Step F: Using the weight maps of each layer The weighted average fusion algorithm is used to obtain the fused images of each layer. The pixel maximization algorithm is used to obtain the final fused image f F .
2. According to claim 1, a zero-learning medical image fusion method based on a twin convolutional neural network is characterized in that: The step A comprises: Sub-step A1: pre-processing the source image to generate an input source image, and then performing normalization processing to generate a normalized source image; The source image is φ k , if φ k is a grayscale image, then let f k =φ k , get the input source image f k , normalize it and get the normalized source image The calculation formula is: Where k∈{1,2} is the image index, f k and They represent the kth input source image before normalization and the kth normalized source image after normalization, respectively. kmin and f kmax Respectively represent the kth input source image f k The minimum and maximum values of pixels in ; If the source image φ k In the figure, φ1 is a grayscale image and φ2 is an RGB color image. φ2 needs to be preprocessed and converted to YUV space. The conversion formula from RGB to YUV image space is: Where φ 2R ,φ 2G ,φ 2B Respectively represent the red, green, and blue channel images of the source image φ2 in the RGB color space, Represent the converted images respectively Three component images of brightness, color and saturation in YUV color space; Let f1 = φ1, At this time, the input source image f k In the figure, both f1 and f2 are grayscale images, which are normalized to obtain the normalized source image. At the same time and The components are also normalized to obtain a normalized image and The calculation formula is: Where k∈{1,2} is the image index, f k and They represent the kth input source image before normalization and the kth normalized source image after normalization, respectively. kmin and f kmax Respectively represent the kth input source image f k The minimum and maximum values of pixels in U min Indicates the minimum pixel value in the U component image, U min Indicates the maximum pixel value in the U component image, V min Indicates the minimum pixel value in the V component image, V max Indicates the minimum pixel value in the V component image; Sub-step A2: Construct a twin convolutional neural network model, where the input to output is: convolutional layer 1, convolutional layer 2, pooling layer, convolutional layer 3 and fully connected layer; The twin convolutional neural network includes two network branches with the same weight parameters. Each branch contains 3 convolutional layers, 1 maximum pooling layer and 1 fully connected layer. The order is convolutional layer 1>convolutional layer 2>maximum pooling layer>convolutional layer 3>fully connected layer. The convolution kernel size and stride of the convolution layer in the two network branches are 3×3 and 1 respectively, the kernel size and stride of the maximum pooling layer are 2×2 and 2 respectively, and the weight vector of the fully connected layer is 1×32768; the normalized source image input in the two branches In the CNN, two 16×16 image blocks are obtained by two sliding windows with a step size of 2 and sent to the twin convolutional neural network. The number of feature maps generated by convolution layer 1, convolution layer 2 and convolution layer 3 on each network branch are 64, 128 and 256 respectively, and the sizes are 16×16, 16×16 and 8×8 respectively. The fully connected layer concatenates a total of 512 feature maps of size 8×8 generated by convolution layer 3 in the two network branches to form a 32768×1 matrix, and then performs linear weighted operations with two weight vectors of the same size and different elements to obtain two different neurons. In the two network branches, when the two sliding windows traverse the two normalized source images respectively, all the neurons corresponding to the two sliding windows will be obtained. Finally, two initial significant feature maps representing the clarity of the normalized source image are generated in the fully connected layer. Since there is a max pooling layer after convolution layer 2, the size of the 64 feature maps in convolution layer 1 and the 128 feature maps in convolution layer 2 are the same as the normalized source image. The size is consistent, the size of the 256 feature maps in the convolution layer 3 and the initial salient feature map in the fully connected layer is one quarter of the normalized source image; Normalize the source image After the twin convolutional neural network model is operated, M feature maps will be generated in the convolution layer, k∈{1,2} is the input normalized source image index, M∈{64,128,256} is the number of convolutional layer feature maps, and the feature maps of the three convolutional layers are uniformly expressed as d∈{1,2,3} is the convolutional layer index, and the initial salient feature map generated by the fully connected layer is expressed as 1:M represents from feature map 1 to feature map M.
3. The zero-learning medical image fusion method based on twin convolutional neural network according to claim 1, characterized in that: The step B comprises: The nuclear norm of a matrix is the sum of the singular values of the matrix. In , a 5×5×M window is selected by window sliding mode, and it is arranged into a 25×M matrix. The sum of the singular values of the matrix is calculated, which is the nuclear norm of the window, and a pixel value in the initial significant feature map is obtained. When the sliding window traverses the entire feature map, the initial significant feature map of the convolution layer is obtained. The initial significant feature map of each convolution layer is expressed as 4. The zero-learning medical image fusion method based on a twin convolutional neural network according to claim 1, characterized in that: The step C comprises: The initial salient feature maps of convolutional layer 3 and fully connected layer are represented as and The bicubic interpolation algorithm is used to perform upsampling operations on it, and the significant feature maps of the convolution layer 3 and the fully connected layer are obtained, which are represented as and The calculation formula is: Where s∈{3,4} is the index value of the initial salient feature map of convolutional layer 3 and fully connected layer, is the point to be interpolated, Points to be interpolated The pixel value at for Nearby 4×4 neighborhood points, m,n∈{0,1,2,3}, For point The pixel value at and is the coefficient value, and its calculation formula is as follows: In the formula, τ is -0.5 After bicubic interpolation, and The size of the normalized source image The size is consistent, so Get the salient feature map of convolutional layer 1 make Get the salient feature map of convolutional layer 2 In this way, the salient feature maps of each layer are expressed as l∈{1,2,3,4} is the layer index of the salient feature map.
5. The zero-learning medical image fusion method based on a twin convolutional neural network according to claim 1, characterized in that: The step D comprises: Perform exponential operations on the salient feature maps of each layer and apply the Softmax operator to generate the initial weight map The calculation formula is: Where e (·) is the exponential operation with base e, is the initial weight map, is the salient feature map of each layer, and l∈{1,2,3,4} is the layer index of the salient feature map.
6. The zero-learning medical image fusion method based on a twin convolutional neural network according to claim 1, characterized in that: The step E comprises: The initial weight map As the input image of the guided filter, normalize the source image As the guidance image of the guided filter, the output image of the guided filter is According to the representation method of pixel output, the output calculation formula of the guided filter is: In the formula and The output images of the guided filters are and the normalized source image The pixel value at point θ in the middle, θ is the window ω centered at point z z The pixel point inside, ω z The size of is (2λ+1)×(2λ+1), where λ is the window size radius, and are δ in all overlapping windows containing point θ and point z respectively. z and h z The average value of is calculated as Where ω θ is a window containing points θ and z, and its size is still (2λ+1)×(2λ+1), δ z and h z The calculation formulas for the values are: Where μ z and is the normalized source image In ω z The mean and variance in |ω| is ω z The number of pixels within is the initial weight map In ω z The mean of , ε is the regularization parameter to prevent the denominator from being zero. The above calculation process traverses the entire image and can obtain the output image of the guided filter Apply the Softmax operator to generate the weight graph: In the formula is the weight graph of the lth layer.
7. The zero-learning medical image fusion method based on twin convolutional neural network according to claim 1, characterized in that: The step F comprises: If the source image φ k For grayscale images, use the weight map generated by each layer The weighted average fusion algorithm is used to obtain the fused images of each layer. Then the pixel maximization algorithm is used to obtain the final fused image f F , the weighted average fusion calculation formula of each layer is: In the formula is the image after fusion at the lth layer, is the weight map of the lth layer, is the normalized source image; The calculation formula of the pixel maximum algorithm is: In the formula, max{·} is the maximum value operation, f F is the final fused image, They are the fused images of the first, second, third and fourth layers respectively; If the source image φ k In the figure, φ1 is a grayscale image, φ2 is an RGB color image, and the weight map generated by each layer is used. The weighted average fusion algorithm is used to obtain the fused images of each layer. Then the pixel maximization algorithm is used to obtain the grayscale fusion image f Y , the weighted average fusion calculation formula is: In the formula is the image after fusion at the lth layer, is the weight map of the lth layer, is the normalized source image; The calculation formula of the pixel maximum algorithm is: In the formula, max{·} is the maximum value operation, f Y is the grayscale fusion image, They are the fused images of the first, second, third and fourth layers respectively; The grayscale fused image f Y Assign to Right now and and Together they form a YUV space image, which is then converted to RGB color space to form the final color fused image f F , the conversion formula from YUV space to RGB color space is: Where f FR 、f FG and f FB They are red, green and blue three-channel images in RGB color space, and They are the brightness, color and saturation components of the YUV space, respectively. FR 、f FG and f FB The three-channel images form a fused image f in RGB color space F .
