Multi-focus microscopic image fusion method based on convolutional neural network

Abstract

The invention discloses a multi-focus microscopic image fusion method based on a convolutional neural network. The convolutional neural network is trained to fuse microscopic images of a plurality of focal layers. A plurality of partially focused images in the same scene are given, and feature extraction is performed through a convolutional neural network to obtain significant image features. Afterwards, the convolution features of a plurality of input images are fused according to the element-by-element maximum fusion rule, and the fused features are reconstructed through two convolution layers, so that a fused image with rich information is generated. According to the method, the images of multiple focal planes of the pathological section under microscopic scanning are input and effectively fused, so that the detail representation of the microscopic image is improved, the fused image has good visibility in focal and non-focal areas, and the advantages of flexibility and generalization of deep learning are fully utilized. Parameters which need to be manually and continuously adjusted in a traditional method are replaced by a neural network structure, and convergence of an algorithm and reconstruction precision are theoretically supported.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer and image processing, and in particular, relates to a multi-focus microscopic image fusion method based on convolutional neural network. Background Art

[0002] Microscopes are indispensable research tools in the fields of biological science, material science, medicine, etc. Through microscopic imaging, researchers can deeply observe microscopic structures and processes. Optical microscopes can be used to observe the morphology and characteristics of microorganisms such as bacteria, monitor morphological changes of cells, and diagnose diseases such as cancer.

[0003] When people use a high-magnification microscope to observe samples, it is difficult to obtain a globally clear microscopic image by adjusting the focus of the microscope. Since the structure of the observed specimen is three-dimensional and the surface is usually uneven, only those structures near the focal plane are clear during imaging, and the parts of the specimen outside the focal plane will be out of focus. Usually, the focal depth of the lens in the microscope optical system is limited. When the three-dimensional depth of the observed specimen is far beyond the depth of field, it is difficult to obtain a fully focused and clear image of a large-sized sample in a single imaging. At the same time, there is a mutually restrictive relationship between the depth of field and the microscopic magnification. The greater the magnification of the lens, the more severely the imaging process is restricted by the depth of field, and the more obvious the image blur. Therefore, it is impossible to obtain a clear image of the entire object with only one focusing step. By adjusting the position of the lens multiple times along the specimen z-axis, a series of local focused images are collected at different depths of the specimen.

[0004] If you want the whole image to be in focus, you can use the multi-focus image fusion method. Multi-focus image fusion aims to extract features from two or more partially focused images of the same scene, fuse their respective clear areas, and finally obtain a fully clear image. As an important branch of modern imaging technology, multi-focus image fusion technology has received widespread attention and application in recent years.

[0005] The research on multi-focus fusion technology of microscopic images has a profound theoretical foundation and broad application prospects. With the development of medical imaging, microscopic image fusion technology can play an important role in early diagnosis of diseases, formulation of treatment plans and efficacy evaluation. In cancer research, pathologists usually need to combine the patient's pathological sections for comprehensive analysis. Microscopic image fusion technology can make the information of pathological samples more comprehensive and improve the accuracy of diagnosis. Doctors can more accurately judge the nature and development stage of the disease, so as to formulate personalized treatment plans, which is of great significance to improving medical standards and the quality of life of patients. In the field of life sciences, microscopic image fusion is of great significance for reconstructing clear images. For example, the monitoring of phytoplankton cells is an important part of water quality biological evaluation. Some algae cells are large in size or have filamentous morphological structures. For example, the length of Anabaena can reach hundreds of microns. During microscopic imaging, the distance of algae cells in the depth of field direction is easy to exceed the depth of field range of the microscope, resulting in blurring caused by defocusing in the area outside the depth of field range in the microscopic image. The acquisition of multiple microscopic images and the use of image fusion methods provide ideas for obtaining full-depth microscopic imaging of phytoplankton cells.

[0006] Therefore, it is of great practical significance to develop an efficient multi-focus microscopic image fusion method to improve the detail performance of microscopic images and ensure that the fused image has good visibility in both the focal and non-focal areas. Summary of the invention

[0007] In order to solve the problem that the depth of field of the lens in the microscope optical system is limited and a fully focused clear image cannot be obtained, a multi-focus microscopic image fusion method based on a convolutional neural network is proposed. The present invention is implemented by the following technical solutions:

[0008] The present invention discloses a multi-focus microscopic image fusion method based on a convolutional neural network, and the specific steps are as follows:

[0009] 1) Obtain microscopic images of multiple focal planes, convert them into tensors and normalize them before input to obtain normalized data;

[0010] 2) Construct an image fusion framework based on convolutional neural network, which includes three main modules: feature extraction module, feature fusion module and image reconstruction module;

[0011] 3) Based on the normalized data obtained in step 1), the first two are selected as input, and rich low-level features are extracted from the normalized data through two convolutional layers of the feature extraction module in step 2);

[0012] 4) Based on the low-level features obtained in step 3), the convolutional features of multiple inputs are fused through the feature fusion module in step 2);

[0013] 5) Based on the fusion features obtained in step 4), the image reconstruction module in step 2) reconstructs the fused features through two convolutional layers to generate a preliminary fusion image;

[0014] 6) Based on the preliminary fused image obtained in step 5), convert it into a tensor and normalize it, input the obtained normalized data and the third normalized data of the multiple focal plane microscopic images in step 1) into the convolutional neural network, repeat the cycle of steps 3)-5) to obtain the second fused image, and repeat in sequence until the last normalized data of the multiple focal plane microscopic images participates in the fusion to obtain the final fused image.

[0015] As a further improvement, in step 2) of the present invention: the image fusion framework based on convolutional neural network includes three main modules: feature extraction, feature fusion and image reconstruction.

[0016] As a further improvement, in step 3) of the present invention: the feature extraction module includes two convolutional layers CONV1 and CONV2; the first convolutional layer of the advanced ResNet101 pre-trained on ImageNet is used as the first convolutional layer, CONV1 includes 64 convolutional kernels of size 7×7, a step size of 1, and a padding of 3, the convolutional kernels are sufficient to extract a wide range of image features, and CONV1 has been trained on the largest natural image dataset ImageNet; CONV1 is used to extract effective image features, so that the parameters of CONV1 are fixed when training the proposed model; the features extracted by CONV1 are initially used for classification tasks, and directly inputting them into the feature fusion module may not be suitable for image fusion tasks; a second convolutional layer CONV2 is added to adjust the convolutional features of CONV1, CONV2 includes 64 3×3 convolutional kernels, a step size of 1, and a padding of 1 to adapt to feature fusion.

[0017] As a further improvement, in step 4) of the present invention: the feature fusion module adopts the element-by-element maximum fusion rule, and the specific formula is:

[0018]

[0019] in, represents the j-th feature map of the i-th input image extracted by CONV2, It represents the j-th channel of the feature map fused by the feature fusion module, fuse represents the element-wise maximum fusion rule, and N represents the number of input images.

[0020] As a further improvement, in step 5) of the present invention: the image reconstruction module includes two convolutional layers CONV3 and CONV4; CONV3 includes 64 3×3 convolution kernels with a step size of 1 and a padding of 1; CONV4 includes 3 1×1 convolution kernels, which are used to compress the feature map of 64 channels to 3 channels.

[0021] As a further improvement, in step 6) of the present invention, in the training of the microscopic image fusion network, perceptual loss and basic loss are used to optimize and update the parameter configuration of the network, and the specific loss function is:

[0022]

[0023]

[0024] T loss =ω 1 P loss +ω 2 B loss

[0025] Among them, P loss is the perceptual loss, which is composed of the mean square error between the predicted fused image feature map extracted by the last convolutional layer of ResNet101 and the label fused image. and They represent the feature map of the predicted fused image and the feature map of the fused image ground truth, i is the channel index of the feature map, C f , H f , W f Respectively represent the number of channels, height, and width of the feature map; B loss The basic loss is and They represent the predicted fused image and the ground truth of the fused image respectively, i is the RGB image channel index, H g , W g are the height and width of the fused image ground truth respectively; T loss is the total loss, ω 1 and ω 2 is the weight coefficient, set to 1.

[0026] The beneficial effects of the present invention are as follows:

[0027] The present invention discloses a multi-focus microscopic image fusion method based on a convolutional neural network, the purpose of which is to train a convolutional neural network to fuse microscopic images of multiple focal planes. Given multiple partially focused images in the same scene, the method first extracts features through a convolutional neural network to obtain significant image features. Afterwards, the convolutional features of multiple input images are fused through an element-by-element maximum fusion rule. Finally, the fused features are reconstructed through two convolutional layers to generate an information-rich fused image. The present invention improves the detail expression of the microscopic image by inputting images of multiple focal planes of pathological sections under microscopic scanning and effectively fuses these images, thereby achieving good visibility of the fused image in both the focal and non-focal areas. The method makes full use of the flexibility and generalization advantages of deep learning. The neural network structure is used to replace the parameters that need to be manually adjusted continuously in the traditional method, and the convergence of the algorithm and the accuracy of reconstruction are both theoretically supported.

[0028] In the image fusion framework of the present invention, two loss functions are introduced in the optimization process: perceptual loss and basic loss. Perceptual loss uses the pre-trained network of ResNet101 to extract the image feature map, calculates the mean square error between the predicted fused image and the label image, and further optimizes the information retention in the feature extraction and fusion process. The basic loss directly measures the pixel-level difference between the predicted image and the actual image, and effectively guides the reconstruction module to adjust the image details. By jointly optimizing these two loss functions, the global consistency and high-quality reconstruction effect of the fused image in space and content are ensured, thereby improving the overall performance and adaptability of the image fusion framework. This method can make the information of pathological samples more comprehensive, improve the analysis accuracy and diagnostic efficiency of pathological samples, and then more accurately judge the nature and development stage of the disease, so as to formulate personalized treatment plans, which is of great significance to improving medical standards and the quality of life of patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of the image fusion method based on convolutional neural network of the present invention;

[0030] Figure 2 These are 16 source images of brain nerve slices in the present invention;

[0031] Figure 3 is the fused brain nerve slice image in the present invention;

[0032] Figure 4 These are 5 source images of the cell smear in the present invention;

[0033] Figure 5 This is the cell smear image after fusion in the present invention. DETAILED DESCRIPTION

[0034] In order to describe the present invention more clearly, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0035] like Figure 1 As shown, the specific implementation steps of the multi-focus microscopic image fusion method based on convolutional neural network of the present invention are as follows:

[0036] 1) Obtain microscopic images of multiple focal planes, convert them into tensors and normalize them before input;

[0037] 2) Construct an image fusion framework based on convolutional neural networks, which includes three main modules: feature extraction, feature fusion and image reconstruction;

[0038] 3) Based on the normalized data obtained in step 1), the first two are selected as input, and after two convolutional layers of the feature extraction module in step 2), rich low-level features are extracted from the normalized data; the feature extraction module contains two convolutional layers CONV1 and CONV2; the first convolutional layer of the advanced ResNet101 pre-trained on ImageNet is used as the first convolutional layer, CONV1 contains 64 convolutional kernels of size 7×7, with a step size of 1 and a padding of 3, which are sufficient to extract a wide range of image features, and CONV1 has been trained on the largest natural image dataset (i.e., ImageNet); therefore, CONV1 can be used to extract effective image features, so the parameters of CONV1 are fixed when training the proposed model; however, the features extracted by CONV1 are originally used for classification tasks, so directly inputting them into the feature fusion module may not be suitable for image fusion tasks; therefore, the second convolutional layer CONV2 is added to adjust the convolutional features of CONV1, and CONV2 contains 64 convolutional kernels of size 3×3, with a step size of 1 and a padding of 1 to adapt to feature fusion. ;

[0039] 4) Based on the low-level features obtained in step 3), the convolutional features of multiple inputs are fused through the feature fusion module in step 2), and the element-by-element maximum fusion rule is adopted. The specific formula is:

[0040]

[0041] in, represents the j-th feature map of the i-th input image extracted by CONV2, represents the j-th channel of the feature map fused by the feature fusion module, fuse represents the element-by-element maximum fusion rule, and N represents the number of input images;

[0042] 5) Based on the fusion features obtained in step 4), the image reconstruction module in step 2) reconstructs the fused features through two convolutional layers to generate a preliminary fusion image. The image reconstruction module includes two convolutional layers CONV3 and CONV4; CONV3 contains 64 3×3 convolution kernels with a step size of 1 and a padding of 1; CONV4 contains 3 1×1 convolution kernels, the purpose of which is to compress the 64-channel feature map into 3 channels;

[0043] 6) Based on the preliminary fused image obtained in step 5), convert it into a tensor and normalize it, input the obtained normalized data and the third normalized data of the multiple focal plane microscopic images in step 1) into the convolutional neural network, repeat the cycle of steps 3)-5) to obtain the second fused image, and repeat in sequence until the last normalized data of the multiple focal plane microscopic images participates in the fusion to obtain the final fused image.

[0044] 7) Based on the final fused image obtained in step 6), the perceptual loss and basic loss are used to optimize the parameter configuration of the updated network. The specific loss function is:

[0045]

[0046]

[0047] T loss =ω 1 P loss +ω 2 B loss

[0048] Among them, P loss is the perceptual loss, which is composed of the mean square error between the predicted fused image feature map extracted by the last convolutional layer of ResNet101 and the label fused image. and They represent the feature map of the predicted fused image and the feature map of the fused image ground truth, i is the channel index of the feature map, C f , H f , W f Respectively represent the number of channels, height, and width of the feature map; B loss The basic loss is and They represent the predicted fused image and the ground truth of the fused image respectively, i is the RGB image channel index, H g , W g are the height and width of the fused image ground truth respectively; T loss is the total loss, ω 1 and ω 2 is the weight coefficient, set to 1.

[0049] Combining the above-mentioned feature extraction module, feature fusion module, image reconstruction module and other parts, the structure of the multi-focus microscopic image fusion method based on convolutional neural network proposed in the present invention is as follows: Figure 1 shown.

[0050] In the specific implementation process, the deep learning training and testing of the present invention are based on pytorch and python environment, and the graphics card used is a single NVIDIATITAN RTX with a video memory of 24G. Some basic settings in the image fusion framework are: batch size is 32, learning rate size is 0.001, training times is 5000, and the optimization algorithm uses the Adam optimizer.

[0051] In order to verify the fusion effect of the network model in this method, a large and real multi-focus dataset Real-MFF was used, which contains 710 pairs of source images and corresponding gold standards. This dataset is generated by light field images, and both the source images and ground truth images are real. Three sets of image sets Real-mff1, Real-mff2, and Real-mff3 in Real-MFF were selected to evaluate the training effect, and the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) of the final results and the gold standard were calculated. The peak signal-to-noise ratio is used to measure the ratio between the effective information and noise of the image, which can reflect whether the image is distorted. The larger the PSNR value, the better the quality of the fused image. The structural similarity has a value range of [-1,1]. The closer it is to 1, the higher the similarity and the better the fusion quality. The specific calculation formula is as follows:

[0052]

[0053]

[0054]

[0055] Among them, MSE is the fusion image (I 1 ) and the ideal / reference image (I 2 ), where n is the number of pixels. L corresponds to the maximum dynamic range of the number of pixels. In a standard 8-bit image, L is equal to 255. In the SSIM equation, M corresponds to the number of local windows in the image, and l, c, and s correspond to brightness, contrast, and structural similarity, respectively.

[0056] Table 1 Evaluation indicators PSNR, SSIM and fusion time of three groups of image sets

[0057] Dataset PSNR(dB) SSIM Fusion time (s) Real-mff1 35.39 0.9565 1.389 Real-mff2 31.21 0.9227 1.551 Real-mff3 34.86 0.9519 1.713

[0058] Table 2 Spatial frequency SF and average gradient AG of source images and fused images of three image sets

[0059]

[0060] The spatial frequency SF (Spatial Frequency) reflects the rate of change of the image grayscale. The larger the spatial frequency, the clearer the image and the better the quality of the fused image. The average gradient AG (Average Gradient) can be used to measure the clarity of the fused image. It can be considered that the larger the average gradient, the better the image clarity and the better the fusion quality. As can be seen from Tables 1 and 2, the fused image has been improved in all indicators compared with the source image.

[0061] In order to test the fusion effect of the network model on microscopic images in this method, 16 source images of brain nerve slices and 5 source images of cell smears were used, as shown in the following figure. Figure 2 and Figure 4 Due to the lack of a gold standard, after obtaining the final result, the spatial frequency SF and average gradient AG of the final result and each source image were calculated.

[0062] The fusion time of brain nerve slices is 7.325s. Figure 3 Table 3 shows the maximum, minimum and average values ​​of the spatial frequency SF and average gradient AG of the 16 source images for the fusion results. The spatial frequency SF of the final fused image is 5.739, and the average gradient AG is 21.366. It can be seen that the indicators of the final fused image of the brain nerve slice are better than those of all source images.

[0063] Table 3 Spatial frequency SF and average gradient AG of 16 source images of brain nerve slices

[0064] Evaluation indicators quantity Maximum Minimum average value SF 16 3.068 1.612 2.656 AG 16 15.778 6.871 12.891

[0065] The fusion time of the cell smear was 3.098s. Figure 5 Table 4 is the fusion result. The spatial frequency and average gradient of the cell smear fusion image and the source image are shown in Table 4. It can be seen from Table 4 that both indicators of the fusion image are better than those of all source images.

[0066] Table 4 Cell smear source images and fused images

[0067] Number of image stacks Spatial frequency Average Gradient Fusion 5.847 24.058 -2 4.165 19.471 -1 4.227 19.882 0 4.485 21.327 1 4.471 21.698 2 3.493 16.182

[0068] In order to analyze whether the fusion order of multiple images in this method affects the fusion effect, two sets of images were fused in the order of image stack numbers from small to large and from large to small. In the above-mentioned experiment, the fusion order was based on the number of stacks from small to large, such as the brain nerve slices were fused from -16 to -1 in a loop, and the cell smears were fused in the order of -2, -1, 0, 1, 2. The comparison of the spatial frequency SF and the average gradient AG of the fused image is shown in Table 5. It can be seen that the fusion image obtained by the fusion order from small to large image stack numbers is better in both indicators.

[0069] Table 5 Spatial frequency SF and average gradient AG of fused images under two fusion orders

[0070]

[0071] The above description of the embodiments is to facilitate the understanding and application of the present invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments, and improvements and modifications made by those skilled in the art based on the disclosure of the present invention should be within the scope of protection of the present invention.

Claims

1. A multi-focus microscopic image fusion method based on convolutional neural network, the specific steps are as follows: 1) Obtain microscopic images of multiple focal planes, convert them into tensors and normalize them before input to obtain normalized data; 2) Construct an image fusion framework based on convolutional neural network, which includes three main modules: feature extraction module, feature fusion module and image reconstruction module; 3) Based on the normalized data obtained in step 1), the first two are selected as input, and rich low-level features are extracted from the normalized data through two convolutional layers of the feature extraction module in step 2); 4) Based on the low-level features obtained in step 3), the convolutional features of multiple inputs are fused through the feature fusion module in step 2); 5) Based on the fusion features obtained in step 4), the image reconstruction module in step 2) reconstructs the fused features through two convolutional layers to generate a preliminary fusion image; 6) Based on the preliminary fused image obtained in step 5), convert it into a tensor and normalize it, input the obtained normalized data and the third normalized data of the multiple focal plane microscopic images in step 1) into the convolutional neural network, repeat the cycle of steps 3)-5) to obtain the second fused image, and repeat in sequence until the last normalized data of the multiple focal plane microscopic images participates in the fusion to obtain the final fused image.

2. The multi-focus microscopic image fusion method based on convolutional neural network according to claim 1, characterized in that: In the step 2): the image fusion framework based on convolutional neural network includes three main modules: feature extraction, feature fusion and image reconstruction.

3. The multi-focus microscopic image fusion method based on convolutional neural network according to claim 1, characterized in that: In the step 3), the feature extraction module includes two convolutional layers CONV1 and CONV2; the first convolutional layer of the advanced ResNet101 pre-trained on ImageNet is used as the first convolutional layer, CONV1 includes 64 convolutional kernels of size 7×7, a step size of 1, and a padding of 3, the convolutional kernels are sufficient to extract a wide range of image features, and CONV1 has been trained on the largest natural image dataset ImageNet; CONV1 is used to extract effective image features, so that the parameters of CONV1 are fixed when training the proposed model; the features extracted by CONV1 are initially used for classification tasks, and directly inputting them into the feature fusion module may not be suitable for image fusion tasks; a second convolutional layer CONV2 is added to adjust the convolutional features of CONV1, CONV2 includes 64 3×3 convolutional kernels, a step size of 1, and a padding of 1 to adapt to feature fusion.

4. The multi-focus microscopic image fusion method based on convolutional neural network according to claim 1, characterized in that: In step 4), the feature fusion module adopts the element-by-element maximum fusion rule, and the specific formula is: in, represents the j-th feature map of the i-th input image extracted by CONV2, It represents the j-th channel of the feature map fused by the feature fusion module, fuse represents the element-wise maximum fusion rule, and N represents the number of input images.

5. The multi-focus microscopic image fusion method based on convolutional neural network according to claim 1, characterized in that: In the step 5), the image reconstruction module includes two convolutional layers CONV3 and CONV4; CONV3 includes 64 3×3 convolution kernels with a step size of 1 and a padding of 1; CONV4 includes 3 1×1 convolution kernels for compressing the feature map of 64 channels to 3 channels.

6. The multi-focus microscopic image fusion method based on convolutional neural network according to claim 1, characterized in that: In the step 6), in the training of the microscopic image fusion network, the perceptual loss and the basic loss are used to optimize and update the parameter configuration of the network. The specific loss function is: T loss =ω1P loss +ω2B loss Among them, P loss is the perceptual loss, which is composed of the mean square error between the predicted fused image feature map extracted by the last convolutional layer of ResNet101 and the label fused image. and They represent the feature map of the predicted fused image and the feature map of the fused image ground truth, i is the channel index of the feature map, C f , H f , W f Respectively represent the number of channels, height, and width of the feature map; B loss The basic loss is and They represent the predicted fused image and the ground truth of the fused image respectively, i is the RGB image channel index, H g , W g are the height and width of the fused image ground truth respectively; T loss is the total loss, ω1 and ω2 are weight coefficients, which are set to 1.

Images