Multi-focus Image Fusion Method Based on Region Difference Prior-guided Deep Neural Network

Abstract

The present invention claims protection for a multi-focus image fusion method of a region-difference prior-guided deep neural network, which relates to technical fields such as digital image processing, computer vision, and deep learning. The specific steps are as follows: 1) Make a publicly available multi-focus image dataset; 2) Perform dataset preprocessing on the publicly available multi-focus images, including techniques such as image denoising, image enhancement, and image registration; 3) Use morphological operations - dilation and erosion to strengthen the differences between paired multi-focus images to obtain region-difference prior information; 4) Design a region-difference prior-guided deep neural network; 5) Use the trained model to fuse the multi-focus images in the test set to obtain the final fusion result. This method utilizes the proposed region-difference prior and combines the model obtained by training with the existing deep neural network to improve the accuracy of its focus measurement and obtain higher-quality fused images.

Description

Technical Field

[0001] The present invention relates to a multi-focus image fusion method for a deep neural network guided by regional difference prior, and belongs to the technical fields of digital image processing, computer vision, deep learning, etc. Background Art

[0002] Due to the limitation of the depth of field of the existing optical camera system, the images obtained when shooting the same scene are usually images with some regions being clear while some regions being blurred or defocused, and it is difficult to generate an ideal all-in-focus image. However, this poses a great challenge to tasks that require accurate analysis of the entire scene, such as robot vision, medical imaging, etc. And many other popular computer vision tasks often also require all-in-focus images for further processing, such as detection and segmentation. For the above problems, the current general measure is the multi-focus image fusion algorithm. With the rapid development of deep learning and fusion technologies, new methods and technologies are constantly being explored and developed in the field of multi-focus image fusion. So far, the research on multi-focus image fusion has lasted for more than 30 years, and numerous algorithms have been published during this period. According to the differences in the fusion strategies adopted during the fusion process, these algorithms can generally be divided into two categories: methods based on reconstruction strategies and methods based on decision-making strategies. The methods based on reconstruction strategies usually consist of three steps: image decomposition, fusion of decomposition coefficients, and reconstruction. Such algorithms regard the defocused images under different focusing scenarios as the degraded images of the ideal all-in-focus image. In this mode, multi-focus image fusion actually becomes an image enhancement task of obtaining an all-in-focus image from the degraded images. Similar to other enhancement tasks, such methods based on reconstruction strategies will inevitably generate pixel defects in the generated fusion images, such as brightness and color distortion. Therefore, the methods based on decision-making strategies start to become popular.

[0003] Different from the methods based on reconstruction strategies, the work based on decision-making strategies mainly focuses on how to generate an accurate decision map. Then, the focused parts are selected from the defocused images through the decision map and combined to form an all-in-focus image. In such methods, the step of focus measure (FM) is used to obtain the decision map, and the focus measure can be completed through artificial design and deep learning. According to how to implement the focus measure, the methods based on decision-making strategies can be further divided into methods based on artificially designed FM and methods based on learnable FM. The methods based on artificially designed FM are more popular, but such methods are difficult to handle complex scenes and limit the applicable space, greatly restricting the performance of multi-focus image fusion.

[0004] With the continuous development of deep learning technology, methods based on learnable FM mainly use convolutional neural networks as the main architecture for research. In such methods, deep neural networks are regarded as tools for implementing FM and can directly learn a mapping from defocused images to decision maps. Methods based on learnable FM can usually automatically learn a relatively accurate decision map, which is difficult to achieve with methods based on manually designed FM. Due to the powerful learning ability of CNN, the performance of the multi-focus image fusion task is further improved compared with methods based on manually designed FM. However, although the current methods based on learnable FM have achieved certain improvement effects to a certain extent, there are still hole phenomena in the generated decision maps. Therefore, how to effectively improve the accuracy of focus measurement and effectively improve the hole phenomenon in the decision map is becoming increasingly important.

[0005] CN113313663A, a multi-focus image fusion method based on zero-shot learning, uses a multi-focus image fusion network structure IM-Net to fuse the information contained in the input multi-focus images. IM-Net includes two joint sub-networks I-Net and M-Net. I-Net models the depth prior of the fused image, and M-Net models the depth prior of the focus map. Zero-shot learning is achieved through the extracted prior information. A reconstruction constraint is imposed on IM-Net to ensure that the information of the source image pair can be better transmitted to the fused image. The high-level semantic information can maintain the brightness consistency of adjacent pixels, and the guiding loss provides guiding information for IM-Net to find clear regions. The experimental results show the effectiveness of the method of the present invention.

[0006] The methods in this patent still achieve their purposes by designing complex network structures. The improvement of the fusion effect is limited while increasing the computational burden. The present invention thinks about how to improve the fusion efficiency from a brand-new perspective, that is, by strengthening the difference between the focused part and the defocused part in the defocused image pair to provide a regional difference prior. With this prior, a simple neural network can achieve a good fusion effect. Further, the use of the regional difference prior in the neural network is simple and effective and has universality. Summary of the Invention

[0007] The present invention aims to solve the problems existing in the existing methods based on learnable FM. A multi-focus image fusion method of a deep neural network guided by regional difference prior is proposed. The technical solution of the present invention is as follows:

[0008] A multi-focus image fusion method of a deep neural network guided by regional difference prior, which includes the following steps:

[0009] (1), Collect original image samples and make a multi-focus image training set for training;

[0010] (2) Perform image preprocessing operations including image denoising, image enhancement, and image registration on the multi-focus images to achieve data augmentation;

[0011] (3) Use morphological operations, namely dilation and erosion, to enhance the differences between paired multi-focus images to obtain regional difference prior information;

[0012] (4) Based on the regional difference prior information obtained in step (3), design a deep neural network guided by regional difference prior and perform model training;

[0013] (5) Use the model trained in step (4) to test the multi-focus images in the test set to obtain the final fusion result.

[0014] Further, the step (2) performs image preprocessing operations including image denoising, image enhancement, and image registration on the multi-focus images to achieve data augmentation, specifically including:

[0015] Image denoising is specifically: Use some artificially designed low-pass filters, such as median filtering and Wiener filtering, to remove image noise;

[0016] Image enhancement is specifically: Directly perform various linear or non-linear operations on the image to enhance the pixel gray values of the image;

[0017] Image registration is specifically: Extract feature points from two images; Find matching feature point pairs through similarity measurement; Then obtain the image spatial coordinate transformation parameters through the matching feature point pairs; Finally, perform image registration by the coordinate transformation parameters.

[0018] Further, the step (3), using morphological operations, namely dilation and erosion, to enhance the differences between paired multi-focus images to obtain regional difference prior information, specifically includes:

[0019] The dilation of a structuring element SE on an image f at position (x, y) is defined as follows:

[0020]

[0021] where SE is the structuring element; f is the grayscale image; is the dilation operation; s, t are the moving steps;

[0022] The erosion of a structuring element SE on an image f at position (x, y) is defined as follows:

[0023]

[0024] is the erosion operation;

[0025] Given a pair of grayscale images I a and I b , perform a global dilation operation on I a and I b to obtain a pair of images D a and D b , which are defined as follows:

[0026]

[0027] where dilate is the global dilation operation. Since the dilation of the grayscale image by SE at any position (x,y) is defined as the maximum value of the overlapping region, thus, the grayscale values of the pixels in images D a and D b will increase. Subtract I from D, and at the same time define the changes in the focused and defocused parts between D and I as:

[0028]

[0029] where D focus , I focus represent the focused parts in D and I;

[0030] Δ focus↑ , Δ defocus↑ represent the change amounts of the focused and defocused parts between D and I respectively;

[0031] D defocus , I defocus represent the defocused parts in D and I; Similarly, by performing a global erosion operation on images I a and I b , a pair of images E a and E b are obtained, which can be described as:

[0032]

[0033] where erode is the global erosion operation. The erosion of the grayscale image by SE at any position (x,y) is defined as the minimum value of the overlapping region; therefore, the grayscale values of the pixels in images E a and E b will decrease; Subtract E from I, and define the changes in the focused and defocused parts between I and E, which can be described as:

[0034]

[0035] Considering the purpose of regional difference enhancement, that is, Δ focus↑ Δ defocus↑ and Δ focus↓ Δdefocus↓ All can be effectively established. The best way to obtain the regional difference prior is as follows:

[0036] dilate(I a ), erode(I a );dilate(I b ), erode(I b )

[0037] where I a , dilate(I a ) represents concatenating I a and dilate(I a ) together in the channel dimension.

[0038] Furthermore, in step (4), design a deep neural network guided by the regional difference prior and perform model training. The training process is as follows:

[0039]

[0040] where p is the generated focus map, is the reference image,

[0041] i represents the i-th batch;

[0042] j represents the j-th channel;

[0043] represents the reference image of the j-th channel in the i-th batch;

[0044] represents the focus map of the j-th channel in the i-th batch;

[0045] N represents the batch size; C represents the number of channels. By training, a two-channel focus map p is used as the output of the neural network, where each output value is the focus score of the corresponding pixel in the paired source images; subsequently, by comparing the focus scores of the two channels in p, an initial decision map is generated; a fully connected conditional random field CRF is used to refine the initial decision map, and then the final decision map W is obtained.

[0046] Furthermore, by comparing the focus scores of the two channels in p, an initial decision map T is generated; a fully connected conditional random field CRF is used to refine the initial decision map, and then the final decision map W is obtained, which specifically includes:

[0047] Obtain the initial decision map;

[0048] For each pixel i with class label x i and the corresponding observation value y i, such that each pixel point serves as a node, and the relationship between pixels serves as an edge, thus forming a conditional random field. By observing the variable y i to infer the class label x corresponding to pixel i i ; Through the above technology, the initial decision graph can be improved.

[0049] Furthermore, in step (5), the final image fusion is performed, which is expressed as follows:

[0050] F fusion (x, y) = A(x, y)W(x, y) + B(x, y)(1 - W(x, y))

[0051] where A and B are source images, W is the final decision graph, and it is pixel dot multiplication. F fusion is the final fusion result.

[0052] The advantages and beneficial effects of the present invention are as follows:

[0053] A multi-focus image fusion method based on a region difference prior-guided deep neural network proposed by the present invention is an effective method that can effectively improve the focusing measurement accuracy based on region difference prior information. This method improves the problems existing in the previous learnable FM method, can effectively perform fusion in the face of multiple fusion scenarios to obtain high-quality fusion images, and the region difference prior can be applied to other neural network models, having generality.

[0054] The present invention utilizes technologies such as digital image processing, computer vision, and deep learning to achieve the multi-focus image fusion task. The present invention is a method based on region difference prior information. It uses morphological operations of dilation and erosion to strengthen the differences between paired defocused images and obtain the region difference prior, and then designs a new type of deep neural network to generate a better decision graph to obtain the final fusion result. The present invention has the following advantages:

[0055] (1) Using the pytorch platform for training and testing, with high efficiency;

[0056] (2) It is a method for improving the focusing measurement accuracy based on region difference prior. Only simple morphological operations can be used to achieve the purpose of strengthening region differences and obtain the region difference prior. The use of region difference prior information is not only simple but also efficient;

[0057] (3) The region difference prior obtained by the present invention can effectively improve the quality of fusion images obtained under various fusion scenarios;

[0058] (4) The present invention also has a good improvement effect on other neural network methods;

[0059] (5) High performance improvement, for any different fusion scenarios, the quality of the fused image is very high; for the difficult-to-decide regions in multi-focus images, the regional difference prior can bring a certain degree of improvement in the accuracy of focus measurement;

[0060] (6) It can assist related image detection and image segmentation work, which has practical significance and achieves good results.

[0061] The innovation of the present invention is mainly the steps of claims 3 and 4.

[0062] Considering enhancing the difference between the focused part and the defocused part in the defocused image to obtain the regional difference prior information is a major innovation, and no other method has ever viewed multi-focus image fusion from this perspective.

[0063] Ingenuity: The reason for obtaining the regional difference prior is that the focused part and the defocused part have completely different sensitivities to morphological operations - dilation and erosion, that is, the focused part responds more strongly to dilation and erosion operations. Therefore, after the operations are applied, the focused part and the defocused part will produce completely different changes, so as to achieve the purpose of difference enhancement, which fits our goal. At the same time, the regional difference prior proposed by the present invention is universal and can be used in other multi-focus image fusion methods to improve their focus measurement accuracy. For any different fusion scenarios, the quality of the fused image obtained by using the regional difference prior is very high, and the use of the regional difference prior is simple and efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 is the system flowchart of the preferred embodiment provided by the present invention;

[0065] Figure 2 (a1)-(a6) are the original images and the images after the regional difference enhancement operation;

[0066] Figure 2 (b1)-(b4) are the decision diagrams obtained without the participation of the regional difference prior and the decision diagrams obtained with the participation of the regional difference prior;

[0067] Figure 2 (b5)-(b6) are examples of two final fused images. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0068] Next, the technical solutions in the embodiments of the present invention will be clearly and detailedly described in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only a part of the embodiments of the present invention.

[0069] The technical solution for the present invention to solve the above technical problems is:

[0070] AsFigure 1 As shown in Figure 1 , a multi-focus image fusion method for a deep neural network guided by regional difference prior includes the following steps:

[0071] Step 1: Collect original image samples and produce a multi-focus image training set for training;

[0072] Step 2: Use common image preprocessing operations on the produced multi-focus image training set to achieve data augmentation;

[0073] Step 3: Utilize morphological operations - dilation and erosion to strengthen the differences between paired multi-focus images to obtain regional difference prior information. This step includes performing dilation operations and erosion operations on the multi-focus images and effectively combining them to achieve the purpose of regional difference strengthening. The specific steps are as follows:

[0074] Since the focused part and the defocused part have different sensitivities to dilation operations and erosion operations, that is, the focused part is more sensitive to morphological operations. Therefore, when performing morphological operations on these two parts, obvious different changes will occur in the focused part and the defocused part. The dilation and erosion operations on an image depend on the structuring element (SE). The dilation of a structuring element SE on an image f at position (x, y) is defined as follows:

[0075]

[0076] where SE is the structuring element; f is the grayscale image; is the dilation operation.

[0077] The erosion of a structuring element SE on an image f at position (x, y) is defined as follows:

[0078]

[0079] where SE is the structuring element; f is the grayscale image; is the erosion operation.

[0080] Given a pair of grayscale images I a and I b , perform a global dilation operation on I a and I b to obtain a pair of images D a and D b , which is defined as follows:

[0081]

[0082] where dilate is the global dilation operation. Since the dilation of the grayscale image by SE at any position (x, y) is defined as the maximum value of the overlapping region. Thus, the images D a and Db The gray value of the pixels will increase. Subtract I from D, and at the same time define the change in the focused and defocused parts between D and I as:

[0083]

[0084] where D focus , I focus represent the focused parts in D and I; D defocus , I defocus represent the defocused parts in D and I. Similarly, by performing a global erosion operation on images I a and I b , a pair of images E a and E b are obtained. It can be described as:

[0085]

[0086] where erode is the global erosion operation. The erosion of the SE on the grayscale image at any position (x, y) is defined as the minimum value of the overlapping area. Therefore, the gray values of the pixels in images E a and E b will decrease. Subtract E from I, and define the change in the focused and defocused parts between I and E, which can be described as:

[0087]

[0088] It can be noted that existing learnable FM-based methods are all designed with complex network structures to improve the accuracy of focus measurement, which can achieve a certain improvement effect to a certain extent but there are still hole phenomena, and the computational burden is additionally increased. Therefore, the present invention proposes a regional difference prior as the guiding prior of the deep neural network, considering the purpose of strengthening regional differences, that is, Δ focus↑ Δ defocus↑ and Δ focus↓ Δ defocus↓ are both effectively established. The best way to obtain the regional difference prior is given as follows:

[0089] dilate(I a ), erode(I a ); dilate(I b ), erode(I b )

[0090] where, I a , dilate(I a ) represent dilating I a and dilate(I a) are connected together. The purpose of the above formula is to maximize the difference between the focused and defocused parts, significantly improving the accuracy of focus measurement.

[0091] Step 4: Design a region-difference-prior-guided deep neural network for the region-difference prior obtained in step (3) and perform model training. The training process is as follows:

[0092]

[0093] where p is the generated focus map, is the reference image, N represents the batch size; C represents the number of channels. Through effective training, a two-channel focus map p is used as the output of the neural network, where each output value is the focus score of the corresponding pixel in the paired source images. Subsequently, by comparing the focus scores of the two channels in p, an initial decision map is generated. Considering that there may be some imperfections in the initial decision map, a fully connected conditional random field (CRF) is used to improve the initial decision map, and then the final decision map W is obtained.

[0094] Step 5: Use the model trained in step (4) to perform fusion testing on the input multi-focus images to obtain the final fusion result.

[0095] Experimental method:

[0096] During this experiment, we made a public multi-focus image dataset, and used 90% of it as the training set to train the neural network, and used the other 10% as the validation set to verify the fusion quality of the proposed region-difference-prior-guided deep neural network multi-focus image fusion method.

[0097] Step 1: Use image preprocessing operations to perform data augmentation on the made multi-focus image training set, including image denoising, image enhancement, and image registration.

[0098] Step 2: Run the python program, input the training set pictures and their corresponding labels into the deep neural network, and after tuning the training parameters, obtain the finally trained model.

[0099] Step 3: Use the trained model to test the images in the test set and calculate the quality metrics of the fused images.

[0100] Experiments prove that the method proposed in the present invention can effectively improve the accuracy of focus measurement after training, and for multi-focus images in various fusion scenarios, the quality of the fused images is effectively improved.

[0101] The systems, devices, modules, or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions.

[0102] It should also be noted that the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, commodity or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, commodity or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, commodity or device comprising said element.

[0103] The above embodiments should be understood as being only for the purpose of illustrating the present invention and not for limiting the scope of protection of the present invention. After reading the content described in the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A multi-focus image fusion method for a region-difference prior-guided deep neural network, characterized in that, it includes the following steps: (1). Collect original image samples and produce a multi-focus image training set for training; (2). Perform image preprocessing operations including image denoising, image enhancement, and image registration on the multi-focus images to achieve data augmentation; (3). Use morphological operations, namely dilation and erosion, to strengthen the differences between paired multi-focus images to obtain region-difference prior information; Given a pair of grayscale images I a and I b , perform a global dilation operation on I a and I b to obtain a pair of images D a and D b , which are defined as follows: Among them, dilate is a global dilation operation. Since the dilation of the grayscale image by SE at any position (x, y) is defined as the maximum value of the overlapping region, thus, the grayscale values of the pixels in image D a and D b will increase. Subtract I from D, and at the same time, define the change in the focused and defocused parts between D and I as: Among them, D focus and I focus represent the focused parts in D and I; Δ focus↑ and Δ defocus↑ respectively represent the change amounts of the focusing and defocusing parts between D and I; D defocus 、I defocus represent the defocused portions in D and I; Similarly, by performing a global erosion operation on images I a and I b a pair of images E a and E b is obtained, which can be described as: Where erode is a global erosion operation, the erosion of the SE on the grayscale image at any position (x, y) is defined as the minimum value of the overlapping region; thus, the gray values of the pixels in image E a and E b will decrease; subtracting E from I and defining the change in the focused and defocused parts between I and E, it can be described as: For the purpose of enhancing the consideration of regional differences, i.e., Δ focus↑ >> Δ defocus↑ and Δ focus↓ >> Δ defocus↓ are all effectively valid. The best way to obtain the regional difference prior is as follows: dilate(I a ),erode(I a );dilate(I b ),erode(I b ) wherein, erode(I a ), dilate(I a ) represents concatenating erode(I a ) and dilate(I a ) along the channel dimension; (4). Based on the region-difference prior information obtained in step (3), design a region-difference prior-guided deep neural network and perform model training; (5). Use the model trained in step (4) to test the multi-focus images in the test set to obtain the final fusion result.

2. The multi-focus image fusion method for a region-difference prior-guided deep neural network according to claim 1, characterized in that, the step (2) performs image preprocessing operations including image denoising, image enhancement, and image registration on the multi-focus images to achieve data augmentation, specifically including: Image denoising is specifically: Use some artificially designed low-pass filters, such as median filtering and Wiener filtering, to remove image noise; Image enhancement is specifically: Directly perform various linear or non-linear operations on the image to enhance the pixel gray values of the image; Image registration is specifically: Extract features from two images to obtain feature points; Find matching feature point pairs by performing similarity measurement; Then obtain the image spatial coordinate transformation parameters through the matching feature point pairs; Finally, perform image registration by the coordinate transformation parameters.

3. The multi-focus image fusion method for a region-difference prior-guided deep neural network according to claim 1, characterized in that, the step (3). Use morphological operations, namely dilation and erosion, to strengthen the differences between paired multi-focus images to obtain region-difference prior information, specifically including: The dilation of a structuring element SE on an image f at position (x, y) is defined as follows: where SE is the structuring element; f is the grayscale image; is the dilation operation; s and t are the moving steps; The erosion of a structuring element SE on an image f at position (x, y) is defined as follows: It is an etching operation.

4. The multi-focus image fusion method for a region-difference prior-guided deep neural network according to claim 1, characterized in that, in the step (4), design a region-difference prior-guided deep neural network and perform model training, and its training process is as follows: where p is the generated focused image, is the reference image, i represents the i-th batch; j represents the j-th channel; denote the reference image of the j-th channel in the i-th batch; Denote the focusing map of the j-th channel in the i-th batch; N represents the batch size; C represents the number of channels. By training, a two-channel focus map p is used as the output of the neural network, where each output value is the focus score of the corresponding pixel in the paired source images; Subsequently, by comparing the focus scores of the two channels in p, an initial decision map is generated; Use a fully connected conditional random field CRF to refine the initial decision map, and then obtain the final decision map W.

5. The multi-focus image fusion method for a region-difference prior-guided deep neural network according to claim 4, characterized in that, by comparing the focus scores of the two channels in p, an initial decision map T is generated; Use a fully connected conditional random field (CRF) to refine the initial decision graph, and then obtain the final decision graph W, which specifically includes: Obtain the initial decision diagram; For each pixel i, there is a class label x i and a corresponding observation value y i , such that each pixel forms a node and the relationship between pixels forms an edge, thus constituting a conditional random field. By means of the observed variable y i to infer the class label x corresponding to pixel i i ; Through the above techniques, the initial decision diagram can be improved.

6. The multi-focus image fusion method of a deep neural network guided by regional difference prior according to claim 5, characterized in that in step (5), the final image fusion is performed, and its expression is as follows: F fusion (x,y) = A(x,y) ⊙ W(x,y) + B(x,y) ⊙ (1 - W(x,y)) Where A and B are source images, W is the final decision map, ⊙ is pixel multiplication, and F fusion is the final fusion result.

Images