A multi-focus image fusion method based on multi-scale context perception

Abstract

The application discloses a kind of multi-focus image fusion methods based on multi-scale context perception, comprising:1 data preparation and pretreatment, construct multi-scale context perception network, including: encoder, coarse positioning decoder, receptive field enhancement module and multi-scale feature interaction module, wherein, encoder is visual converter, for multi-scale feature extraction, decoder includes: convolution and activation function, for multi-scale feature coarse decoding, receptive field enhancement module includes convolution and activation function, for feature receptive field enhancement, multi-scale feature interaction module includes: convolution and activation function, for multi-level feature fusion;2 on the input multi-focus image is fused, including: network training and multi-focus image fusion.The application can make full use of the complementary and redundant information in different defocus images to fuse out better quality all-focus image, provide better quality image for human eye observation, while providing support for image recognition, segmentation and other computer vision tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-focus image fusion, and particularly relates to a multi-focus image fusion method based on multi-scale context perception. BACKGROUND

[0002] Due to the limited imaging function of a camera, it is usually difficult to take an image in which all objects are in focus. Specifically, the camera is limited by the depth of field, and only the image inside the depth of field is kept in focus, while the image outside the depth of field becomes blurred. Therefore, in the current production and life, the information contained in the collected images is different and incomplete, and these images contain different contour, texture and other feature information, which is inconvenient for image feature analysis, but the comprehensive information can provide a more comprehensive and clear understanding of the target in the image scene. Therefore, the purpose of multi-focus image fusion is to extract and integrate important information into an image. In order to solve this problem, multiple images with different focus regions can be selected for image fusion to obtain a full-focus image, thereby making up for the limitation of the depth of field. We call this technology multi-focus image fusion.

[0003] As one of the important branches of image fusion, the main purpose of multi-focus image fusion is to fuse multiple images with different focus regions in the same scene to obtain a full-focus image in which all regions are clear. The existing methods have the following problems: one is that the traditional method is mainly designed by human hand to extract features and rules, which has strong constraints and poor robustness, and the effect in different scenes is poor, which may cause focus region misjudgment and artifacts, thereby affecting the final fusion image; the other is that the deep learning-based method is more dependent on the design of network structure, and the previous method is mainly based on convolution, which only focuses on the use of local feature information, ignores global information, and lacks multi-scale feature fusion and interaction. SUMMARY

[0004] In order to overcome the problems of the existing image fusion in multi-focus image fusion, the present application provides a multi-focus image fusion method based on multi-scale context perception, so as to fully utilize the complementary and redundant information of images with different focus regions to provide better image feature expression, and to reconstruct a full-focus image with higher quality, thereby providing a better image for human eye observation, and providing support for computer vision tasks such as image recognition, classification and segmentation, thereby assisting human eye recognition and computer analysis and processing research.

[0005] The present application adopts the following technical solutions to solve the above problems:

[0006] The multi-focus image fusion method based on multi-scale context perception has the following characteristics:

[0007] Step 1: Obtain P pairs of RGB multi-focus images and convert them into gray images, denoted as and as a training set, wherein, and represent the foreground focus image and the background focus image in the pth pair of gray images respectively; the pth real ground mask corresponding to the pth pair of gray images is taken as the pth label, denoted as G p , thereby constructing a label set {G 1 , G 2 , …, G p , …, G P} of P pairs of RGB multi-focus images;

[0008] Step 2: Construct a multi-scale context perception network, including an encoder, a coarse positioning decoder, a receptive field enhancement module and a multi-scale feature interaction module;

[0009] Step 2.1: the encoder includes a first convolutional block Conv 3×3 and Y visual transformers, wherein Conv 3×3 represents a convolutional layer with a convolution kernel of 3x3 and a ReLU activation function;

[0010] After the pth pair of gray images and are spliced in the channel dimension, they are input into the multi-scale context perception network and processed by the first convolutional block Conv 3×3 of the encoder to obtain the pth input feature I p , which is then processed by the Y visual transformers to obtain Y primary feature maps corresponding to the pth pair of gray images , wherein, represents the yth primary feature map;

[0011] Step 2.2: the coarse positioning decoder is composed of a plurality of multi-level cross-scale connected second convolutional blocks Conv 3×3 and a first convolutional block Conv 1×1 , and performs M R stage feature decoding on the Y primary feature maps to obtain the pth coarse positioning decoder feature and the pth initial decision map , wherein Conv 1×1 represents a convolutional layer with a convolution kernel of 1x1;

[0012] Step 2.3: the receptive field enhancement module is composed of 4 receptive field enhancement branches with the same structure but different parameters k and r, and 5 second convolution blocks Conv 1×1 , 1 ReLU activation function, wherein each receptive field enhancement branch is formed by stacking 1 asymmetric convolution block Conv 1×k , 1 asymmetric convolution block Conv k×1 and 1 first stride convolution block Conv k×k,r in sequence; wherein Conv 1×k represents an asymmetric convolution layer with a convolution kernel of 1xk, Conv k×1 represents an asymmetric convolution layer with a convolution kernel of kx1, and Conv k×k,r represents a symmetric convolution layer with a convolution kernel of kxk and a stride of r.

[0013] Y-1 primary feature maps are input into the receptive field enhancement module in parallel, wherein after adjusting the channels by 5 second convolution blocks Conv 1×1 , 5 output features are obtained. The last 4 output feature maps are input into 4 receptive field enhancement branches respectively for processing, and 4 receptive field enhancement branch feature maps are obtained. After channel dimension splicing with , the y-th fused feature map is obtained. After adjusting the channels by the second convolution block Conv 1×1 again, the adjusted y-th feature map is obtained. After adding and processing by the activation function ReLU, the y-th receptive field enhanced feature map is obtained as the final output. Thus, Y-1 receptive field enhanced feature maps are obtained.

[0014] Step 2.4: the multi-scale feature interaction module is composed of a preprocessing module, a multi-scale feature pyramid module, and a third convolution block Conv 3×3 , and after processing the p-th coarse positioning decoder feature and the p-th initial stage decision map in sequence, 1-Y-1 receptive field enhanced feature maps corresponding to the p-th series of multi-scale interaction feature maps and the p-th series of decision maps are obtained, wherein represents the y-th receptive field enhanced feature map, represents the kth stage feature map in the pth series of multi-scale interaction feature maps, represents the kth stage decision map in the pth series of decision maps;

[0015] Step 2.5: the pth down-sampling decision map After up-sampling processing, the first fusion decision map is obtained respectively with the pth series of decision maps After step-by-step addition, the pth series of fusion decision maps is obtained wherein, represents the kth fusion decision map of the pth pair of grayscale images and .

[0016] Step 2.6: After up-sampling operation and Sigmoid activation function processing, the pth multi-level output decision map is obtained wherein, represents the kth output decision map of the pth pair of grayscale images and , and is taken as the final decision map of the pth pair of grayscale images and .

[0017] Step 3: constructing a loss function using formula (1)

[0018]

[0019] In formula (1), L wBCE represents a weighted binary cross-entropy loss, and L wIOU represents a weighted intersection over union loss.

[0020] The total loss function L total of the multi-scale context perception network is constructed using formula (2):

[0021]

[0022] Step 4: based on the training set, the multi-scale context perception network is trained using a back propagation algorithm, and the total loss function L total is calculated to adjust the network parameters until the maximum iteration number is reached, thereby obtaining a trained multi-scale context perception network.

[0023] Step 5: using the final decision map , the inverse decision map is obtained after inversion respectively taking the final decision map and the inverse decision map with After pixel-by-pixel multiplication, a partially clear image is obtained with The partially clear image is multiplied by with Pixel-by-pixel addition is performed to obtain the p-th pair of gray-scale images and The predicted all-focus image of

[0024] The multi-focus image fusion method based on multi-scale context perception according to the application also has the following characteristics: the step 2.2 comprises:

[0025] Step 2.2.1, when r = 1, the coarse positioning decoder is at the M r th stage, and the Y-th primary feature map is subjected to two times of up-sampling operations with different weights respectively, so that the sizes of and are the same, and the r-th up-sampled feature map and the r+1-th up-sampled feature map are obtained, which are respectively input into the r-th and r+1-th second convolution blocks Conv 3×3 for processing, and the r-th and r+1-th feature maps and are obtained. is multiplied by , and then is spliced with in the channel dimension, and after sequentially passing through the r+2-th and r+3-th second convolution blocks Conv 3×3 for processing, the M r th stage output feature is obtained.

[0026] When r = 2, the coarse positioning decoder is at the M r th stage, and the Y-th primary feature map and the Y-1-th primary feature map are subjected to up-sampling operations respectively, so that the sizes of and are the same as the size of , and the r+1-th up-sampled feature map and the r+2-th up-sampled feature map are obtained, which are input into the r+3-th and r+4-th second convolution blocks Conv 3×3 for processing, and the r+1-th and r+2-th feature maps and are obtained. is multiplied by , and then is spliced with In the channel dimension, splicing is performed, and the r+5th, r+6th second convolutional block Conv 3×3 is sequentially passed through to obtain the M r th stage output feature

[0027] When r = 3, 4, …, R-1, the coarse positioning decoder is in the M r th stage, and the same processing is performed on the Yth primary feature map to the Y-r+1th primary feature map to obtain the rth to (R-1)th stage output feature

[0028] When r = R, the output feature of the (R-1)th stage is input into the last 2 second convolutional blocks Conv 3×3 for processing to obtain the output feature map of the Rth stage , which is the pth coarse positioning decoder feature map finally output by the coarse positioning decoder, and after being input into the first convolutional block Conv 1×1 , the pth initial stage decision map is obtained

[0029] The multi-scale feature pyramid module in step 2.4 is composed of 4 multi-scale feature extraction branches with the same structure but different parameters k and r, 5 third convolutional blocks Conv 1×1 , and 1 ReLU activation function, wherein each multi-scale feature extraction branch is formed by stacking a symmetric convolutional block Conv k×k and a second stride convolutional block Conv k×k,r in sequence, wherein Conv k×k represents a symmetric convolutional layer with a convolution kernel of k x k and a ReLU activation function.

[0030] Step 2.4.0, define the current stage as k, and initialize k = 1; the pth initial stage decision map is taken as the (k-1)th stage pth decision map

[0031] Step 2.4.1: the preprocessing module performs downsampling operation on the (k-1)th stage pth decision map so that the sizes of and are the same, and the (k-1)th stage pth downsampled decision map is obtained, and then a Sigmoid activation function operation is performed to obtain the (k-1)th stage pth weight map Meanwhile, the pth coarse positioning decoder feature After the same downsampling operation, the pth feature map is obtained

[0032] Step 2.4.2: The preprocessing module subtracts the pth weight map by “1” After that, the k-1th reverse weight map of the pth feature map is obtained Then the is multiplied by the pth feature map respectively, and the k-1th forward feature map of the pth feature map is obtained and the k-1th reverse feature map of the pth feature map

[0033] Step 2.4.3: The multi-scale feature pyramid module processes the and the respectively, and the output feature map is obtained and the is respectively connected with the pth feature map After the short connection, the pth forward short feature map is obtained and the pth reverse short feature map

[0034] Step 2.4.3.1: After adjusting the channels by the 4 third convolution blocks Conv 1×1 , the k-1th stage 4 channel output feature maps are obtained

[0035] Step 2.4.3.2: The k-1th stage 4 channel output feature maps are respectively input into 4 multi-scale feature extraction branches, and the k-1th stage 4 symmetric convolution feature maps are obtained by the symmetric convolution blocks Conv k×k After processing by the second stride convolution block Conv k×k,r , the k-1th stage 4 stride convolution feature maps are obtained

[0036] Step 2.4.3.3: The k-1th stage 4 channel output feature maps the k-1th stage 4 symmetric convolution feature maps and the k-1th stage 4 stride convolution feature maps are respectively spliced along the output order of the 4 multi-scale feature extraction branches in the channel dimension, and the k-1th stage 4 multi-scale fusion feature maps are obtained

[0037] Step 2.4.3.4: The k-1th stage 4 multi-scale fusion feature maps ​​Respectively, the first k-1 stage before a plurality of scale fusion features are superimposed, and the corresponding k-1 stage 4 superimposed feature maps are obtained

[0038] Step 2.4.3.5: After the k-1 stage 4 superimposed feature maps are again spliced in the channel dimension, the k-1 stage 1 spliced feature map is obtained Input the fifth third convolution block Conv 1×1 After adjusting the channel, the k-1 stage adjusted feature map is obtained

[0039] Step 2.4.3.6: the k-1 stage 1 spliced feature map is input into the fourth third convolution block Conv After adding and processing through the ReLU activation function, the pth multi-scale feature pyramid forward feature map is obtained

[0040] Step 2.4.3.7: the feature map is processed in the same way according to the process of step 2.4.3.1-step 2.4.3.6, and the pth multi-scale feature pyramid reverse feature map is obtained

[0041] Step 2.4.4: the pth forward short feature map feature And the pth reverse short feature map After multiplying two self-learning parameters "a" and "b" respectively, the pth self-learning feature map pair is obtained And

[0042] Step 2.4.5: the pth feature map After the third convolution block Conv 3×3 And up-sampling operation, the pth up-sampling feature map is obtained

[0043] Step 2.4.6: the pth self-learning feature map Subtract the pth up-sampling feature , the pth fusion feature F p is obtained

[0044] Step 2.4.7: the pth fusion feature F p , Add the pth self-learning feature map , the k stage pth multi-scale feature interaction feature map is obtained

[0045] ​​​Step 2.4.8: judge whether k=K is true, if true, it means that 1-Y-1 receptive field enhanced feature maps are obtained Corresponding K-stage p-series multi-scale interactive feature map And K-stage p-series decision map Otherwise, step 2.4.9 is executed.

[0046] Step 2.4.9: the kth stage After 3 times of the third convolution block Conv 3×3 , the kth stage pth decision map is obtained

[0047] Step 2.4.10: the kth stage And the kth stage pth decision map D k p Respectively, the up-sampling operation is performed, so that And The size is the same, and the kth stage pth up-sampling decision map And the kth stage pth up-sampling feature map Are obtained, and are input into the multi-scale feature interaction module together with the receptive field enhanced feature map , k is assigned to k+1, and then step 2.4.1 is returned for sequential execution.

[0048] The electronic device of the present application comprises a memory and a processor, characterized in that the memory is used to store a program supporting the processor to execute the multi-focus image fusion method, and the processor is configured to execute the program stored in the memory.

[0049] The computer readable storage medium of the present application stores a computer program, characterized in that the computer program is executed by the processor to execute the steps of the multi-focus image fusion method.

[0050] Compared with the prior art, the beneficial effects of the present application are as follows:

[0051] 1. The present application proposes a unified network framework to realize the multi-focus image fusion task, which fully utilizes the redundancy and complementary information between different modal images, and fuses high-quality images. Compared with the existing method, the present application method learns the multi-scale context perception features combining local and global information, simulates the mechanism of human visual system, better promotes the learning of network, and thus obtains higher quality image fusion results.

[0052] 2、The application designs a multi-scale feature combined with CNN and transformer, which effectively extracts local and global features; by introducing the transformer model, the long-distance dependency relationship is established while the multi-scale information in the image is extracted, and the receptive field enhancement module is used to enhance the shallow feature information; it can better make each different scale feature have strong semantic information; the bottom layer details and high layer semantic information are integrated, so that the fusion result brings better detail performance.

[0053] 3、The application designs a coarse positioning decoder, which aggregates multi-scale features extracted from the transformer backbone network step by step by using the coarse positioning decoder, and the aggregated features contain context information, and generates a rough feature and a decision map to provide guidance for the subsequent steps. In addition, the application designs a multi-scale feature interaction module to simultaneously focus on the information of the focus area and the defocus area, promote the interaction between the two types of information, enrich the local details, and suppress the wrong classification area at the same time. The multi-scale feature interaction module is used to realize the fusion and interaction of multi-scale information between the shallow feature and the decoder feature, optimize the rough feature and the decision map obtained through the decoder, guide the network to better learn the detail feature, realize more accurate defocus area detection, and strengthen the image quality of the fusion. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 A multi-scale context-aware multi-focus image fusion method flow chart of the application;

[0055] Figure 2 A network structure schematic diagram of the application;

[0056] Figure 3 A fusion structure schematic diagram of the application;

[0057] Figure 4 A coarse positioning decoder structure schematic diagram of the application;

[0058] Figure 5 A receptive field enhancement module structure schematic diagram of the application;

[0059] Figure 6a A multi-scale feature interaction module structure schematic diagram of the application;

[0060] Figure 6b A multi-scale feature pyramid module structure schematic diagram of the application. DETAILED DESCRIPTION

[0061] In this embodiment, a multi-scale context-aware multi-focus image fusion method, as shown in Figure 1 , includes the following steps:

[0062] Step 1: Obtain P pairs of RGB multi-focus images and convert them into gray images, denoted as and as a training set, wherein, and respectively represent the foreground focus image and the background focus image in the pth pair of gray images; the pth real ground mask corresponding to the pth pair of gray images is taken as the pth label, denoted as G p , thereby constructing a label set {G 1 , G 2 , …, G p , …, G P} of P pairs of RGB multi-focus images.

[0063] Step 2: Construct a multi-scale context perception network as shown in Figure 2 , including an encoder, a coarse positioning decoder, a receptive field enhancement module and a multi-scale feature interaction module. Figure 2 , + in the circular symbol shown in represent Sigmoid activation function processing and pixel-level addition processing, respectively.

[0064] Step 2.1: The encoder includes a first convolutional block Conv 3×3 and Y visual converters, wherein Conv 3×3 represents a convolutional layer with a 3x3 convolutional kernel and a ReLU activation function.

[0065] After the pth pair of gray images and are spliced in the channel dimension, they are input into the multi-scale context perception network and processed by the first convolutional block Conv 3×3 of the encoder to obtain the pth input feature I p , which is then processed by the Y visual converters in turn to obtain Y primary feature maps corresponding to the pth pair of gray images , wherein, represents the yth primary feature map.

[0066] In this embodiment, as shown in the structure diagram Figure 2 , Y=4; the pth pair of gray images and are a pair of 512x512x1 images, which are input into the multi-scale context perception network after being spliced in the channel dimension, and the channel number is adjusted by the first convolutional block Conv 3×3 to obtain the pth input feature map I p, size is 512x512x3, and after being processed by four visual converters in turn, the primary deep feature maps are 128x128x64, 64x64x128, 32x32x320, and 16x16x512, respectively, denoted as where the visual converter is a standard visual converter;

[0067] Step 2.2: The coarse positioning decoder is composed of several multi-level cross-scale connected second convolution blocks Conv 3×3 and one first convolution block Conv 1×1 , and performs M stage feature decoding on Y primary feature maps R to obtain the pth coarse positioning decoder feature and the pth initial decision map , where Conv 1×1 represents a convolution layer with a 1x1 convolution kernel;

[0068] In this embodiment, as shown in the structure diagram Figure 4 , M R = 4, and the circular symbols shown in the figure represent pixel-level multiplication processing and channel dimension splicing processing, respectively.

[0069] Step 2.2.1, when r = 1, the coarse positioning decoder is at the M r th stage, and two up-sampling operations with different weights are performed on the Yth primary feature map , so that the sizes of and are the same, and the rth up-sampling feature map and the r+1th up-sampling feature map are obtained, which are then input into the rth and r+1th second convolution blocks Conv 3×3 for processing, and the rth and r+1th feature maps and are obtained. is multiplied by , and then spliced with in the channel dimension, and after being processed by the r+2th and r+3th second convolution blocks Conv 3×3 in turn, the M r th stage output feature is obtained.

[0070] When r = 2, the coarse positioning decoder is at the M r th stage, and up-sampling operations are performed on the Yth primary feature map and the Y-1th primary feature map , so that the sizes of and Size and The sizes are the same, and the (r+1)th upsampled feature map is obtained. and the (r+2)th upsampled feature map Then, input the (r+3)th and (r+4)th second convolutional blocks Conv 3×3 The process is performed to obtain the (r+1)th and (r+2)th feature maps. and Will and Multiply, then add The concatenation is performed along the channel dimension, and then sequentially passed through the (r+5)th and (r+6)th second convolutional blocks. 3×3 After processing, we obtain the Mth... r Output characteristics of each stage

[0071] When r = 3, 4, ..., R-1, the coarse localization decoder is at the Mth position. r Each stage, and the Y-th primary feature map up to the (Y-r+1)th primary feature map After performing the same processing, the output features of stages r to R-1 are obtained.

[0072] When r = R, the output features of the (R-1)th stage will be... Input the last two second convolutional blocks Conv 3×3 The process is performed to obtain the output feature map of the Rth stage. This is the p-th coarse localization decoder feature map output by the coarse localization decoder, which is then input into the first convolutional block Conv. 1×1 Then, the p-th initial stage decision graph is obtained.

[0073] like Figure 2 The structure diagram shown represents the primary deep feature map. The input is fed into the coarse localization decoder to obtain the coarse localization feature map. and initial decision graph The sizes are 128×128×64 and 128×128×1 respectively;

[0074] Step 2.3: The receptive field enhancement module consists of 4 receptive field enhancement branches with the same structure but different parameters k and r, and 5 second convolutional blocks (Conv). 1×1 It consists of one ReLU activation function, where each receptive field enhancement branch is composed of one asymmetric convolutional block Conv. 1×k 1 asymmetric convolutional block Conv k×1 and 1 first stride convolutional block Convk×k,r Formed by stacking sequentially; wherein, Conv 1×k This represents an asymmetric convolutional layer with a kernel of 1×k. k×1 This represents an asymmetric convolutional layer with a kernel size of k×1. k×k,r This represents a symmetric convolutional layer with a kernel size of k×k and a stride of r.

[0075] In this embodiment, as Figure 5 The diagram shows the structure, where the parameters k and r in the four receptive field enhancement branches are {3, 5, 7, 9} and {3, 5, 7, 9}, respectively. The + and C symbols in the diagram represent pixel-level additive processing and channel-level splicing processing, respectively.

[0076] Y-1 primary feature maps The parallel inputs are fed into the receptive field enhancement module, where... Through 5 second convolutional blocks Conv 1×1 After adjusting the channels, five output features are obtained. The last four output feature maps The inputs are processed into four receptive field enhancement branches to obtain four receptive field enhancement branch feature maps. Again with After concatenating along the channel dimension, the y-th fused feature map is obtained. And input the second convolutional block Conv again 1×1 After adjusting the channels, the adjusted y-th feature map is obtained. Will and After summing and processing with the ReLU activation function, we obtain the final output feature map of the y-th receptive field enhancement. This yields Y-1 feature maps with enhanced receptive fields.

[0077] like Figure 2 The structure diagram shown represents the primary shallow feature map. The parallel inputs are fed into the receptive field enhancement module, and the resulting enhanced feature maps are denoted as follows: The sizes are 128×128×64, 64×64×128, and 32×32×320 respectively;

[0078] Step 2.4: The multi-scale feature interaction module consists of a preprocessing module, a multi-scale feature pyramid module, and a third convolutional block (Conv). 3×3 The system consists of four multi-scale feature pyramid modules with identical structures but different parameters k and r, and five third convolutional blocks (Conv). 1×1, 1 ReLU activation function, wherein each multi-scale feature extraction branch is composed of 1 symmetric convolution block Conv k×k and 1 second stride convolution block Conv k×k,r are stacked in sequence, wherein Conv k×k represents a symmetric convolution layer with a convolution kernel of k x k and a ReLU activation function;

[0079] In this embodiment, as shown in the structure diagram of Figure 6a and Figure 6b , wherein Figure 6a the circular symbols shown in +, -, x, respectively represent pixel-level addition processing, pixel-level subtraction processing and pixel-level multiplication processing, and the block symbols shown in 1, a, β represent a constant "1" and two self-learning parameters "a" and "β", respectively; Figure 6b The parameters k and r in the 4 multi-scale feature extraction branches in are {1, 3, 5, 7}, {1, 2, 4, 8}, respectively, and the circular symbols shown in +, C represent pixel-level addition processing and channel dimension splicing processing, respectively;

[0080] Step 2.4.0, define the current stage as k, and initialize k = 1; the pth initial stage decision graph is taken as the (k-1)th stage decision graph

[0081] Step 2.4.1: the preprocessing module performs downsampling operation on the pth initial stage decision graph so that and have the same size, and obtains the pth down-sampling decision graph then performs Sigmoid activation function operation to obtain the pth weight graph At the same time, after the same downsampling operation is performed on the pth coarse positioning decoder feature , the pth feature map is obtained.

[0082] Step 2.4.2: the preprocessing module subtracts "1" from the pth weight graph to obtain the pth reverse weight graph Then and are multiplied with the pth feature map , respectively, to obtain the pth forward feature map and the pth reverse feature map

[0083] Step 2.4.3: the multi-scale feature pyramid module processes and to obtain the output feature map with respectively, and the pth forward short feature map is obtained after short connection with the pth feature map

[0084] Step 2.4.3.1: After adjusting the channels by 4 third convolution blocks Conv 1×1 , 4 channel output feature maps are obtained

[0085] Step 2.4.3.2: 4 channel output feature maps are respectively input into 4 multi-scale feature extraction branches, and 4 symmetric convolution feature maps are obtained by symmetric convolution blocks Conv k×k After processing by a second stride convolution block Conv k×k,r , 4 stride convolution feature maps are obtained respectively

[0086] Step 2.4.3.3: 4 channel output feature maps 4 symmetric convolution feature maps and 4 stride convolution feature maps are respectively spliced in the channel dimension along the output order of the 4 multi-scale feature extraction branches, and 4 multi-scale fusion feature maps are obtained accordingly

[0087] Step 2.4.3.4: 4 multi-scale fusion feature maps are respectively superimposed with the previous multi-scale fusion features, and 4 superimposed feature maps are obtained accordingly

[0088] Step 2.4.3.5: after the 4 superimposed feature maps are spliced again in the channel dimension, a spliced feature map is obtained is input into a fifth third convolution block Conv 1×1 to adjust the channels, and an adjusted feature map is obtained

[0089] Step 2.4.3.6: after is added to and processed by a ReLU activation function, the pth multi-scale feature pyramid forward feature map is obtained

[0090] Step 2.4.3.7: according to the process of step 2.4.3.1-step 2.4.3.6, the feature map ​​​​After the same processing, the pth multi-scale feature pyramid reverse feature map is obtained

[0091] Step 2.4.4: pth forward short feature map feature and the pth reverse short feature map After being multiplied by two self-learning parameters "a" and "b" respectively, the pth self-learning feature map pair is obtained and

[0092] Step 2.4.5: the pth feature map is processed through the third convolutional block Conv 3×3 After the upsampling operation, the pth upsampled feature map is obtained

[0093] Step 2.4.6: the pth self-learning feature map is subtracted from the pth upsampled feature to obtain the pth fusion feature F p ;

[0094] Step 2.4.7: the pth fusion feature F p , is added to the pth self-learning feature map to obtain the pth multi-scale feature interaction feature map

[0095] Step 2.4.8: determine whether k=K is true, if true, it means that 1-Y-1 receptive field enhanced feature maps are obtained corresponding to the pth series of multi-scale interaction feature maps and the pth series of decision maps wherein, represents the yth receptive field enhanced feature map, represents the kth stage feature map in the pth series of multi-scale interaction feature maps, represents the kth stage decision map in the pth series of decision maps; otherwise, step 2.4.9 is executed; in this embodiment, K=3;

[0096] Step 2.4.9: the pth series of decision maps is processed through the third convolutional block Conv 3×3 for 3 times to obtain the pth kth stage decision map

[0097] Step 2.4.10: the pth series of decision maps is respectively subjected to an upsampling operation with the pth kth stage decision map so that and The size of the pth up-sampling decision map is the same as that of the pth initial decision map, and the pth up-sampling decision map is obtained accordingly The size of the pth up-sampling feature map is the same as that of the pth initial decision map, and the pth up-sampling feature map is obtained accordingly The size of the pth up-sampling feature map is the same as that of the pth initial decision map, and the pth up-sampling feature map is obtained accordingly The size of the pth up-sampling feature map is the same as that of the pth initial decision map, and the pth up-sampling feature map is obtained accordingly

[0098] As shown in the structural diagram of FIG. 6, the pth coarse positioning feature map and the pth initial decision map are input into the multi-scale feature interaction module, and the pth series of multi-scale interaction features are obtained after k is assigned to k+1 and the step 2.4.1 is sequentially executed. Figure 2 The size of the pth series of multi-scale interaction features is 128x128x64, 64x64x128, and 32x32x320, respectively. The size of the pth series of decision maps is 128x128x1, 64x64x1, and 32x32x1, respectively.

[0099] Step 2.5: the pth down-sampling decision map is obtained by performing down-sampling processing on the pth series of decision maps. After up-sampling processing, the first fusion decision map is obtained. After the pth series of decision maps are sequentially added, the pth series of fusion decision maps are obtained. The size of the pth series of fusion decision maps is 32x32x1, 32x32x1, 64x64x1, and 128x128x1, respectively. The size of the pth series of fusion decision maps is 32x32x1, 32x32x1, 64x64x1, and 128x128x1, respectively. The size of the pth series of fusion decision maps is 32x32x1, 32x32x1, 64x64x1, and 128x128x1, respectively. The size of the pth series of fusion decision maps is 32x32x1, 32x32x1, 64x64x1, and 128x128x1, respectively. The size of the pth series of fusion decision maps is 32x32x1, 32x32x1, 64x64x1, and 128x128x1, respectively.

[0100] As shown in the structural diagram of FIG. 6, the pth coarse positioning feature map and the pth initial decision map are input into the multi-scale feature interaction module, and the pth series of multi-scale interaction features are obtained after k is assigned to k+1 and the step 2.4.1 is sequentially executed. Figure 2 The size of the pth series of fusion decision maps is 32x32x1, 32x32x1, 64x64x1, and 128x128x1, respectively.

[0101] Step 2.6: After up-sampling operation and Sigmoid activation function processing, the pth multi-level output decision map is obtained. The size of the pth series of fusion decision maps is 32x32x1, 32x32x1, 64x64x1, and 128x128x1, respectively. The size of the pth series of fusion decision maps is 32x32x1, 32x32x1, 64x64x1, and 128x128x1, respectively. The size of the pth series of fusion decision maps is 32x32x1, 32x32x1, 64x64x1, and 128x128x1, respectively. The size of the pth series of fusion decision maps is 32x32x1, 32x32x1, 64x64x1, and 128x128x1, respectively. The size of the pth series of fusion decision maps is 32x32x1, 32x32x1, 64x64x1, and 128x128x1, respectively. The size of the pth series of fusion decision maps is 32x32x1, 32x32x1, 64x64x1, and 128x128x1, respectively. The size of the pth series of fusion decision maps is 32x32x1, 32x32x1, 64x64x1, and 128x128x1, respectively. ​​​​​​​​

[0102] like Figure 2 The structure diagram shown is the p-th series fusion decision diagram. After upsampling to match the input image size (512×512×1), and then applying the Sigmoid activation function, the p-th multi-level output decision map is obtained. in Let p be the final decision map for the p-th pair of grayscale images;

[0103] Step 3: Construct the loss function using equation (1)

[0104]

[0105] In equation (1), L wBCE L represents the weighted binary cross-entropy loss. wIOU Indicates the weighted average loss;

[0106] The total loss function L of the multi-scale context-aware network is constructed using equation (2). total :

[0107]

[0108] Step 4: Based on the training set, train the multi-scale context-aware network using the backpropagation algorithm and calculate the total loss function L. total The network parameters are adjusted until the maximum number of iterations is reached, thereby obtaining a trained multi-scale context-aware network. In this embodiment, the Adam optimizer is used to optimize the total loss.

[0109] Step 5: Utilize the final decision diagram After inverting, we obtain the reverse decision graph. The final decision diagram will be displayed separately. and Reverse Decision Graph and After performing pixel-by-pixel multiplication, a partially clear image is obtained. and Partially clear images and Perform pixel-by-pixel addition to obtain the p-th grayscale image. and Predicted full-focus image

[0110] In this embodiment, as Figure 3 The structural diagram shown indicates that the × and + symbols in the circles represent pixel-level multiplication and pixel-level addition, respectively; using the final decision graph... After inverting, we obtain the reverse decision graph. respectively, and the final decision map and the inverse decision map and after pixel-by-pixel multiplication, a partially clear image is obtained and the partially clear image and is added pixel by pixel, thereby obtaining a predicted all-in-focus image of the p-th pair of gray scale images and .

[0111] In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0112] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program is run by a processor to perform the steps of the above method.

Claims

1. A multi-focus image fusion method based on multi-scale context awareness, characterized in that, The method comprises the following steps: Step 1: Obtain P pairs of RGB multi-focus images and convert them into gray images, denoted as and , and take them as the training set, where, and represent the foreground focus image and the background focus image in the pth pair of gray images respectively; take the pth real ground mask corresponding to the pth pair of gray images as the pth label, denoted as G p , so as to construct the label set of P pairs of RGB multi-focus images {G 1 , G 2 , …, G p , …G P}; Step 2: constructing a multi-scale context perception network, comprising: an encoder, a coarse positioning decoder, a receptive field enhancement module and a multi-scale feature interaction module; Step 2.1: the encoder includes a first convolutional block Conv 3×3 and Y visual converters, wherein Conv 3×3 represents a convolutional layer with a 3x3 convolution kernel and a ReLU activation function; the p-th pair of gray-scale images and After the concatenation in the channel dimension, the input is input into a multi-scale context perception network, and after the processing of the first convolution block Conv 3×3 of the encoder, the p-th input feature I p is obtained. After the processing of the Y visual converters in turn, Y primary feature maps corresponding to the p-th pair of gray-scale images are obtained in turn. wherein, y represents the y-th primary feature map. Step 2.2: the coarse positioning decoder is composed of several multi-stage cross-scale connected second convolution blocks Conv 3×3 and 1 first convolution block Conv 1×1 , and performs M R stage feature decoding on Y primary feature maps to obtain the pth coarse positioning decoder feature and the pth initial decision map wherein Conv 1×1 represents a convolution layer with a 1x1 convolution kernel; Step 2.3: the receptive field enhancement module is composed of 4 receptive field enhancement branches with the same structure but different parameters k and r, and 5 second convolution blocks Conv 1×1 , 1 ReLU activation function, wherein each receptive field enhancement branch is formed by stacking 1 asymmetric convolution block Conv 1×k , 1 asymmetric convolution block Conv k×1 and 1 first stride convolution block Conv k×k,r in sequence; wherein Conv 1×k represents an asymmetric convolution layer with a convolution kernel of 1xk, Conv k×1 represents an asymmetric convolution layer with a convolution kernel of kx1, and Conv k×k,r represents a symmetric convolution layer with a convolution kernel of kxk and a stride of r; Y-1 primary feature maps Parallelly input into the receptive field enhancement module, wherein, Through 5 second convolution blocks Conv 1×1 After adjusting the channel, 5 output features The last 4 output feature maps Correspondingly input into 4 receptive field enhancement branches for processing, to obtain 4 receptive field enhancement branch feature maps And After splicing in the channel dimension, the yth fusion feature map is obtained And input into the second convolution block Conv 1×1 After adjusting the channel, the adjusted yth feature map is obtained The And After adding and processing through the activation function ReLU, the final output of the yth receptive field enhanced feature map is obtained Thus, Y-1 receptive field enhanced feature maps are obtained Step 2.4: the multi-scale feature interaction module is composed of a preprocessing module, a multi-scale feature pyramid module, a third convolutional block Conv 3×3 and sequentially processes the pth coarse positioning decoder feature and the pth initial stage decision map to obtain 1-Y-1 receptive field enhanced feature maps corresponding to the pth series of multi-scale interaction feature maps and the pth series of decision maps wherein, represents the yth receptive field enhanced feature map, represents the kth stage feature map in the pth series of multi-scale interaction feature maps, represents the kth stage decision map in the pth series of decision maps. Step 2.5: the pth down-sampling decision map After up-sampling, the first fusion decision map is obtained respectively with the pth series of decision maps After step-by-step addition, the pth series of fusion decision maps is obtained wherein, denotes the kth fusion decision map of the pth pair of gray-scale images and ​ Step 2.6: After the up-sampling operation and the Sigmoid activation function processing, the pth multi-level output decision map is obtained wherein, represents the kth output decision map of the pth pair of gray-scale images and , and is taken as the final decision map of the pth pair of gray-scale images and ; Step 3: Constructing the loss function with formula (1) In formula (1), L wBCE denotes a weighted binary cross-entropy loss, L wIOU denotes a weighted intersection over union loss; The total loss function L of the multi-scale context perception network is constructed by using formula (2) total : Step 4: based on the training set, the multi-scale context-aware network is trained by using a back propagation algorithm, and the total loss function L is calculated total to adjust the network parameters until the maximum number of iterations is reached, thereby obtaining the trained multi-scale context-aware network; Step 5: Using the final decision map After negation, the reverse decision map is obtained The final decision map and the reverse decision map are respectively multiplied pixel by pixel to obtain a partially clear image and The partially clear image is added pixel by pixel to obtain the predicted all-in-focus image of the pth pair of gray-scale images and and ​​ 2. The multi-focus image fusion method based on multi-scale context perception according to claim 1, characterized in that, The step 2.2 comprises: Step 2.2.1, when r = 1, the coarse positioning decoder is in the M r th stage, and the Yth primary feature map is respectively subjected to twice upsampling operation with different weights, so that and have the same size, and the rth upsampling feature map and the r+1th upsampling feature map are obtained. After that, the rth and r+1th second convolutional blocks Conv 3×3 are respectively input for processing, and the rth and r+1th feature maps and are obtained. is multiplied by , and then is spliced in the channel dimension, and after sequentially passing through the r+2th and r+3th second convolutional blocks Conv 3×3 , the M r th stage output feature is obtained. When r=2, the coarse positioning decoder is in the M r th stage, the Yth primary feature map and the Y-1th primary feature map are respectively subjected to up-sampling operations, so that the sizes of and are the same as the size of , and the r+1th up-sampled feature map and the r+2th up-sampled feature map are obtained. After that, the r+3th and r+4th second convolutional blocks Conv 3×3 are input for processing, and the r+1th and r+2th feature maps and are obtained. is multiplied by , and then is spliced with in the channel dimension, and sequentially passes through the r+5th and r+6th second convolutional blocks Conv 3×3 for processing, and the M r th stage output feature is obtained. When r = 3, 4,..., R-1, the coarse positioning decoder is at the M r th stage and the same processing is performed on the Yth primary feature map to the Y-r+1th primary feature map to obtain the rth to the R-1th stage output feature When r=R, the output feature of the R-1th stage is taken as the input of the Rth stage The last two second convolution blocks Conv 3×3 The output feature map of the Rth stage is obtained by processing the input in the Rth stage The pth coarse positioning decoder feature map finally output by the coarse positioning decoder is taken as the input of the first convolution block Conv 1×1 The pth initial stage decision map is obtained after the first convolution block Conv 3. The multi-focus image fusion method based on multi-scale context perception according to claim 2, characterized in that, The multi-scale feature pyramid module in step 2.4 is composed of 4 multi-scale feature extraction branches with the same structure but different parameters k and r, and 5 third convolution blocks Conv 1×1 , 1 ReLU activation function, wherein each multi-scale feature extraction branch is formed by stacking 1 symmetric convolution block Conv k×k , 1 second stride convolution block Conv k×k,r in sequence, wherein Conv k×k represents a symmetric convolution layer with a convolution kernel of k x k and a ReLU activation function. Step 2.4.0, define the current stage as k and initialize k = 1; draw the pth initial stage decision graph as the (k-1)th stage pth decision graph Step 2.4.1: the pre-processing module performs a down-sampling operation on the pth decision graph of the k-1th stage so that and have the same size, and obtains the pth down-sampled decision graph of the k-1th stage then performs a Sigmoid activation function operation to obtain the pth weight graph of the k-1th stage Meanwhile, the pth coarse positioning decoder feature is subjected to the same down-sampling operation to obtain the pth feature map Step 2.4.2: The preprocessing module subtracts the p-th weight map from the value of "1". Then, the p-th reverse weight map of the (k-1)-th stage is obtained. Then and Each with the p-th feature map After multiplication, the p-th positive feature map of the (k-1)-th stage is obtained. Compared with the p-th inverse feature map of the (k-1)th stage Step 2.4.3: The multi-scale feature pyramid module respectively processes the pth feature map and the (p-1)th feature map to obtain the output feature map and the output feature map After respectively performing short connection with the pth feature map and the (p-1)th feature map, the pth forward short feature map and the pth reverse short feature map Step 2.4.3.1: After passing through 4 third convolutional blocks Conv 1×1 The output feature map of the k-1 stage is obtained after adjusting the channel Step 2.4.3.2: 4-channel output feature maps of the k-1th stage respectively input 4 multi-scale feature extraction branches, by the symmetric convolution block Conv k×k obtain 4 symmetric convolution feature maps of the k-1th stage After the processing of the second stride convolution block Conv k×k,r obtain 4 stride convolution feature maps of the k-1th stage respectively Step 2.4.3.3: the k-1 stage 4-channel output feature maps the k-1 stage 4-channel symmetric convolution feature maps and the k-1 stage 4-channel stride convolution feature maps After respectively splicing the output sequences along the channel dimension of the 4 multi-scale feature extraction branches, the k-1 stage 4 multi-scale fusion feature maps are obtained Step 2.4.3.4: superimpose the k-1th stage 4 multi-scale fusion feature maps with the previous several multi-scale fusion features respectively, and obtain the k-1th stage 4 superimposed feature maps respectively ​ Step 2.4.3.5: After concatenating the four superimposed feature maps of stage k-1 again along the channel dimension, we obtain a single concatenated feature map of stage k-1. Will Input the 5th third convolutional block Conv 1×1 After adjusting the channels, the feature map after the (k-1)th stage adjustment is obtained. Step 2.4.3.6: After adding and processing through the ReLU activation function, the pth multi-scale feature pyramid forward feature map is obtained and ​ Step 2.4.3.7: Perform the same process as step 2.4.3.1-step 2.4.3.6 on the feature map After the same processing, the pth multi-scale feature pyramid reverse feature map is obtained Step 2.4.4: p-th forward short feature map feature with the p-th backward short feature map After multiplying by two self-learning parameters "a" and "b" respectively, the p-th self-learning feature map pair is obtained and Step 2.4.5: the p-th feature map is obtained by performing convolution operation on the (p-1)-th feature map After the third convolution block Conv 3×3 After the up-sampling operation, the p-th up-sampled feature map is obtained Step 2.4.6: subtracting the pth self-learning feature map from the pth up-sampling feature map to obtain the pth fusion feature F p ;​​ Step 2.4.7: add the p-th fusion feature F p , and the p-th self-learning feature map to obtain the k-th stage p-th multi-scale feature interaction feature map Step 2.4.8: judge whether k = K is true, if true, it means that 1-Y-1 feature maps with enhanced receptive field are obtained Corresponding K-stage p-series multi-scale interaction feature map K-stage p-series decision map Otherwise, step 2.4.9 is executed; Step 2.4.9: The kth stage is obtained by processing the (k-1)th stage through the third convolutional block Conv After processing through the third convolutional block Conv for 3 times, the kth stage is obtained 3×3 The kth stage is obtained by processing the (k-1)th stage through the third convolutional block Conv Step 2.4.10: [The text appears to be incomplete and contains several grammatical errors. A more accurate With the p-th decision graph D in the k-th stage k p Perform upsampling operations separately, so that... and The dimensions are the same, and the p-th upsampling decision map of the k-th stage is obtained accordingly. and the p-th upsampled feature map in the k-th stage and feature maps enhanced with receptive fields After inputting k+1 into the multi-scale feature interaction module, and assigning k+1 to k, return to step 2.4.1 for sequential execution.

4. An electronic device comprising a memory and a processor, characterized in that The memory is configured to store a program supporting the processor to execute the multi-focus image fusion method of any one of claims 1-3, and the processor is configured to execute the program stored in the memory.

5. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to execute the steps of the multi-focus image fusion method of any one of claims 1-3.

Images