Multi-focus image fusion method based on multilayer semantics and multi-scale self-attention

Abstract

The invention relates to a multi-focus image fusion method based on multi-layer semantics and multi-scale self-attention, and the method is characterized in that the method comprises the steps: obtaining a source image, inputting the source image into an MSI-Dtrans model, carrying out the feature extraction of the source image through Haar wavelet transform, obtaining the feature information of different scales, and carrying out the fusion of the multi-focus image through the MSI-Dtrans model. Inputting the feature information of different scales into a dynamic Transform for processing, and obtaining an initial focus image; wherein the network architecture of the MSI-Dtrans model adopts a multi-stage up-down sampling structure of a U-shaped network and is obtained through training of a training set, and the training set comprises multiple image pairs generated based on blurred images of different degrees and training labels; and fusing the initial focus image with the source image to obtain a final fused image. According to the invention, image distortion is effectively avoided and a good visual effect is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-focus image fusion, and in particular to a multi-focus image fusion method based on multi-layer semantics and multi-scale self-attention. Background Art

[0002] Traditional multi-focus image fusion methods can be roughly divided into two categories: fusion methods based on the change domain and fusion methods based on the spatial domain. Although the change domain-based method has strong adaptability, it is easy to improperly handle the boundary between the clear and blurred areas, which in turn produces edge artifacts or edge blur in the fused image, resulting in poor quality of the fused image. Although the spatial domain-based method has high computational efficiency and is more suitable for real-time processing scenarios. However, operating directly at the pixel level may ignore the spatial relationship between pixels, resulting in problems such as block effects or edge discontinuities in the fused image.

[0003] With the development of deep learning models, the solution in the field of image fusion has gradually shifted from statistical methods to deep learning. Although the deep learning-based method can automatically extract semantic information from images, it still has problems such as large demand for training data, high model complexity, and large demand for computing resources. To solve the above problems, the present invention proposes a multi-focus image fusion method based on multi-layer semantics and multi-scale self-attention. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-focus image fusion method based on multi-layer semantics and multi-scale self-attention, which effectively avoids image distortion and achieves better visual effects.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] Multi-focus image fusion method based on multi-layer semantics and multi-scale self-attention, including:

[0007] Get the source image;

[0008] The source image is input into the MSI-Dtrans model, and Haar wavelet transform is used to extract features of the source image to obtain feature information of different scales, and the feature information of different scales is input into the dynamic Transformer for processing to obtain an initial focus map; wherein the network architecture of the MSI-Dtrans model adopts a multi-level up-down sampling structure of a U-type network, which is obtained through training with a training set, and the training set includes multiple image pairs generated based on blurred images of different degrees and training labels;

[0009] The initial focus image is fused with the source image to obtain a final fused image.

[0010] Optionally, extracting features from the source image using Haar wavelet transform includes:

[0011] S1. Performing preliminary feature extraction on the source image through a multi-scale semantic extraction module to obtain an intermediate low-frequency feature map and an intermediate high-frequency feature map;

[0012] S2, inputting the intermediate low-frequency feature map and the intermediate high-frequency feature map into a semantic splitting module, performing high-low frequency splitting on the intermediate low-frequency feature map and the intermediate high-frequency feature map using the Haar wavelet transform, and obtaining a low-frequency sub-band and a high-frequency sub-band;

[0013] S3, using a rough semantic enhancement module to perform multi-layer semantic interaction on the low-frequency sub-band to obtain low-frequency features;

[0014] S4, using a fine semantic extraction module to perform multi-layer semantic interaction on the high-frequency sub-band to obtain high-frequency features, wherein the fine semantic extraction module introduces residual connection and Mish activation function;

[0015] S5. Use the low-frequency features and the high-frequency features as the intermediate low-frequency feature map and the intermediate high-frequency features, return to step S2, repeat to a preset number of times, and output feature information of different scales.

[0016] Optionally, a rough semantic enhancement (RSE) module is used to perform multi-layer semantic interaction on the low-frequency sub-band to obtain low-frequency features, including: calculating the difference characteristics of the low-frequency sub-band, splicing the difference features according to the number of channels, extracting information, performing a residual connection between the extracted information and the low-frequency sub-band, and outputting a low-frequency feature map.

[0017] Optionally, the calculation method for outputting the low-frequency feature map is:

[0018]

[0019] Among them, P and P' represent the feature representations obtained after convolution with kernel size 3 and 1, respectively, h represents parallel path 1 or 2, j represents the relative position of the coarse semantic enhancement module in the overall structure, μ represents the ReLU activation function, * represents the convolution operation, b represents the bias value of the output feature map, D represents the feature representation obtained after difference calculation, F represents the low-frequency feature map, T N×N Represents a convolution operation with a convolution kernel size of N, and X is the input feature map.

[0020] Optionally, inputting the feature information of different scales into a dynamic Transformer for optimization, and obtaining an initial focus map includes:

[0021] Using depthwise separable convolution to replace the fully connected layer in the Transformer model to integrate the feature information of different scales, wherein each of the depthwise separable convolutions is set with a different dilation rate and convolution step size;

[0022] The integrated feature information of different scales is input into a multi-head attention module to perform multi-scale joint attention calculation to obtain the initial focus map.

[0023] Optionally, the multi-scale joint attention is calculated as:

[0024]

[0025] Multi_scale_joint_attention

[0026] =Concat(Att1,Att2,...,Att n )W 0

[0027] i∈(1,2,3,...,n)

[0028] Among them, Conv Dia and Conv 1×1 Represents depth-wise separable convolution and point-wise convolution, Dia represents dilated convolution, f i , p i ,s i , d i Respectively represent the corresponding convolution kernel size, edge padding size, convolution step size and dilation convolution rate, Att i represents the attention feature of the i-th feature scale, W is the learnable weight matrix used to generate different Query, Value and Key sequences, and W 0 is the learnable weight matrix used in multi-scale attention calculation, q i , v i , k i are the Query, Value, and Key sequences generated at the i-th scale, respectively. X is the input feature map. Q, V, and K are the Query, Value, and Key sequences generated after the Concat operation. m , V m , K m are the Query, Value, and Key sequences input into the m-th multi-head attention mechanism for calculation, and n is the number of depth-wise separable convolutions with n scales.

[0029] Optionally, obtaining the final fused image includes:

[0030] Obtaining an initial focus map, binarizing the initial focus map, obtaining an initial decision map, performing small area filtering on the initial decision map, and obtaining a final decision map;

[0031] According to the pixel weighted average strategy, the final decision graph is fused with the source image to obtain the final fused image.

[0032] Optionally, the method further comprises: simultaneously performing model training by back propagation using a loss function;

[0033] The loss function is:

[0034] Loss sum =λLoss L1 +μLoss SSIM +σLoss DICE

[0035] Among them, Loss sum is the joint training loss function of the network, Loss L1 、Loss SSIM and Loss DICE They represent L1 loss function, SSIM loss function and DICE loss function respectively. λ, μ and σ are hyperparameters for balancing the proportions of the three loss functions.

[0036] The beneficial effects of the present invention are:

[0037] On the one hand, the present invention makes full use of all the effective information carried by the source image, integrates Haar wavelet into network feature extraction, and proposes a multi-layer semantic interaction strategy to enhance the interaction between high-frequency and low-frequency information, gradually mine more abstract semantic information, and guide the generation of feature maps from coarse to fine. On the other hand, the use of dynamic receptive field and dynamic Token embedding overcomes the performance degradation problem when processing multi-scale objects, enables self-attention to integrate long-range dependencies between objects of different scales, and reduces computational overhead. A large number of experimental results show that the proposed method effectively avoids image distortion and achieves better visual effects. Compared with many state-of-the-art methods, it has good competitiveness in qualitative, quantitative analysis and efficiency comparison. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0039] Figure 1This is a workflow diagram of a multi-focus image fusion method based on multi-layer semantics and multi-scale self-attention according to an embodiment of the present invention;

[0040] Figure 2 A comparison of the process of processing the intermediate feature map by the traditional multi-level Haar wavelet transform of the embodiment of the present invention and the method proposed in this embodiment;

[0041] Figure 3 It is an intermediate feature map generated after processing using a multi-level deep semantic interaction strategy and optimized Haar wavelet in an embodiment of the present invention;

[0042] Figure 4 This is a schematic diagram of the dynamic Token embedding process according to an embodiment of the present invention;

[0043] Figure 5 It is a schematic diagram of the fusion process of an embodiment of the present invention;

[0044] Figure 6 Schematic diagram of the process of preparing a training set according to an embodiment of the present invention;

[0045] Figure 7 This is a structural diagram of the MSI-Dtrans model of an embodiment of the present invention;

[0046] Figure 8 1 is a graph showing the visual effects of different methods of the embodiments of the present invention on the "Grayscale" dataset, wherein (a) is the source image A, (b) is the source image B, (c) is the fusion result graph of the method GFF, (d) is the difference graph between the method GFF and the source image B, (e) is the fusion result graph of the method MWGF, (f) is the difference graph between the method MWGF and the source image B, (g) is the fusion result graph of the method IFCNN, (h) is the difference graph between the method IFCNN and the source image B, (i) is the fusion result graph of the method U2Fusion, (j) is the difference graph between the method U2Fusion and the source image B, (k) is the fusion result graph of the method SwinFusion, and (l) is the fusion result graph of the method SwinFusion and the source image B. (m) is the fusion result image of method ZMFF, (n) is the difference image between method ZMFF and source image B, (o) is the fusion result image of method FusionDiff, (p) is the difference image between method FusionDiff and source image B, (q) is the fusion result image of method DRPL, (r) is the difference image between method DRPL and source image B, (s) is the fusion result image of method SESF, (t) is the difference image between method SESF and source image B, (u) is the fusion result image of method MFIF-GAN, (v) is the difference image between method MFIF-GAN and source image B, (w) is the fusion result image of method MSI-Dtrans, and (x) is the difference image between method MSI-Dtrans and source image B. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] like Figure 1 As shown, the present invention discloses a multi-focus image fusion method based on multi-layer semantics and multi-scale self-attention, including:

[0050] Obtain the source image, input the source image into the MSI-Dtrans model, use Haar wavelet transform to extract features of the source image, obtain feature information of different scales, input the feature information of different scales into the dynamic Transformer for processing, and obtain the initial focus image;

[0051] The network architecture of the MSI-Dtrans model adopts a multi-level up-down sampling structure of a U-shaped network, which is obtained through training with a training set. The training set includes multiple image pairs generated based on blurred images of different degrees and training labels.

[0052] The initial focus image is fused with the source image to obtain the final fused image.

[0053] Furthermore, the feature extraction of the source image using Haar wavelet transform includes:

[0054] S1, perform preliminary feature extraction on the source image through a multi-scale semantic extraction module to obtain an intermediate low-frequency feature map and an intermediate high-frequency feature map;

[0055] S2, inputting the intermediate low-frequency feature map and the intermediate high-frequency feature map into the semantic splitting module, performing high-low frequency splitting on the intermediate low-frequency feature map and the intermediate high-frequency feature map using Haar wavelet transform, and obtaining low-frequency sub-band and high-frequency sub-band;

[0056] S3, using the rough semantic enhancement module to perform multi-layer semantic interaction on the low-frequency sub-band to obtain low-frequency features;

[0057] S4, using the fine semantic extraction module to perform multi-layer semantic interaction on the high-frequency sub-band to obtain high-frequency features, where the fine semantic extraction module introduces residual connection and Mish activation function;

[0058] S5. Use the low-frequency features and the high-frequency features as the intermediate low-frequency feature map and the intermediate high-frequency features, return to step S2, repeat to a preset number of times, and output feature information of different scales.

[0059] Output low-frequency feature map; the calculation method of the output feature map is shown in formulas (3)-(6).

[0060] Furthermore, the rough semantic enhancement (RSE) module is used to perform multi-layer semantic interaction on the low-frequency sub-bands to obtain low-frequency features, including: calculating the difference characteristics of the low-frequency sub-bands, splicing the difference features according to the number of channels, extracting information, performing residual connection between the extracted information and the low-frequency sub-bands, and outputting a low-frequency feature map.

[0061] Furthermore, feature information of different scales is input into the dynamic Transformer for optimization, and the initial focus map is obtained including:

[0062] Depthwise separable convolution is used to replace the fully connected layer in the Transformer model to integrate feature information of different scales, where each depthwise separable convolution is set with a different dilation rate and convolution step size;

[0063] The integrated feature information of the same scale is input into the multi-head attention module to perform multi-scale joint attention calculation to obtain the initial focus map, i.e., the initial decision map. The multi-scale joint attention calculation method is shown in formulas (9)-(13):

[0064] Obtaining the final fused image includes:

[0065] Obtain an initial focus map, binarize the initial focus map, obtain an initial decision map, perform small area filtering on the initial decision map, and obtain a final decision map;

[0066] According to the pixel weighted average strategy, the final decision map is fused with the source image to obtain the final fused image.

[0067] Furthermore, the method also includes: using the loss function to simultaneously perform model training through back propagation; the loss function calculation formula is shown in (14).

[0068] The present invention will be further explained below with reference to the accompanying drawings:

[0069] According to Haar wavelet change theory:

[0070]

[0071] Wherein, f(x, y) represents the pixel value of the image at the position (x, y). After a single Haar wavelet transform, cA represents a low-frequency subband that approximates the source image, mainly including information such as the color and overall structure of the image. cH, cV, and cD represent high-frequency subbands of the source image in the horizontal, vertical, and diagonal directions, respectively, mainly including information such as edges and textures. The existing multi-level Haar wavelet transform only performs multiple Haar transforms on cA and partially or completely integrates the results into the neural network as an information supplement for the intermediate feature map. Although this approach can enrich the feature representation of the image to a certain extent. However, the transformed cA is similar to the source image, and performing multiple Haar transforms on it will lead to duplication and redundancy of information. At the same time, the blurring degree of the cA subband is deepened. The high-frequency information obtained by decomposition often carries more noise and pseudo-features than the high-frequency information of the previous level. It is directly used as a supplement to the intermediate feature map, resulting in inaccurate results when the neural network performs the next step of processing the image. Therefore, in order to overcome the above problems, this embodiment starts with the object after Haar transform, combines all subbands according to a certain combination strategy, and then inputs them into the differentiated module for processing to ensure the effectiveness of semantic fusion.

[0072] According to the scaled dot product attention calculation:

[0073]

[0074] Among them, Q, K, and V are three vectors obtained by linear transformation of the input sequence, which play an important role in the self-attention mechanism. In the self-attention layer of the traditional Transformer, the transformed Q, K, and V are finally represented by the weighted sum after self-attention calculation. Although this mechanism enables the Transformer to process the association between each position in the input sequence, so as to better capture long-distance dependencies, it cannot process feature information of different scales. The main reason is that the Query, Key, and Value sequences used for attention calculation are homologous, which makes it only able to model the attention of the domain where the current feature sequence is located, and cannot pay attention to the cross-domain interactive features. In addition, since Q, K, and V at each position need to perform dot product calculations with other positions, the amount of calculation will increase with the increase of sequence length. To overcome the above problems, starting from the embedding process of Q, K, and V, a variable embedding strategy is introduced to optimize the diversity of feature sequences, and at the same time, the computational overhead of attention is reduced by constructing a variable feature sequence length.

[0075] like Figure 1As shown, the network architecture proposed in this embodiment adopts a multi-level up and down sampling structure of a U-type network as a whole, which can be divided into two stages according to different functions: a feature extraction stage and a decision graph stage. Specifically, in the feature extraction stage, a new multi-level semantic interaction strategy is designed to realize the complex semantic information extraction task of the input source image. In order to generate a higher quality decision graph, a dynamic Transformer is designed in the decision graph generation stage to be responsible for long-term modeling of the intermediate feature graph and mining more related information. Secondly, in order to reduce the computational overhead, the problem of high algorithm time complexity caused by multi-level semantic interaction processing and Transformer architecture stacking due to multiple classifications is avoided as much as possible: Haar wavelet changes are used instead of traditional U-Net to use convolution for multi-level downsampling, and the resolution of the intermediate feature graph is gradually reduced without increasing the number of parameters. At the same time, compared with the traditional Transformer architecture, the dynamic Transformer architecture proposed in this embodiment uses a deep separable convolution mapping instead of the traditional linear mapping and position encoding operation. By changing the convolution step size, the downsampling operation is implemented on the intermediate feature map after convolution, and the information from different domains is integrated in a single attention calculation, further reducing the time overhead.

[0076] The introduced information is only spliced ​​according to the channel dimension, resulting in a large amount of information being ignored, which increases the computational consumption while the fusion effect is not as good as expected. Therefore, this embodiment overcomes the above problems through the multi-focus image fusion model MSI-DTrans of multi-layer semantic interaction and dynamic Transformer. In the feature extraction stage, the multi-level improved Haar transform is used to continuously split and combine the high and low frequencies of the network intermediate feature map, overcoming the disadvantage that the original Haar wavelet only has four directional sub-bands and is difficult to capture complex texture and shape features. At the same time, with the help of the multi-level semantic interaction strategy, the existing multi-level Haar wavelet only filters the cA sub-band multiple times to introduce pseudo features, resulting in poor fusion effect. In this embodiment, the generated high and low frequency images are classified and integrated, so that the intermediate low frequency feature map contains both the main information in the lower frequency range and the image edge features from the higher frequency, and has a more comprehensive image feature description. For the intermediate high frequency feature map, each level of decomposition and fusion will gradually refine the detailed features of the image and improve the sensitivity to small changes. Secondly, Haar wavelet is a reversible transform that reduces the image size during the decomposition process, thereby achieving lossless downsampling of feature maps and reducing the computational load. Figure 2 The processing differences between the traditional multi-level Haar wavelet transform and the method proposed in this embodiment are demonstrated.

[0077] In order to enhance the network's ability to extract and process rich semantic information in images and to better preserve the important features and details of the source image during the fusion process, the feature extraction stage includes the following four network modules:

[0078] (1) The structure of the multi-scale semantic extraction module MSSE-Module (Multi-scale semantic extraction module) is as follows Figure 7 As shown in the figure, the module first uses two 3×3 convolutions to extract the initial features of the source image, and uses the attention mechanism to evaluate the importance of the intermediate feature map at the pixel level to enhance its information representation ability. Secondly, parallel differential convolution kernels are used to extract the next step of features to obtain multi-scale receptive fields and rich feature representations, thereby improving the model's understanding and expression capabilities of semantic information.

[0079] (2) The semantic splitting module SS Block (Semantic Splitting Block) converts the intermediate feature map generated by the previous module into feature information with different frequencies, namely low-frequency sub-bands and high-frequency sub-bands, through Haar wavelet transform, and semantically fuses the feature information according to the frequency with the help of a multi-layer semantic interaction strategy, thereby enhancing the interaction between high-frequency and low-frequency information and guiding the network to gradually mine more abstract semantic information. In addition, the two carefully designed FSE and RSE modules ensure the consistency of semantic information fusion.

[0080] (3) For the low-frequency subband generated by the SS Block in the previous stage, Figure 3 As shown in the lower left corner, this intermediate feature map example not only retains the overall trend information of the image in the lower frequency range, but also contains the edge detail information of the image in the higher frequency range, and has rich feature characteristics. In order to achieve better information capture and feature enhancement, a rough semantic enhancement module (RSE-module) is designed. The specific structure is as follows Figure 7 As shown in the figure, first, a 1×1 convolution layer is used to integrate spatial information without changing the channel dimension. Then, two parallel paths are used to calculate the difference characteristics between the two sets of features. Finally, the difference features obtained from the two parallel paths are concatenated according to the number of channels, and information is extracted and residually connected with the input feature map to obtain the output feature map F. The process can be expressed by a mathematical equation:

[0081]

[0082] Among them, P and P' represent the feature representations obtained after convolution with kernel size 3 and 1, respectively, h represents parallel path 1 or 2, j represents the relative position of the coarse semantic enhancement module in the overall structure, μ represents the ReLU activation function, * represents the convolution operation, b represents the bias value of the output feature map, D represents the feature representation obtained after difference calculation, F represents the low-frequency feature map, T N×NRepresents a convolution operation with a convolution kernel size of N, and X is the input feature map.

[0083] (4) For the high frequency subband generated by the SS Block in the previous stage, Figure 3 As shown in the lower right corner, this intermediate feature map example has the detailed information of the original signal in a higher range. At the same time, after multi-level optimization of Haar wavelet transform, it overcomes the disadvantage that the original Haar wavelet only has three high-frequency decomposition directions, so that the intermediate feature map generated after the combination has more detailed decomposition features from more angles, providing more levels of high-frequency feature description of the image. In order to achieve better information capture and feature enhancement, a fine semantic extraction module FSE-Module (Fine semantic extraction module) is designed. For the high-frequency combination feature map after Haar wavelet transform, the texture features are obvious. In order to reduce the computational overhead, a more streamlined design is used to extract the detailed features. The specific structural design is as follows: Figure 1 As shown in the figure, this module only uses two convolutional layers with a convolution kernel size of 3×3 for feature extraction. At the same time, in order to prevent the degradation problem of the deep network, this module introduces residual connections and uses Mish as the activation function to introduce nonlinear changes to prevent the gradient from disappearing. The mathematical expression of the Mish activation function is as follows:

[0084] Mish(x)=x·tanh(Softplus(x)) (7)

[0085] Softplus(x)=ln(1+e x ) (8)

[0086] The Mish function provides a linear response similar to the ReLU function in the positive interval while maintaining a smooth gradient in the negative interval, which helps to maintain a more stable gradient flow during training.

[0087] In the feature extraction stage, the main task is to extract significant features from images with different focal points, such as edges, textures, and colors. These features can reflect important information in the image. In the feature map generation stage, the main goal of generating a decision map is to determine which areas should retain clear image information based on the extracted features. The decision map can be regarded as a binary image, where the value of each pixel indicates the source of the image information at that location (i.e., which image is clearer at that location). The feature extraction stage provides the basic information required for decision map generation. By analyzing the feature strength, the decision map can effectively determine which areas should come from which source image.

[0088] The Transformer architecture was originally designed for the field of natural language processing (NLP), and the content of the input module is a text sequence. For the field of computer vision, the input of the visual Transformer architecture is an image. Compared with a text sequence, an image has the characteristics of a large data scale and a large number of high-dimensional features. If the input sequence is flattened into a one-dimensional sequence according to the initial design of the Transformer, and the similarity between all positions is calculated. For larger image inputs, as the feature size increases, the complete feature sequence of equal length also increases the overhead of attention calculation to a certain extent, which leads to high computational complexity, consumes a lot of computing resources and memory, and limits the scalability and applicability of the Transformer architecture. Secondly, Q, K, and V for the scaled dot product attention calculation are a set of identical feature sequences generated after linear mapping. They can only perform attention modeling on the domain where the current feature sequence is located, and do not fully consider the spatial structure characteristics of the image, and cannot capture structural correlations. In this regard, by Figure 4 As shown in the figure, firstly, the deep separable convolution is used to replace the traditional fully connected layer, which solves the problem that the position information of the image block cannot be expressed in the calculation of the self-attention mechanism, and reduces the number of parameters and calculations. In the design process, different dilation rates and convolution steps are set for each deep separable convolution. In detail, the use of different dilation rates is used to expand the receptive field without increasing the number of parameters, so as to mine the feature information of objects of different scales. The use of different convolution steps for convolution is equivalent to scaling the feature size after convolution, realizing the downsampling of the input feature map, that is, changing the length of the Token embedding, and to a certain extent avoiding the problem of huge computational overhead caused by the continuous increase of feature sequences and the constant length of the Token sequence. Then, the processed feature matrices of different scales will be flattened to obtain feature sequences, which are juxtaposed into multi-dimensional feature sequences by channel. For the obtained multi-dimensional feature sequences, the idea of ​​differentiated Query, Key, and Value is adopted, that is, the Query sequence is randomly shuffled by the channel dimension using the Shuffle algorithm, while the Key and Value sequences remain unchanged. The three sequences generated will be used as the input of the multi-scale joint attention calculation module.

[0089] In order to mine the attention features between objects of different scales and reduce the redundant computational overhead, the proposed method designs a multi-scale joint attention computation scheme. The main difference between this scheme and other schemes is that it can process features of different scales in parallel in a single attention computation. Vision Transformer divides the input features into several small-sized features and performs a multi-head attention computation respectively. Swin Transformer uses a sliding window strategy to perform multi-head attention computation in four smaller windows. Pyramid Vision Transformer obtains smaller feature sizes through a larger convolution step size, thereby improving the efficiency of attention computation. The proposed method integrates multi-scale features with the help of dynamic Token embedding, and replaces the original feature sequence with the integrated multi-scale feature sequence and inputs it into the multi-head attention module. Multi-head attention can be understood as a computational mechanism composed of multiple scaled dot product attentions in parallel. In the proposed method, each specific scale feature combination in the integrated feature sequence corresponds one-to-one to each scaled dot product attention, and a multi-scale feature attention computation can be regarded as a special multi-head attention computation. Therefore, only one multi-head attention computation is required to realize the joint processing of features of different scales, which can effectively reduce computational redundancy. The proposed multi-scale joint attention is calculated as follows:

[0090]

[0091] Among them, Conv Dia and Conv 1×1 Represents depth-wise separable convolution and point-wise convolution, Dia represents dilated convolution, f i , p i ,s i , d i Respectively represent the corresponding convolution kernel size, edge padding size, convolution step size and dilation convolution rate, Att i represents the attention feature of the i-th feature scale, W is the learnable weight matrix used to generate different Query, Value and Key sequences, and W 0 is the learnable weight matrix used in multi-scale attention calculation, q i , v i , k i are the Query, Value, and Key sequences generated at the i-th scale, respectively. X is the input feature map. Q, V, and K are the Query, Value, and Key sequences generated after the Concat operation. m , V m , K mare the Query, Value, and Key sequences input into the m-th multi-head attention mechanism for calculation, and n is the number of depth-wise separable convolutions with n scales.

[0092] The loss function plays a vital role in neural network training and is a key component for evaluating network performance and guiding model optimization. In this embodiment, the following loss function is used to simultaneously perform network training through back propagation:

[0093] Loss sum =λLoss L1 +μLoss SSIM +σLoss DICE (14)

[0094] μ=1-(λ+σ) (15)

[0095] Among them, Loss sum is the joint training loss function of the network. For the discriminant network based on the decision graph in this embodiment, if only a single loss function is used, the network will be unable to fully consider the multi-faceted information of the image, limiting the fusion effect. Therefore, the loss function used in this embodiment is composed of three loss functions in a certain proportion. L1 、Loss SSIM and Loss DICE Represent the L1 loss function, SSIM loss function and DICE loss function respectively, and λ, μ, σ are hyperparameters for balancing the proportions of the three loss functions. For multi-focus image fusion tasks, the pixels at the edge of the object often only occupy a small part of the pixels of the entire image. Therefore, it is more hoped that the network can first effectively complete the task, and then make a good judgment on the edge of the object. Therefore, the values ​​of λ, μ, σ here are 0.7, 0.2, and 0.1 respectively.

[0096] For the loss function Loss L1 The mean absolute error (MAE) is used to measure the difference between the initial focus map generated by the network and the true decision map. When the error between the network output decision map and the true decision map is measured, the L1 norm has a higher correlation with the image quality than the L2 norm, so the L1 norm is used as the loss function to evaluate the quality of the decision map. The loss function Loss L1 It is expressed as follows:

[0097]

[0098] Among them, I f and I gt They represent the initial focus map generated by the network and the true decision map label, respectively. H and W represent the height and width of the feature map.

[0099] For the loss function Loss SSIM , used to compare the similarity of two images and simulate the human eye's perception of images. It has a wide range of applications in evaluating image quality, image compression, and image processing, and calculates the loss function Loss according to the following formula SSIM :

[0100] Loss SSIM =1-SSIM(I f (x,y),I gt (x,y)) (17)

[0101]

[0102] Among them, μ f and σ f Represents the mean and standard deviation of the network prediction value, μ gt and σ gt They represent the mean and standard deviation of the true label respectively. The setting of the two constants C1 and C2 is to avoid the result when the denominator is zero. The larger the SSIM value, the higher the similarity.

[0103] In order to achieve a good boundary division effect, this embodiment introduces the loss function Loss DICE Used to measure the difference between the network predicted focus map and the true label, loss function Loss DICE It is expressed as follows:

[0104] Loss DICE =1-DICE(I f (x,y),I gt (x,y)) (19)

[0105]

[0106] Where H and W represent the height and width of the network output decision graph, respectively. The constant 1 is added to avoid the situation where the denominator is zero. The larger the value of DICE, the higher the similarity.

[0107] Due to the limited expressive power of the neural network and the irregular noise from the source image, the initial focus map D generated by the trained network out (x, y) is not always completely reliable. To ensure the reliability of the quality of the subsequent fused image, this embodiment first performs a binarization operation on the initial focus map with a fixed threshold of 0.5 to generate an initial decision map, so that it is strictly divided into a binary map of either 0 or 1.

[0108]

[0109] Among them, D init(x, y) is the pixel value at position (x, y) of the initial decision graph obtained after segmentation according to the above strategy. out (x, y) represents the initial focus map generated by the network. According to previous experiments, small area filtering is necessary to correct the network's misjudgment and further improve the accuracy of the focus area prediction. Therefore, a small area filter with a threshold of threshold = 0.001 is used to further correct the initial decision map to obtain the final decision map D(x, y).

[0110] D(x, y) = filtering(D init (x,y),threshold) (22)

[0111] The final decision graph obtained after the small area filtering operation will be further fused with the source image according to the following formula:

[0112] F(x,y)=D(x,y)I A +(1-D(x,y))I B (twenty three)

[0113] According to the pixel weighted average strategy, the clearer parts of images A and B are selected to obtain the final fused image. Figure 5 shown.

[0114] The network proposed in this embodiment is a supervised network. At present, there is no large-scale public standard training data set in the field of multi-focus image fusion. This embodiment uses Pascal Visual Object Classes 2012 (VOC2012) to construct a training data set, which consists of 20 common object categories: including people, dogs, cars, airplanes, etc. The training set production steps used in this embodiment can be roughly divided into 5 steps: Gaussian filtering, image conversion, image inversion, element-by-element multiplication and element-by-element addition. Specifically, first, the source image and Gaussian filtering with filter kernels of different sizes are used to generate blurred images of different degrees to simulate different defocus effects. In image conversion and image inversion, the mask is binarized and converted into training labels. Then the training labels are globally inverted to obtain complementary labels, which are used to focus the foreground and background in the blurred image respectively. Finally, the blurred images of different degrees generated in the first step are multiplied and added element by element with the training labels to generate multiple image pairs. Figure 6 The process of making the training set is shown.

[0115] In order to further test the performance of the network and its adaptability to data in real scenes, the test data sets selected are "Lytro", "MFFW", "MFI-WHU" and "Grayscale". The Lytro data set is a data set widely used to verify the effect of multi-focus image fusion algorithms. The MFFW data set has different sizes and serious defocus diffusion effect, which further tests the algorithm. "MFI-WHU" is a novel data set that contains many common life scenes. All test images of "Grayscale" are grayscale images, which can test the robustness of the algorithm for grayscale image fusion.

[0116] The network proposed in this embodiment is based on Python language and implemented with PyTroch framework. The training process is accelerated by NVIDIA GeForce RTX 3090. During training, data enhancement is performed by means of random cropping, horizontal flipping, etc. to enhance the robustness of the model and reduce the risk of overfitting. Finally, the quality and reliability of the image are ensured through consistency testing. Thanks to the excellent performance of MSI-DTrans, a relatively ideal fusion effect is achieved in about 37 epochs.

[0117] In order to verify the effectiveness of the method proposed in this embodiment, 10 state-of-the-art (SOTA) multi-focus image fusion methods were selected for comparison. These methods range from early traditional methods to recent novel methods, including transform domain-based, spatial domain-based and deep learning-based methods. These include GFF, MWGF, IFCNN, U2Fusion, SwinFusion, ZMFF, FusionDiff, DRPL, SESF, and MFIF-GAN. These methods basically cover all current categories of MFIF methods. The detailed description of the methods used for comparison is shown in Table 1.

[0118] Table 1

[0119]

[0120] Due to the lack of real labels for the evaluation data set, it is impossible to directly perform statistical analysis on the difference between the fused image generated by the network and the true value. Therefore, in the embodiment, 8 objective evaluation indicators are used to objectively evaluate the performance of all the test methods. Tables 2 and 3 respectively show the average scores of 10 SOTA methods and the method proposed in this embodiment on the data sets "Lytro" and "MFFW" for 8 evaluation indicators. The higher the score of each indicator, the better. Among them, the highest score in each type of evaluation indicator is highlighted by bold, and the second place is highlighted by underline. It can be seen that the proposed method has obtained the first or second score in almost all evaluation indicators on the data set "Lytro", and the score in the evaluation indicator with the second score is not much different from the score of the first place. In the data set "MFFW", the method proposed in this embodiment has declined in the score ranking of the Q_G and Q_CB evaluation indicators, but still maintains the second level, and has achieved considerable scores in the measurement of other indicators. In general, although the method proposed in this embodiment did not maintain the first place in each evaluation index, it achieved good results in the comparison of objective evaluation index scores with the SOTA method and effectively completed the multi-focus image fusion task.

[0121] Table 2

[0122]

[0123] Table 3

[0124]

[0125]

[0126] In order to verify the high generalization of the architecture proposed in this embodiment, this embodiment also selects the public dataset Grayscale to demonstrate the excellent performance of MSI-DTrans when processing both color images and grayscale images. Figure 8 (a)-(x) It can be seen that compared with other SOTA methods, the method proposed in this embodiment not only does not have the problem of regional misjudgment, but also retains the edge details of the stone lion to the highest extent, achieving accurate pixel-level segmentation effect.

[0127] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A multi-focus image fusion method based on multi-layer semantics and multi-scale self-attention, characterized by: include: Get the source image; The source image is input into the MSI-Dtrans model, and Haar wavelet transform is used to extract features of the source image to obtain feature information of different scales, and the feature information of different scales is input into the dynamic Transformer for processing to obtain an initial focus map; wherein the network architecture of the MSI-Dtrans model adopts a multi-level up-down sampling structure of a U-type network, which is obtained through training with a training set, and the training set includes multiple image pairs generated based on blurred images of different degrees and training labels; The initial focus image is fused with the source image to obtain a final fused image.

2. The multi-focus image fusion method based on multi-layer semantics and multi-scale self-attention according to claim 1, characterized in that: Extracting features from the source image using Haar wavelet transform includes: S1. Performing preliminary feature extraction on the source image through a multi-scale semantic extraction module to obtain an intermediate low-frequency feature map and an intermediate high-frequency feature map; S2, inputting the intermediate low-frequency feature map and the intermediate high-frequency feature map into a semantic splitting module, performing high-low frequency splitting on the intermediate low-frequency feature map and the intermediate high-frequency feature map using the Haar wavelet transform, and obtaining a low-frequency sub-band and a high-frequency sub-band; S3, using a rough semantic enhancement module to perform multi-layer semantic interaction on the low-frequency sub-band to obtain low-frequency features; S4, using a fine semantic extraction module to perform multi-layer semantic interaction on the high-frequency sub-band to obtain high-frequency features, wherein the fine semantic extraction module introduces residual connection and Mish activation function; S5. Use the low-frequency features and the high-frequency features as the intermediate low-frequency feature map and the intermediate high-frequency features, return to step S2, repeat to a preset number of times, and output feature information of different scales.

3. The multi-focus image fusion method based on multi-layer semantics and multi-scale self-attention according to claim 2 is characterized in that: The low-frequency sub-band is subjected to multi-layer semantic interaction using a rough semantic enhancement (RSE) module to obtain low-frequency features, including: calculating the difference characteristics of the low-frequency sub-band, splicing the difference features according to the number of channels, extracting information, performing a residual connection between the extracted information and the low-frequency sub-band, and outputting a low-frequency feature map.

4. The multi-focus image fusion method based on multi-layer semantics and multi-scale self-attention according to claim 2 is characterized in that: The calculation method for outputting the low-frequency feature map is: Among them, P and P ’ They represent the feature representations obtained after convolution with kernel sizes of 3 and 1, respectively. h represents parallel path 1 or 2. j represents the relative position of the coarse semantic enhancement module in the overall structure. μ represents the ReLU activation function. * represents the convolution operation. b represents the bias value of the output feature map. D represents the feature representation obtained after difference calculation. F represents the low-frequency feature map. T N×N Represents a convolution operation with a convolution kernel size of N, and X is the input feature map.

5. The multi-focus image fusion method based on multi-layer semantics and multi-scale self-attention according to claim 1, characterized in that: The feature information of different scales is input into the dynamic Transformer for optimization, and the initial focus map is obtained including: Using depthwise separable convolution to replace the fully connected layer in the Transformer model to integrate the feature information of different scales, wherein each of the depthwise separable convolutions is set with a different dilation rate and convolution step size; The integrated feature information of different scales is input into a multi-head attention module to perform multi-scale joint attention calculation to obtain the initial focus map.

6. The multi-focus image fusion method based on multi-layer semantics and multi-scale self-attention according to claim 5, characterized in that: The multi-scale joint attention is calculated as: Multi_scale_joint_attention =Concat(Att1,Att2,...,Att n )W 0 i∈(1,2,3,...,n) Among them, Conv Dia and Conv 1×1 Represents depth-wise separable convolution and point-wise convolution, Dia represents dilated convolution, f i , p i ,s i , d i Respectively represent the corresponding convolution kernel size, edge padding size, convolution step size and dilation convolution rate, Att i represents the attention feature of the i-th feature scale, W is the learnable weight matrix used to generate different Query, Value and Key sequences, and W 0 is the learnable weight matrix used in multi-scale attention calculation, q i , v i , k i are the Query, Value, and Key sequences generated at the i-th scale, respectively. X is the input feature map. Q, V, and K are the Query, Value, and Key sequences generated after the Concat operation. m , V m , K m are the Query, Value, and Key sequences input into the m-th multi-head attention mechanism for calculation, and n is the number of depth-wise separable convolutions with n scales.

7. The multi-focus image fusion method based on multi-layer semantics and multi-scale self-attention according to claim 1, characterized in that: Obtaining the final fused image includes: Obtaining an initial focus map, binarizing the initial focus map, obtaining an initial decision map, performing small area filtering on the initial decision map, and obtaining a final decision map; According to the pixel weighted average strategy, the final decision graph is fused with the source image to obtain the final fused image.

8. The multi-focus image fusion method based on multi-layer semantics and multi-scale self-attention according to claim 1, characterized in that: The method further comprises: simultaneously training the model by back propagation using the loss function; The loss function is: Loss sum =λLoss L1 +μLoss SSIM +σLoss DICE Among them, Loss sum is the joint training loss function of the network, Loss L1 、Loss SSIM and Loss DICE They represent L1 loss function, SSIM loss function and DICE loss function respectively. λ, μ and σ are hyperparameters for balancing the proportions of the three loss functions.

Images