Image fusion feature extraction method, device, electronic device and storage medium

Abstract

The present invention discloses a feature extraction method, device, electronic device and storage medium for image fusion, which are used to solve the technical problem of poor feature extraction effect in existing image fusion methods. The method comprises: firstly, performing mean filtering on the source image of the input image fusion feature extraction module to obtain a mean image; then performing difference calculation between the source image and the mean image to obtain intermediate features; then performing parameter generation processing based on MobileVit‑v3 on the source image to obtain guided filtering parameters; finally, based on the source image and the intermediate features, guided filtering is performed according to the guided filtering parameters to obtain output image features with better feature extraction effect corresponding to the source image.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to a feature extraction method, device, electronic equipment and storage medium for image fusion. Background Art

[0002] Deep learning methods have become a very active research direction in multi-focus image fusion. At present, deep learning models including Convolutional Neural Network (CNN) with residual connections and Generative Adversarial Network (GAN) with unsupervised learning have been successfully applied to the field of multi-focus image fusion.

[0003] The self-attention mechanism is currently receiving widespread attention in related fields. Therefore, more and more studies are adding the self-attention mechanism to the model in the hope of improving the model's effect. Feature extraction technology based on deep learning can achieve good feature extraction results without manually setting parameters, but it requires sufficient hardware support. Due to hardware limitations, the multi-focus image fusion model using the self-attention mechanism often takes about one second to fuse two images. As a traditional feature extraction method, the feature extraction method based on guided filtering is simple to calculate, fast, and has edge-preserving properties.

[0004] When using an end-to-end deep learning method for feature extraction, all images are used as input. When the input image volume is too large, insufficient bandwidth and insufficient memory will occur during device processing. Feature extraction based on guided filtering also has defects. In the feature extraction process of image fusion, it is necessary to manually set the mean filter kernel size, the filter kernel size of the guided filter, and some parameters required in subsequent image processing, such as the area threshold used for small area filtering, so that image fusion can achieve better results. Summary of the invention

[0005] The present invention provides a feature extraction method, device, electronic device and storage medium for image fusion, which are used to solve or partially solve the technical problem of poor feature extraction effect in existing image fusion methods.

[0006] The present invention provides a feature extraction method for image fusion, which is applied to a feature extraction module for image fusion. The method comprises:

[0007] Perform mean filtering on the input source image to obtain a mean image;

[0008] Performing a difference calculation between the source image and the mean image to obtain an intermediate feature;

[0009] Performing parameter generation processing based on MobileVit-v3 on the source image to obtain guided filtering parameters;

[0010] Based on the source image and the intermediate features, guided filtering is performed according to the guided filtering parameters to obtain output image features corresponding to the source image.

[0011] Optionally, the feature extraction module includes a classification head structure and a parameter generation model based on MobileVit-v3, and the parameter generation processing based on MobileVit-v3 is performed on the source image to obtain the guided filtering parameters, including:

[0012] Input the source image into the parameter generation model based on MobileVit-v3 to extract parameter features and output parameter image features;

[0013] Inputting the parameter image features into the classification head structure for feature quantization and converting them into initial guided filtering parameters expressed in numerical form;

[0014] Perform filter kernel correction on the initial guided filter parameters to obtain final guided filter parameters.

[0015] Optionally, the parameter generation model based on MobileVit-v3 includes a MobileVit-v3 module, a feedforward neural network module and an activation module, and the step of inputting the source image into the parameter generation model based on MobileVit-v3 to extract parameter features and outputting parameter image features includes:

[0016] Input the source image into the MobileVit-v3 module to perform a deep separable fusion operation based on an attention mechanism to obtain deep extraction features;

[0017] Inputting the deep layer extracted features into the feedforward neural network module to perform continuous nonlinear transformation feature extraction multiple times to obtain nonlinear image features;

[0018] The nonlinear image features are input into the activation module to complete the multi-classification task and output parameter image features.

[0019] Optionally, the MobileVit-v3 module includes a local representation module, a global representation module and a fusion module, and the source image is input into the MobileVit-v3 module to perform a deep separable fusion operation based on an attention mechanism to obtain deep extraction features, including:

[0020] Input the source image into the local representation module to first perform depth-separable convolution, and then perform 1×1 convolution to obtain depth separation features;

[0021] Inputting the deep separation feature into the global representation module to perform feature extraction based on a separable self-attention mechanism to obtain a linear separation feature;

[0022] The linear separation features are input into the fusion module for feature fusion to obtain deep extraction features.

[0023] Optionally, the local representation module includes a depthwise separable convolution module, the depthwise separable convolution module includes a depthwise convolution submodule and a pointwise convolution submodule, and the depthwise separable convolution step includes:

[0024] Splitting the source image into a plurality of input features based on the number of channels;

[0025] Input the plurality of input features into the deep convolution submodule, first convolve each of the input features using a convolution kernel to obtain a convolved feature, and then perform feature splicing on each of the convolved features to obtain a deep separable convolution feature;

[0026] The depth-wise separable convolution feature is input into the point convolution submodule for 1×1 convolution, and the output feature of the depth-wise separable convolution module is output.

[0027] Optionally, the initial guided filter parameters include a guided filter kernel size and a mean filter kernel size, and the execution process of the filter kernel correction includes:

[0028] The guided filter kernel size and the mean filter kernel size are corrected by the following steps:

[0029] Acquire the original filter kernel size corresponding to the source image, perform rounding-half odd number correction based on the original filter kernel size, and use the filter kernel size obtained after rounding-half odd number correction as the filter kernel size label of the source image;

[0030] Performing rounding-down odd-number correction on the guided filter kernel size to obtain a corrected guided filter kernel size, and using the corrected guided filter kernel size as the final guided filter kernel size;

[0031] The mean filter kernel size is rounded down to an odd number to obtain a corrected mean filter kernel size, and the corrected mean filter kernel size is used as the mean filter kernel size finally used.

[0032] Optionally, performing guided filtering based on the source image and the intermediate features according to the guided filtering parameters to obtain output image features corresponding to the source image includes:

[0033] Using the source image as a guide image for guided filtering and using the intermediate features as an input image for guided filtering;

[0034] According to the guided filtering parameters, the guided image and the input image are used to perform guided filtering processing to obtain output image features corresponding to the source image.

[0035] The present invention also provides a feature extraction device for image fusion, which is applied to a feature extraction module for image fusion. The device comprises:

[0036] A mean filter processing module is used to perform mean filtering on the input source image to obtain a mean image;

[0037] An intermediate feature generation module, used to perform a difference calculation between the source image and the mean image to obtain an intermediate feature;

[0038] A parameter generation processing module, used for performing parameter generation processing based on MobileVit-v3 on the source image to obtain guided filtering parameters;

[0039] The guided filtering module is used to perform guided filtering based on the source image and the intermediate features according to the guided filtering parameters to obtain output image features corresponding to the source image.

[0040] The present invention also provides an electronic device, the device comprising a processor and a memory:

[0041] The memory is used to store program code and transmit the program code to the processor;

[0042] The processor is used to execute the feature extraction method for image fusion as described in any one of the above items according to the instructions in the program code.

[0043] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the feature extraction method for image fusion as described in any one of the above items.

[0044] It can be seen from the above technical solutions that the present invention has the following advantages:

[0045] A feature extraction method based on guided filtering and MobileVit-v3 image fusion is provided. First, the source image of the input image fusion feature extraction module is subjected to mean filtering to obtain a mean image; then, the source image and the mean image are used for difference calculation to obtain intermediate features; then, the source image is subjected to parameter generation processing based on MobileVit-v3 to obtain guided filtering parameters; finally, based on the source image and the intermediate features, guided filtering is performed according to the guided filtering parameters to obtain output image features with better feature extraction effect corresponding to the source image. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 It is a flow chart of the steps of a feature extraction method for image fusion;

[0048] Figure 2 It is a schematic diagram of a feature extraction process based on guided filtering;

[0049] Figure 3 It is a schematic diagram of an improved feature extraction process based on guided filtering;

[0050] Figure 4 This is a schematic diagram of the experimental results of comparing small area filtering thresholds of different data sets;

[0051] Figure 5 A schematic diagram of experimental results comparing regularization parameters of guided filtering for different data sets;

[0052] Figure 6 This is a schematic diagram of the experimental results of comparing filter kernels of guided filtering on different data sets;

[0053] Figure 7 This is a schematic diagram of the experimental results of comparing the mean filter kernels of different data sets;

[0054] Figure 8 This is a schematic diagram of the parameter generation model structure based on MobileVit-v3;

[0055] Fig. 9 It is a schematic diagram of the MobileVit-v3 module structure;

[0056] Fig.10 A schematic diagram of a deep convolution process;

[0057] Fig.11 The structure block diagram of a feature extraction device for image fusion. DETAILED DESCRIPTION

[0058] The embodiments of the present invention provide a feature extraction method, device, electronic device and storage medium for image fusion, which are used to solve or partially solve the technical problem of poor feature extraction effect in existing image fusion methods.

[0059] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] As an example, for feature extraction of multi-focus image fusion, feature extraction technology based on deep learning can achieve good feature extraction effect without manual parameter setting, but it requires sufficient hardware support. Due to hardware limitations, the multi-focus image fusion model using the self-attention mechanism often takes about one second to fuse two images. As a traditional feature extraction method, the feature extraction method based on guided filtering is simple to calculate, fast, and has edge-preserving properties.

[0061] When using an end-to-end deep learning method for feature extraction, all images are used as input. When the input image volume is too large, insufficient bandwidth and insufficient memory will occur during device processing. Feature extraction based on guided filtering also has defects. In the feature extraction process of image fusion, it is necessary to manually set the mean filter kernel size, the filter kernel size of the guided filter, and some parameters required in subsequent image processing, such as the area threshold used for small area filtering, so that image fusion can achieve better results.

[0062] Therefore, one of the core invention points of the embodiment of the present invention is: in view of the deficiencies in the prior art, a parameter generation model for a multi-focus image fusion feature extraction module is proposed, and a method for realizing feature extraction by combining the feature extraction process based on guided filtering in traditional feature extraction and the parameter generation model based on MobileVit-v3 in deep learning is further proposed. First, the source image of the input image fusion feature extraction module is subjected to mean filtering to obtain a mean image; then, the source image and the mean image are used to perform difference calculation to obtain intermediate features; then, the source image is subjected to parameter generation processing based on MobileVit-v3 to obtain guided filtering parameters; finally, based on the source image and the intermediate features, guided filtering is performed according to the guided filtering parameters to obtain output image features with better feature extraction effect corresponding to the source image. Thus, a deep learning network based on MobileVit-v3 is embedded in the feature extraction module based on guided filtering, so that the parameters that need to be manually set in the guided filtering are generated by the method of the deep learning model, so as to solve the problem that the optimal parameters of the guided filtering in different occasions are inconsistent and need to be manually set. In addition, the present invention also obtains guided filtering parameters with better effects by fine-tuning the pre-trained MobileVit-v3 model and adding a classification head structure after the model architecture to output the parameters required by the feature extraction module.

[0063] The technical solution of the present invention is aimed at the fusion scene of multiple multi-focus images with unlimited number. At the same time, it is hoped that the image fusion can be performed as quickly and efficiently as possible. On the one hand, the feature extraction method of the pure deep learning solution needs to extract features from each image. In addition to the computing power required, bandwidth is the biggest limiting factor on devices such as cameras and mobile phones. On the other hand, although the optimal parameters of the traditional feature extraction method are different in different scenes and the parameters need to be changed manually, for the same set of images to be fused, the optimal parameters of the traditional feature extraction method are consistent. Therefore, it is hoped that the deep learning model can be used to extract features from one of the images to be fused, output the parameters required for the ideal traditional feature extraction, and then apply this set of parameters to the traditional feature extraction method to extract features from all images to be fused, so as to solve the bandwidth limitation problem existing in the feature extraction of the current multi-focus image fusion and the problem that the guided filter parameters need to be set manually.

[0064] Next, the embodiment of the present invention will combine the feature extraction based on guided filtering in traditional feature extraction and the parameter generation model based on MobileVit-v3 in deep learning as an image fusion feature extraction module to explain the technical solution provided by the present invention in detail, and at the same time combine relevant experiments to verify the effectiveness of the feature extraction module proposed in the present invention.

[0065] Reference Figure 1, shows a flowchart of a feature extraction method for image fusion provided by an embodiment of the present invention, the method is applied to a feature extraction module for image fusion, and the method may specifically include the following steps:

[0066] Step 101, performing mean filtering on the input source image to obtain a mean image;

[0067] From the perspective of practical application, the embodiment of the present invention takes into account that feature extraction requires emphasis on algorithm speed. Therefore, in feature extraction, a feature extraction method based on guided filtering is mainly adopted.

[0068] Guided filtering is an edge-preserving filtering technique. The input is a guide image and a target image. After guided filtering, the output image will be very similar to the target image, and the output image will have the texture of the guide image. Therefore, when the source image is used as the guide image, it has an edge-preserving effect. Guided filtering has been widely used in the fields of deblurring, denoising, and adding filters.

[0069] In order to enable those skilled in the art to better understand the technical solution of the present invention, refer to Figure 2 , shows a schematic diagram of a feature extraction process based on guided filtering before improvement.

[0070] Combination Figure 2 ,Currently the guided filtering technology and mean filtering technology are mainly ,used to form the feature extraction module based on guided ,filtering.

[0071] The specific process in the feature extraction module is: first perform mean filtering on the source image, then use the source image and the mean image to make a difference to obtain the intermediate features, then use the intermediate features as the guided filtering input image, and use the source image as the guided image of the guided filtering to perform guided filtering to obtain the output features.

[0072] Through further research, the present invention has concluded that for the same set of images to be fused, the optimal parameters of the guided filter are consistent. Therefore, by extracting features from one of the images to be fused with the help of a deep learning model, a relatively ideal guided filter parameter is output, and then the set of parameters is applied to the feature extraction process based on the guided filter to extract features from all the images to be fused.

[0073] The embodiment of the present invention adopts the parameter generation model based on MobileVit-v3 as the deep learning parameter generation model. After the deep learning parameter generation model is improved, the feature extraction process based on guided filtering is obtained as follows: Figure 3 As shown. Among them, Figure 3 The specific feature extraction process provided will be described in detail in the following embodiments.

[0074] Therefore, in the actual processing flow, when it is necessary to extract features from a source image, the input source image can be firstly subjected to mean filtering to obtain a mean image.

[0075] Step 102, performing a difference calculation between the source image and the mean image to obtain an intermediate feature;

[0076] Combined with the feature extraction process based on guided filtering, after obtaining the mean image, the source image and the mean image can be used to perform a difference calculation to obtain an intermediate feature (also called a target image). The intermediate feature can be used as the input image of the guided filtering and can be used together with the guided image for guided filtering.

[0077] Step 103, performing parameter generation processing based on MobileVit-v3 on the source image to obtain guided filtering parameters;

[0078] Before formally introducing the parameter generation process based on MobileVit-v3, the present invention first compares and analyzes the relevant parameters involved in the guided filtering by setting up a comparison experiment to verify the conclusion that the effect of multi-focus image fusion is different under different parameter values. In addition, the comparison experiment can also guide the selection of which parameters to be output as the parameters of the subsequent parameter generation model.

[0079] The feature extraction module of this comparison experiment adopts the mainstream image fusion index normalized mutual information The effects of image fusion are compared.

[0080] Specifically, this comparison experiment mainly uses two types of multi-focus images with different resolutions as experimental data sets. The image resolution of the first type of data set is 1277×718, which is referred to as data set 1 in the chart. The image resolution of the second type of data set is 520×520, which is referred to as data set 2 in the chart.

[0081] The experiment mainly compares four parameters: small area filtering threshold, guided filter regularization parameter, guided filter kernel size (hereinafter referred to as guided filter kernel size) and mean filter kernel size (hereinafter referred to as mean filter kernel size). Among them, small area filtering threshold is a parameter used in the subsequent stage of image fusion. Guided filter regularization parameter, guided filter kernel size and mean filter kernel size are parameters used in the feature extraction module based on guided filtering.

[0082] In the experiment, the default value of the small area filtering threshold is 0.001, the default value of the guided filter regularization parameter is 0.3, the default value of the guided filter kernel size is 7, and the default value of the mean filter kernel size is 7.

[0083] The performance of two different image datasets in the normalized mutual information index under different small area filtering thresholds is shown in the figure. Figure 4 shown.

[0084] Depend on Figure 4 It can be seen that in both the first and second image datasets, as the value of the small area filtering threshold increases, the normalized mutual information first increases and then decreases.

[0085] However, the best small area filtering threshold in the first type of image dataset is not equal to the best small area filtering threshold in the second type of image dataset. This shows that the best small area filtering threshold for multi-focus image fusion is different for different images.

[0086] Therefore, in actual situations, different small area filtering thresholds should be selected in different situations so that guided filtering can play a better role in multi-focus image fusion.

[0087] The performance of the two image datasets in the normalized mutual information index under different sizes of guided filter regularization parameters is shown in the figure. Figure 5 shown.

[0088] Depend on Figure 5 It can be seen that in both the first and second image datasets, as the value of the guided filter regularization parameter increases, the normalized mutual information decreases. Therefore, for multi-focus image fusion, when selecting the guided filter regularization parameter, it is only necessary to select a sufficiently small guided filter regularization parameter.

[0089] Next, we analyze the influence of the filter kernel sizes of the two filters on the fusion effect of multi-focus image fusion.

[0090] The performance of different image datasets in the normalized mutual information index under different guided filter kernel sizes is as follows: Figure 6 shown.

[0091] Depend on Figure 6 It can be seen that there is no particularly obvious similarity between the relationship between the guided filter kernel size in the first type of image data set and the guided filter kernel size in the second type of image data set and the normalized mutual information. At the same time, the value of the guided filter kernel size that performs best in the first type of image data set is not equal to the value of the guided filter kernel size that performs best in the second type of image data set. This shows that when performing multi-focus image fusion on different images, the values ​​of the guided filter kernel size that achieve the best effect are different.

[0092] The performance of different image datasets on the normalized mutual information index under different sizes of mean filter kernels is as follows: Figure 7 shown.

[0093] Depend on Figure 7 It can be seen that in the first type of image data set, the mean filter kernel size and the normalized mutual information are not in a simple linear relationship. There is no particularly obvious similarity between the relationship between the mean filter kernel size in the first type of image data set and the mean filter kernel size in the second type of image data set and the normalized mutual information. At the same time, the value of the best performing mean filter kernel size in the first type of image data set is not equal to the value of the best performing mean filter kernel size in the second type of image data set. This shows that when performing multi-focus image fusion on different images, the value of the mean filter kernel size that achieves the best effect is different.

[0094] Next, the parameter generation process based on MobileVit-v3 will be introduced in detail.

[0095] In the embodiment of the present invention, the parameter generation model based on MobileVit-v3 is used to generate the required guided filtering parameters. The overall model structure is as follows: Figure 8 shown.

[0096] Combination Figure 8 The parameter generation model based on MobileVit-v3 can mainly include input (Input), MobileVit-v3 module, feedforward neural network module (Forward Layer), activation module (Softmax) and output (Output).

[0097] Among them, the internal composition of the feedforward neural network module (Forward Layer) is multiple fully connected layers plus residual connections.

[0098] In combination with the above discussion, the feature extraction module for image fusion provided in the embodiment of the present invention may include a classification head structure ( Figure 3 (not shown) and a parameter generation model based on MobileVit-v3. Thus, the process of performing parameter generation processing based on MobileVit-v3 on the source image to obtain the guided filtering parameters can be implemented by executing the following sub-steps S01 to S03:

[0099] Step S01: input the source image into the parameter generation model based on MobileVit-v3 to extract parameter features and output parameter image features;

[0100] Specifically, step S01 can be implemented by executing the following sub-steps S011 to S013:

[0101] Step S011: input the source image into the MobileVit-v3 module to perform a deep separable fusion operation based on the attention mechanism to obtain deep extraction features;

[0102] In order to solve the problems of too many model parameters in the pure Transformer (a deep learning model for processing sequence data) architecture and too slow inference speed when the model is deployed on mobile devices, Wadekar et al. improved the MobileVit model (a lightweight Transformer model) and obtained the MobileVit-v3 model (i.e., the MobileVit-v3 module used in the embodiment of the present invention). The architecture of the MobileVit-v3 module is as follows: Fig. 9 shown.

[0103] Combination Fig. 9 The MobileVit-v3 module can mainly include input (Input), local representation module (Localrepresentation block), global representation module (Global representation block), fusion block (Fusionblock) and output (Output).

[0104] The local representation module is composed of a convolutional neural network. The convolutional block of the local representation module includes depthwise convolution (DWConv-3×3) and 1×1 convolution (Conv-1×1).

[0105] Depthwise separable convolution can reduce network parameters and improve computational efficiency. Specifically, depthwise separable convolution divides the convolution operation into two steps: depthwise convolution and pointwise convolution.

[0106] The deep convolution splits the input by channel, then uses a convolution kernel to convolve the split features of each channel, and then reconcatenates the features of each channel after the convolution calculation in the channel dimension. The deep convolution process is shown below: Fig.10 shown.

[0107] Point convolution is actually a 1×1 convolution operation, which is mainly used to change the number of output channels to facilitate fusion.

[0108] To highlight the advantages of depthwise separable convolution, the following compares the number of parameters of conventional convolution and depthwise separable convolution. Assume that the dimension of the input tensor is (N, H, W, M). Among them, H represents the height, W represents the width, and M represents the number of output channels.

[0109] If the conventional convolution operation is used, assuming the convolution kernel dimension is The first dimension of the input tensor is N, so there are N dimensions in total. Thus, the number of parameters required for conventional convolution is have:

[0110]

[0111] Where D represents the depth and the subscript k represents the number of convolution kernels.

[0112] If depth-wise separable convolution is used, the convolution kernel size is . So the number of parameters required for depth-wise separable convolution is have:

[0113]

[0114] It can be seen that the use of depth-wise separable convolution can greatly reduce the number of parameters required for the convolutional layer.

[0115] The global representation module consists of multiple Linear Transformers blocks. The core of the Transformer model is the attention mechanism, which enables the network to have the advantage of long-distance dependencies.

[0116] Compared with the self-attention in Transformer, a separable self-attention is proposed in Linear Transformers, which reduces the cost of calculating attention.

[0117] Compared to the regular Transformer block, the formula of the Linear Transformer block is as follows:

[0118]

[0119] Among them, y represents the module output; x represents the module input; , , Both are separable self-attention weight matrices; represents the linear layer weight matrix; here is element-wise multiplication; It is element-by-element accumulation; softmax and ReLU are both activation functions.

[0120] The present invention considers using a deep learning network to set the parameters of the traditional feature extraction module. Therefore, it is necessary to select a suitable network as the main framework of the model. Using MobileVit-v3 as the main model of the parameter generation model of the embodiment of the present invention can not only reduce the number of parameters through deep separable convolution, but also speed up the calculation of the self-attention module through separable linear self-attention.

[0121] Combined with the above introduction, in the specific implementation, the source image is input into the MobileVit-v3 module for deep separable fusion operation based on the attention mechanism to obtain deep extraction features, which can be:

[0122] First, the source image is input into the local representation module for depth-wise separable convolution and then 1×1 convolution to obtain depth-wise separable features. Then, the depth-wise separable features are input into the global representation module for feature extraction based on the separable self-attention mechanism to obtain linear separable features. Finally, the linear separable features are input into the fusion module for feature fusion to obtain deep extraction features.

[0123] Furthermore, the local representation module may include a depthwise separable convolution module, and the depthwise separable convolution module may include a depthwise convolution submodule and a pointwise convolution submodule. The specific implementation steps of the depthwise separable convolution may include:

[0124] First, the source image is split into several input features based on the number of channels; then, several input features are input into the deep convolution submodule, and each input feature is first convolved with a convolution kernel to obtain the post-convolution feature, and then each post-convolution feature is feature spliced ​​to obtain the deep separable convolution feature; finally, the deep separable convolution feature is input into the point convolution submodule for 1×1 convolution, and the output feature of the deep separable convolution module is output.

[0125] In addition, in order to verify the superiority of the MobileVit-v3 model, the embodiment of the present invention also conducted a comparative experiment on the current mainstream lightweight computer vision model for mobile terminals.

[0126] The comparison task is the image classification of the ImageNet1K dataset (the most commonly used subset of the ImageNet dataset). The mainstream and faster mobile visual models MobileNet, MobileVitv1, MobileVitv2 and MobileVit-v3 are compared mainly in terms of accuracy and speed. The comparison experiment results are shown in Table 1 below:

[0127]

[0128] Table 1: Comparison experiment of mainstream mobile visual models

[0129] In Table 1, the MobileVit-v3 model is superior to other comparison models in terms of CPU time consumption, GPU time consumption and accuracy (Acc). Therefore, the embodiment of the present invention selects MobileVit-v3 as the backbone network of the parameter generation model.

[0130] Step S012: inputting the deep layer extracted features into the feedforward neural network module to perform continuous nonlinear transformation feature extraction multiple times to obtain nonlinear image features;

[0131] Step S013: input the nonlinear image features into the activation module to complete the multi-classification task and output parameter image features.

[0132] Step S02: inputting the parameter image features into the classification head structure for feature quantization and converting them into initial guided filtering parameters expressed in numerical form;

[0133] The output of the MobileVit-v3 model is image features, while the present invention needs to output a set of guided filtering parameters. Therefore, a network structure (classification head structure) needs to be added after the MobileVit-v3 model to convert the output of the MobileVit-v3 model into output guided filtering parameters through feature quantization. In the embodiment of the present invention, a multi-layer perceptron is used as the classification head structure.

[0134] Step S03: performing filter kernel correction on the initial guided filter parameters to obtain final guided filter parameters.

[0135] The initial guided filter parameters output after processing by the MobileVit-v3 model and the classification head structure are mainly the small area filter threshold, the guided filter kernel size, and the mean filter kernel size. Among them, in practical applications, the small area filter threshold is a real number, while the guided filter kernel size and the mean filter kernel size should be odd numbers. But in fact, the value output by the deep learning network (that is, the MobileVit-v3 model and the classification head structure) is a real number (that is, it may be an odd number or an even number). Therefore, before actually using the guided filter parameters output by the deep learning network for guided filtering, it is also necessary to process the guided filter kernel size and the mean filter kernel size, that is, the size correction of the filter kernel.

[0136] Assume that the filter kernel size corresponding to the source image (the filter kernel size is an odd number) is ,set up:

[0137]

[0138] use Used as the label of the filter kernel. The original filter kernel size corresponding to the source image is The filter kernel size label of . Since it plays a labeling role, the size label may be an odd number or an even number.

[0139] For the guided filter kernel size or the mean filter kernel size, assume that the corresponding position network output is (that is, the filter kernel size output after processing by the deep learning network), then:

[0140]

[0141] in, This is a round-down operation.

[0142] At this time, the As the filter kernel size. That is, As the filter kernel size of the corresponding position network output. When expressed as the guided filter kernel size, The guided filter kernel size corresponds to the corrected guided filter kernel size output by the position network. When expressed as the mean filter kernel size, The mean filter kernel size corresponds to the corrected mean filter kernel size output by the position network.

[0143] Through the above rounding down odd number correction method, the final guided filter kernel size and mean filter kernel size can be made to be odd numbers.

[0144] In a specific implementation, the parameters that need to be corrected in the initial guided filter parameters include the guided filter kernel size and the mean filter kernel size. In combination with the above content, the guided filter kernel size and the mean filter kernel size can be corrected by executing the following sub-steps S11 to S13:

[0145] Step S11: obtaining the original filter kernel size corresponding to the source image, performing rounding-half odd number correction based on the original filter kernel size, and using the filter kernel size obtained after rounding-half odd number correction as the filter kernel size label of the source image;

[0146] Step S12: performing rounding-down correction on the guided filter kernel size to obtain a corrected guided filter kernel size, and using the corrected guided filter kernel size as the final guided filter kernel size;

[0147] Step S13: rounding down the mean filter kernel size to an odd number to obtain a corrected mean filter kernel size, and using the corrected mean filter kernel size as the final mean filter kernel size.

[0148] In addition, in order to make the disclosure more complete, the embodiments of the present invention also provide additional explanations on the training details of the MobileVit-v3 model.

[0149] The model is trained using fine-tune technology and supervised training methods. The specific steps are as follows:

[0150] First, in terms of the data set, the mutual information between the guided filtered image and the source image under different parameters is calculated for the images in the data set, and the parameter group with the largest mutual information is selected.

[0151] For the same set of images to be fused, the guided filter should use the same set of parameters. The reason is that the guided filter uses the same set of parameters to ensure that the feature comparisons between the same set of images to be fused are of the same standard in terms of quantization (equivalent to normalization). If the same set of images to be fused do not use the same set of parameters during the guided filter process, the features will be misclassified in subsequent operations because they are not normalized.

[0152] Therefore, in the embodiment of the present invention, only one of each group of multifocus images to be fused is taken to calculate different parameters, and then the obtained parameters are used for other images in the same group as a part of the data set.

[0153] Secondly, in terms of the model, since the pre-trained model already has a good ability to extract image features, we choose to freeze the parameters of MobileVit-v3 and randomly initialize the parameters of the Forward Layer.

[0154] Finally, for the optimizer, choose to use the Adam (Adaptive Moment Estimation, an optimization algorithm for training deep learning models) optimizer.

[0155] In terms of hyperparameter setting, the embodiment of the present invention sets the setting parameters of the Adam optimizer , , , and set the learning rate .

[0156] After training, a neural network model that can output guided filtering parameters is obtained, which is the parameter generation model based on MobileVit-v3 adopted in the embodiment of the present invention.

[0157] Step 104: Based on the source image and the intermediate features, guided filtering is performed according to the guided filtering parameters to obtain output image features corresponding to the source image.

[0158] Combined with the previous Figure 3 Based on the source image and the intermediate features, guided filtering is performed according to the guided filtering parameters to obtain the output image features corresponding to the source image. Specifically, the source image is used as the guided image of the guided filtering, and the intermediate features are used as the input image of the guided filtering; according to the guided filtering parameters, the guided image and the input image are used to perform guided filtering processing to obtain the output image features corresponding to the source image.

[0159] In an embodiment of the present invention, a parameter generation model for a multi-focus image fusion feature extraction module is proposed, and further a method for realizing feature extraction by combining a feature extraction process based on guided filtering in traditional feature extraction and a parameter generation model based on MobileVit-v3 in deep learning is proposed. The technical solution of the present invention embeds a deep learning network based on MobileVit-v3 in a feature extraction module based on guided filtering, thereby realizing the generation of parameters that need to be manually set in guided filtering through a deep learning model method, so as to solve the problem that the optimal parameters of guided filtering in different situations are inconsistent and need to be manually set. In addition, the present invention also obtains guided filtering parameters with better effects by fine-tuning the pre-trained MobileVit-v3 model and adding a classification head structure after the model architecture to output the parameters required by the feature extraction module.

[0160] Next, a specific parameter comparison experiment is conducted to better verify the feasibility of the technical solution proposed in the embodiment of the present invention.

[0161] Taking multi-focus image fusion based on guided filtering feature extraction as the actual application scenario, this experiment mainly compares the effects of the parameters obtained using the parameter generation model based on MobileVit-v3 with a set of default parameters. The reason for choosing a set of default parameters is that when performing image fusion, it is impossible to know the optimal parameters in advance, and generally a set of parameters that are acceptable in most cases are set as default parameters.

[0162] This experiment uses two types of multi-focus images with different resolutions as experimental data sets. The image resolution of the first type of data set is 1277×718, and the image resolution of the second type of data set is 520×520. Among them, you can continue to use the data set used in the previous comparison experiment, or you can choose another suitable data set as the experimental data set. Under the default parameters, the small area filtering threshold is set to 0.001, the guided filter regularization parameter size is set to 0.3, the guided filter kernel size is set to 7, and the mean filter kernel size is set to 7.

[0163] Since the parameter generation model based on MobileVit-v3 only outputs the small area filtering threshold, guided filter kernel size, and mean filter kernel size, the guided filter regularization parameter is artificially set to 0.3.

[0164] The comparative experimental results of the first type of image dataset are shown in Table 2 below:

[0165]

[0166] Table 2: Comparative experimental results of parameter settings for the first type of image dataset

[0167] Among them, in the first category of image datasets, the small area filtering threshold output by the parameter generation model based on MobileVit-v3 is 0.22, the guided filter kernel size is 5, and the mean filter kernel size is 7.

[0168] The comparative experimental results of the second type of image dataset are shown in Table 3 below:

[0169]

[0170] Table 3: Comparative experimental results of parameter settings for the second type of image dataset

[0171] Among them, in the second category of image datasets, the small area filtering threshold output by the parameter generation model based on MobileVit-v3 is 0.13, the guided filter kernel size is 5, and the mean filter kernel size is 11.

[0172] It can be seen from Table 2 and Table 3 that the parameters obtained by using the parameter generation model based on MobileVit-v3 perform better than the default parameters in both data sets. It can be concluded that compared with the parameters set manually without prior information, the parameters obtained by using the parameter generation model based on MobileVit-v3 provided by the embodiment of the present invention can achieve better results in multi-focus image fusion.

[0173] Therefore, in the embodiment of the present invention, the current traditional feature extraction method is firstly experimented, and it is observed that the parameters of the traditional feature extraction method have an impact on the fusion effect. To address this problem, a feature extraction method based on guided filtering combined with MobileVit-v3 is proposed. This method uses a feature extraction method based on guided filtering for feature extraction, and uses a parameter generation model based on MobileVit-v3 to generate relevant parameters of a feature extraction method based on guided filtering. By adopting this method, better results than the default parameters are achieved in both the first type of image data set comparison experiment and the second type of image data set comparison experiment. As a result, feature extraction can achieve better results with fast speed and less bandwidth, which has certain practical value.

[0174] Reference Fig.11 , shows a structural block diagram of a feature extraction device for image fusion provided by an embodiment of the present invention, a feature extraction module applied to image fusion, and the device may specifically include:

[0175] The mean filter processing module 1101 is used to perform mean filtering on the input source image to obtain a mean image;

[0176] An intermediate feature generation module 1102 is used to perform a difference calculation between the source image and the mean image to obtain an intermediate feature;

[0177] The parameter generation processing module 1103 is used to perform parameter generation processing based on MobileVit-v3 on the source image to obtain guided filtering parameters;

[0178] The guided filtering module 1104 is used to perform guided filtering based on the source image and the intermediate features according to the guided filtering parameters to obtain output image features corresponding to the source image.

[0179] In an optional embodiment, the feature extraction module includes a classification head structure and a parameter generation model based on MobileVit-v3, and the parameter generation processing module 1103 includes:

[0180] A parameter feature extraction module, used for inputting the source image into the parameter generation model based on MobileVit-v3 to extract parameter features and output parameter image features;

[0181] A feature quantization module, used for inputting the parameter image features into the classification head structure for feature quantization, and converting them into initial guided filtering parameters expressed in numerical form;

[0182] The filter kernel correction module is used to perform filter kernel correction on the initial guided filter parameters to obtain final guided filter parameters.

[0183] In an optional embodiment, the parameter generation model based on MobileVit-v3 includes a MobileVit-v3 module, a feedforward neural network module and an activation module, and the parameter feature extraction module includes:

[0184] A deep separable fusion module, used for inputting the source image into the MobileVit-v3 module to perform a deep separable fusion operation based on an attention mechanism to obtain deep extraction features;

[0185] A nonlinear transformation feature extraction module, used for inputting the deep layer extraction features into the feedforward neural network module to perform continuous nonlinear transformation feature extraction multiple times to obtain nonlinear image features;

[0186] The multi-classification task module is used to input the nonlinear image features into the activation module to complete the multi-classification task and output parameter image features.

[0187] In an optional embodiment, the MobileVit-v3 module includes a local representation module, a global representation module and a fusion module, and the deep separable fusion module includes:

[0188] A depth separation feature generation module, used for inputting the source image into the local representation module to first perform depth separable convolution and then perform 1×1 convolution to obtain depth separation features;

[0189] A linear separable feature generation module, used for inputting the deep separation feature into the global representation module to perform feature extraction based on a separable self-attention mechanism to obtain a linear separable feature;

[0190] The deep extraction feature generation module is used to input the linear separation feature into the fusion module for feature fusion to obtain the deep extraction feature.

[0191] In an optional embodiment, the local representation module includes a depth-separable convolution module, the depth-separable convolution module includes a depth convolution submodule and a point convolution submodule, and the depth separation feature generation module includes:

[0192] A feature splitting module, used for splitting the source image into a plurality of input features based on the number of channels;

[0193] A depth-separable convolution feature generation module, used to input the plurality of input features into the depth-convolution submodule, first convolve each of the input features using a convolution kernel to obtain a convolution feature, and then perform feature splicing on each of the convolution features to obtain a depth-separable convolution feature;

[0194] A point convolution processing module is used to input the depth-separable convolution feature into the point convolution submodule for 1×1 convolution, and output the output feature of the depth-separable convolution module.

[0195] In an optional embodiment, the initial guided filter parameters include a guided filter kernel size and a mean filter kernel size, and the filter kernel correction module includes:

[0196] A round-half odd number correction module is used to obtain the original filter kernel size corresponding to the source image, perform round-half odd number correction based on the original filter kernel size, and use the filter kernel size obtained after the round-half odd number correction as the filter kernel size label of the source image;

[0197] A guided filter kernel size correction module, used for rounding down the guided filter kernel size to an odd number to obtain a corrected guided filter kernel size, and using the corrected guided filter kernel size as the final guided filter kernel size;

[0198] The mean filter kernel size correction module is used to perform rounding-down odd-number correction on the mean filter kernel size to obtain a corrected mean filter kernel size, and use the corrected mean filter kernel size as the final mean filter kernel size.

[0199] In an optional embodiment, the guided filtering module 1104 is specifically used for:

[0200] Using the source image as a guide image for guided filtering and using the intermediate features as an input image for guided filtering;

[0201] According to the guided filtering parameters, the guided image and the input image are used to perform guided filtering processing to obtain output image features corresponding to the source image.

[0202] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the aforementioned method embodiment.

[0203] An embodiment of the present invention further provides an electronic device, the device comprising a processor and a memory:

[0204] The memory is used to store the program code and transmit the program code to the processor;

[0205] The processor is used to execute the feature extraction method of image fusion in any embodiment of the present invention according to the instructions in the program code.

[0206] An embodiment of the present invention further provides a computer-readable storage medium, which is used to store program codes, and the program codes are used to execute the feature extraction method for image fusion of any embodiment of the present invention.

[0207] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0208] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0209] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0210] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0211] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0212] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A feature extraction method for image fusion, characterized in that: A feature extraction module applied to image fusion, the method comprising: Perform mean filtering on the input source image to obtain a mean image; Performing a difference calculation between the source image and the mean image to obtain an intermediate feature; Performing parameter generation processing based on MobileVit-v3 on the source image to obtain guided filtering parameters; Based on the source image and the intermediate features, performing guided filtering according to the guided filtering parameters to obtain output image features corresponding to the source image; The feature extraction module includes a classification head structure and a parameter generation model based on MobileVit-v3. The parameter generation process based on MobileVit-v3 is performed on the source image to obtain the guided filtering parameters, including: Input the source image into the parameter generation model based on MobileVit-v3 to extract parameter features and output parameter image features; Inputting the parameter image features into the classification head structure for feature quantization and converting them into initial guided filtering parameters expressed in numerical form; Performing filter kernel correction on the initial guided filter parameters to obtain final guided filter parameters; The step of performing guided filtering based on the source image and the intermediate features according to the guided filtering parameters to obtain output image features corresponding to the source image includes: Using the source image as a guide image for guided filtering and using the intermediate features as an input image for guided filtering; According to the guided filtering parameters, the guided image and the input image are used to perform guided filtering processing to obtain output image features corresponding to the source image.

2. The feature extraction method for image fusion according to claim 1, characterized in that: The parameter generation model based on MobileVit-v3 includes a MobileVit-v3 module, a feedforward neural network module and an activation module. The source image is input into the parameter generation model based on MobileVit-v3 to extract parameter features and output parameter image features, including: Input the source image into the MobileVit-v3 module to perform a deep separable fusion operation based on an attention mechanism to obtain deep extraction features; Inputting the deep layer extracted features into the feedforward neural network module to perform continuous nonlinear transformation feature extraction multiple times to obtain nonlinear image features; The nonlinear image features are input into the activation module to complete the multi-classification task and output parameter image features.

3. The feature extraction method for image fusion according to claim 2, characterized in that: The MobileVit-v3 module includes a local representation module, a global representation module and a fusion module. The source image is input into the MobileVit-v3 module to perform a deep separable fusion operation based on an attention mechanism to obtain deep extraction features, including: Input the source image into the local representation module to first perform depth-separable convolution, and then perform 1×1 convolution to obtain depth separation features; Inputting the deep separation feature into the global representation module to perform feature extraction based on a separable self-attention mechanism to obtain a linear separation feature; The linear separation features are input into the fusion module for feature fusion to obtain deep extraction features.

4. The feature extraction method for image fusion according to claim 3, characterized in that: The local representation module includes a depth-separable convolution module, the depth-separable convolution module includes a depth convolution submodule and a point convolution submodule, and the depth-separable convolution step includes: Splitting the source image into a plurality of input features based on the number of channels; Input the plurality of input features into the deep convolution submodule, first convolve each of the input features using a convolution kernel to obtain a convolved feature, and then perform feature splicing on each of the convolved features to obtain a deep separable convolution feature; The depth-wise separable convolution feature is input into the point convolution submodule for 1×1 convolution, and the output feature of the depth-wise separable convolution module is output.

5. The feature extraction method for image fusion according to claim 1, characterized in that: The initial guided filter parameters include a guided filter kernel size and a mean filter kernel size. The execution process of the filter kernel correction includes: The guided filter kernel size and the mean filter kernel size are corrected by the following steps: Acquire the original filter kernel size corresponding to the source image, perform rounding-half odd number correction based on the original filter kernel size, and use the filter kernel size obtained after rounding-half odd number correction as the filter kernel size label of the source image; Performing rounding-down odd-number correction on the guided filter kernel size to obtain a corrected guided filter kernel size, and using the corrected guided filter kernel size as the final guided filter kernel size; The mean filter kernel size is rounded down to an odd number to obtain a corrected mean filter kernel size, and the corrected mean filter kernel size is used as the mean filter kernel size finally used.

6. A feature extraction device for image fusion, characterized in that: A feature extraction module applied to image fusion, the device comprising: A mean filter processing module is used to perform mean filtering on the input source image to obtain a mean image; An intermediate feature generation module, used to perform a difference calculation between the source image and the mean image to obtain an intermediate feature; A parameter generation processing module, used for performing parameter generation processing based on MobileVit-v3 on the source image to obtain guided filtering parameters; A guided filtering module, configured to perform guided filtering based on the source image and the intermediate features according to the guided filtering parameters to obtain output image features corresponding to the source image; The feature extraction module includes a classification head structure and a parameter generation model based on MobileVit-v3, and the parameter generation processing module includes: A parameter feature extraction module, used for inputting the source image into the parameter generation model based on MobileVit-v3 to extract parameter features and output parameter image features; A feature quantization module, used for inputting the parameter image features into the classification head structure for feature quantization, and converting them into initial guided filtering parameters expressed in numerical form; A filter kernel correction module, used for performing filter kernel correction on the initial guided filter parameters to obtain final guided filter parameters; The guided filtering module is specifically used for: Using the source image as a guide image for guided filtering and using the intermediate features as an input image for guided filtering; According to the guided filtering parameters, the guided image and the input image are used to perform guided filtering processing to obtain output image features corresponding to the source image.

7. An electronic device, characterized in that: The device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the feature extraction method for image fusion according to any one of claims 1 to 5 according to the instructions in the program code.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program codes, and the program codes are used to execute the feature extraction method for image fusion according to any one of claims 1 to 5.

Images