A method and device for integrated multi-source image fusion
By extracting and merging image features, combining sigmoid projection and Gaussian filtering processing, the fusion image is generated, and the problem of defocus effect in multi-source image fusion is solved, the clarity and accuracy of image fusion are improved, and the situation of severe DSE is improved.
Patent Information
- Application Number
- CN202411189603.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-08-28
AI Technical Summary
Existing multi-source image fusion algorithms fail to effectively solve the defocus effect (DSE), resulting in blurred foreground objects that may exceed the boundary, confusing focus detection, and producing artifacts or distortion.
An integrated multi-source image fusion method is proposed. By obtaining the training multi-focus image set, the feature extraction module is used to extract features, merge feature maps, and the initial decision map is generated through sigmoid projection and Gaussian filtering processing, combining the smooth boundary map, and point multiplication is performed. Sum weighted sum, generate fusion images, and optimize the network through L1 loss and L2 loss.
Effectively mitigate the impact of DSE, improve the clarity and accuracy of image fusion, significantly improve the situation of severe DSE, and enhance the boundary definition of images.
Smart Images

Figure CN118711032B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for integrated multi-source image fusion. Background Art
[0002] Currently, many algorithms, except for algorithms such as MFF_Net, MFF_SSIM, and MFIF_GAN, do not consider the defocus spread effect (DSE). In essence, DSE refers to the phenomenon that unfocused foreground objects tend to expand, resulting in an unclear FDB curve. In particular, when the foreground is clear, blurred background objects will not have an impact on the clear foreground area, resulting in a clear FDB. On the other hand, when the foreground is out of focus, the blurred foreground will seep into the background. Therefore, the FDB is not a clear line, and the blurred foreground objects may extend beyond the boundary itself. Therefore, DSE poses two challenges to the MFIF task. First, there is always an unclear area near the FDB, no matter whether it comes from the focused foreground image or the background image. Second, unfocused and enlarged objects confuse focus detection, resulting in artifacts or distortion in the fusion output.
[0003] In summary, it can be seen that there is an urgent need to provide a multi-source image fusion method that can solve the DSE problem, thereby reducing the impact of different degrees of DSE on the multi-source image fusion method. Summary of the invention
[0004] Based on this, the present invention proposes a method and device for integrated multi-source image fusion to solve the technical defects of the prior art.
[0005] In a first aspect, the present invention provides a method for integrated multi-source image fusion, the method comprising:
[0006] Acquire a training multi-focus image set, wherein the training multi-focus image set includes two source images, namely a foreground focused source image A and a background focused source image B;
[0007] Training the network based on the training multi-focus image set to obtain an integrated multi-source image fusion model;
[0008] Inputting the actual multi-focus image set into the integrated multi-source image fusion model to generate a fused image;
[0009] The network is trained based on the training multi-focus image set to obtain an integrated multi-source image fusion model, specifically:
[0010] Step S211, inputting the source image A and the source image B into a feature extraction module respectively for feature extraction to obtain extracted features A and extracted features B;
[0011] Step S212, the extracted feature A and the extracted feature B are combined by 1×1 convolution to obtain a combined feature map;
[0012] Step S213, the merged feature map is processed by sigmoid projection to obtain an initial decision map of the process;
[0013] Step S214, calculating the L between the initial decision diagram of the process and the reference true value 1 loss loss value, if L 1 loss value meets the requirement, then the initial decision diagram of the process is the initial decision diagram; if L 1 If the loss value does not meet the requirement, repeat steps S211-S213 until L 1 The loss value meets the requirement, and the initial decision diagram of the process at this time is the initial decision diagram;
[0014] Step S215, after processing the initial decision graph using a Gaussian filter, threshold processing is performed on the initial decision graph to obtain two boundary graphs with different thicknesses;
[0015] Step S216, using a guided filter to smooth the boundary of the boundary map having a smooth boundary area in the two boundary maps with different thicknesses to obtain a smooth boundary map;
[0016] Step S217, combining the smoothed boundary graph with the initial decision graph to obtain a final decision graph;
[0017] Step S218, performing point multiplication and weighted sum operation on the final decision graph, the source image A and the source image B to obtain a fused image;
[0018] Step S219, respectively calculating the L of the fused image and the source image A and the source image B 2 loss loss value, if L 2 loss value meets the requirement, then the network is an integrated multi-source image fusion model; if L 2 If the loss value does not meet the requirement, repeat steps S211-S218 until L 2 loss value meets the requirements, and the network is now an integrated multi-source image fusion model.
[0019] In a possible implementation, the feature extraction module is two dual-channel conjoined structures with the same structure and weight, and each channel of the feature extraction module includes a downsampling module, a ViT module, an upsampling module and a skip-layer connection;
[0020] The downsampling module includes a first downsampling ResNeSt convolution block, a second downsampling ResNeSt convolution block, a third downsampling ResNeSt convolution block and twelve Transformer layers connected in sequence;
[0021] The ViT module includes a patch embedding module, n ViT layers and 1 1×1 convolution;
[0022] The upsampling module includes a first upsampling ResNeSt convolution block, a second upsampling ResNeSt convolution block, a third upsampling ResNeSt convolution block and a fourth upsampling ResNeSt convolution block which are connected in sequence.
[0023] In a possible implementation, each of the down-sampling ResNeSt convolution blocks includes k feature map groups arranged in parallel, the output features of the k feature map groups are connected, a 1×1 convolution is used to restore the original feature map size, and then the entire down-sampling ResNeSt convolution block is connected in a residual manner;
[0024] Each of the feature map groups includes r segmentation convolution paths and a segmentation attention mechanism, the r segmentation convolution paths are set in parallel, and the output features of the r segmentation convolution paths are all input into the segmentation attention mechanism;
[0025] The segmentation convolution path consists of a 1×1 convolutional layer and a 3×3 convolutional layer connected in sequence.
[0026] In a possible implementation, each of the ViT layers consists of h The multi-head self-attention and multi-layer perceptron modules are composed of h Output features of multi-head self-attention and multi-layer perceptron modules for:
[0027] ,
[0028] ,
[0029] in, For the h - output features of 1 multi-head self-attention and multi-layer perceptron module, LN is the layer normalization operator, MSA For long positions, MLP is a multi-layer perceptron, is an intermediate variable.
[0030] In a possible implementation, step S211 is specifically as follows:
[0031] Inputting the source image A and the source image B into the first downsampling ResNeSt convolution block for downsampling respectively, to obtain a first deep local feature A and a first deep local feature B;
[0032] Inputting the first deep local feature A and the first deep local feature B into the second downsampling ResNeSt convolution block for downsampling respectively, to obtain the second deep local feature A and the second deep local feature B;
[0033] Inputting the second deep local feature A and the second deep local feature B into the third downsampling ResNeSt convolution block for downsampling respectively, to obtain the third deep local feature A and the third deep local feature B;
[0034] Inputting the third deep local feature A and the third deep local feature B into twelve Transformer layers for downsampling respectively, to obtain a fourth deep local feature A and a fourth deep local feature B;
[0035] Inputting the fourth deep local feature A and the fourth deep local feature B into the ViT module respectively for global feature extraction to obtain global feature A and global feature B;
[0036] Inputting the global feature A and the global feature B into the first upsampling ResNeSt convolution block for upsampling respectively, to obtain the first upsampling feature A and the first upsampling feature B;
[0037] The first up-sampled feature A and the first up-sampled feature B are respectively fused with the third deep local feature A and the third deep local feature B, and the fused features are respectively input into the second up-sampled ResNeSt convolution block to obtain the second up-sampled feature A and the second up-sampled feature B;
[0038] The second up-sampled feature A and the second up-sampled feature B are respectively fused with the second deep local feature A and the second deep local feature B, and the fused features are respectively input into the third up-sampled ResNeSt convolution block to obtain the third up-sampled feature A and the third up-sampled feature B;
[0039] The third up-sampled feature A and the third up-sampled feature B are fused with the first deep local feature A and the first deep local feature B respectively, and the fused features are input into the fourth up-sampled ResNeSt convolution block respectively to obtain the extracted feature A and the extracted feature B respectively.
[0040] In a second aspect, the present invention provides a device for integrated multi-source image fusion, the device comprising:
[0041] An image acquisition module, used to acquire a training multi-focus image set, wherein the training multi-focus image set includes a foreground focused source image A and a background focused source image B;
[0042] A model training module, used for training the network based on the training multi-focus image set to obtain an integrated multi-source image fusion model;
[0043] A fused image generation module, used for inputting an actual multi-focus image set into the integrated multi-source image fusion model to generate a fused image;
[0044] Wherein, the model training module includes:
[0045] A feature extraction module, used for performing feature extraction on the source image A and the source image B to obtain extracted feature A and extracted feature B;
[0046] A feature merging module, used for merging the extracted feature A and the extracted feature B to obtain a merged feature map;
[0047] An initial decision graph acquisition module is used to perform sigmoid projection processing on the merged feature graph to obtain an initial decision graph of the process;
[0048] The first error calculation module is used to calculate the L between the initial decision diagram of the process and the reference true value 1 loss loss value, based on L 1 The network is repeatedly trained with loss value, and the network parameters are continuously optimized until a qualified initial decision graph is obtained;
[0049] A boundary map acquisition module, used for processing the initial decision map by using a Gaussian filter, and then performing threshold processing on the initial decision map to obtain two boundary maps with different thicknesses;
[0050] A smoothed boundary map acquisition module, used for smoothing the boundary of a boundary map having a smooth boundary region in two boundary maps with different thicknesses by using a guided filter to obtain a smoothed boundary map;
[0051] A final decision graph acquisition module, used for combining the smoothed boundary graph with the initial decision graph to obtain a final decision graph;
[0052] A fused image acquisition module, used for performing point multiplication and weighted sum operation on the final decision graph, the source image A and the source image B to obtain a fused image;
[0053] The second error calculation module is used to calculate the L error between the fused image and the source image A and the source image B respectively. 2 loss loss value, based on L 2The network is repeatedly trained based on the loss value, and the network parameters are continuously optimized until an integrated multi-source image fusion model is obtained.
[0054] Generally speaking, multi-focus image fusion can be regarded as a binary classification problem. The U-type network used in image segmentation is introduced into the multi-focus image classification problem. Considering that the U-type network may encounter blurred boundaries or classification errors during the classification process, and the pooling operation may cause resolution reduction and information loss. Therefore, the feature map attention and multi-path representation of ResNeSt can effectively make up for the shortcomings of the U-type network. It is worth noting that although the CNN-based network can effectively extract local information from the feature map, it ignores the global information in the feature map. In contrast, the Transformer attention mechanism shows significant advantages in extracting global features and complements CNN. Based on these observations, the present invention proposes a method and device for integrated multi-source image fusion. Compared with the prior art, the present application achieves the following technical effects:
[0055] (1) By replacing the traditional convolutional blocks of the U-type network with ResNeSt, the proposed network can extract and retain the deep local feature information in the source image while improving the classification performance.
[0056] (2) The present invention introduces a ViT module between the downsampling process and the upsampling process of the feature extraction model to extract global features from the downsampling output feature map and input it into the upsampling branch.
[0057] (3) In order to alleviate different degrees of DSE, this paper adopts a method of slightly enlarging the foreground to solve the DSE problem for mild DSE. For severe DSE, a new solution is proposed to enhance the FDB in the image by applying sharpening, thereby significantly improving severe DSE. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0059] Figure 1 A schematic diagram of a flow chart of a method for integrated multi-source image fusion provided by an embodiment of the present invention;
[0060] Figure 2 A structural block diagram of a device for integrated multi-source image fusion provided by an embodiment of the present invention;
[0061] Figure 3A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0062] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0063] It should be clear that the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0064] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the claims are also intended to include plural forms, unless the context clearly indicates other meanings.
[0065] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0066] See also Figure 1 , is a flow chart of a method for integrated multi-source image fusion provided by an embodiment of the present invention. Figure 1 As shown, the method includes:
[0067] Step S1, obtaining a training multi-focus image set, wherein the training multi-focus image set includes two source images, namely a foreground focused source image A and a background focused source image B;
[0068] Step S2, training the network based on the training multi-focus image set to obtain an integrated multi-source image fusion model;
[0069] Step S3: inputting the actual multi-focus image set into the integrated multi-source image fusion model to generate a fused image.
[0070] Furthermore, the actual multi-focus image set includes an actual source image A focused on the foreground and an actual source image B focused on the background.
[0071] Furthermore, a technical solution of step S2 is specifically as follows:
[0072] Step S211, inputting the source image A and the source image B into a feature extraction module respectively for feature extraction to obtain extracted features A and extracted features B;
[0073] Step S212, the extracted feature A and the extracted feature B are combined by 1×1 convolution to obtain a combined feature map;
[0074] Step S213, the merged feature map is processed by sigmoid projection to obtain an initial decision map of the process;
[0075] It should be noted that the merged feature map is non-binary. In order to generate a binary initial decision map, a sigmoid projection needs to be added to convert the merged feature map into a binary initial decision map. The sigmoid projection can be expressed as follows:
[0076] ,
[0077] in, y represents the initial decision graph, x represents the initial decision graph, c Indicates the steepness of the control curve and its closeness to the original Heaviside function. c Meaning a closer approach.
[0078] Step S214, calculating the L between the initial decision diagram of the process and the reference true value 1 loss loss value, if L 1 loss value meets the requirement, then the initial decision diagram of the process is the initial decision diagram; if L 1 If the loss value does not meet the requirement, repeat steps S211-S213 until L 1 The loss value meets the requirement, and the initial decision diagram of the process at this time is the initial decision diagram;
[0079] The present invention discloses an integrated multi-source image fusion method, which is an unsupervised learning method. The network is optimized by designing a loss function, and a complex data relationship is established between the source image and the fused image through a large amount of training data and the constraints of the loss function. By minimizing the loss function, the method is prompted to continuously optimize the network parameters, so that the network generates a decision diagram close to the true value, and the fused image contains more effective information in the source image.
[0080] Step S215, after processing the initial decision graph by using a Gaussian filter, threshold processing is performed on the initial decision graph to obtain two boundary graphs with different thicknesses, so as to achieve seamless fusion of boundaries and reduce DSE;
[0081] Step S216, using a guided filter to smooth the boundary of the boundary map having a smooth boundary area in the two boundary maps with different thicknesses to obtain a smooth boundary map;
[0082] Step S217, combining the smoothed boundary graph with the initial decision graph to obtain a final decision graph;
[0083] That is, the final decision graph is characterized by the boundary graph portion with smooth boundaries in its FDB region, while the rest of the final decision graph consists of the initial decision graph.
[0084] Step S218, performing point multiplication and weighted sum operation on the final decision graph, the source image A and the source image B to obtain a fused image;
[0085] The specific calculation formula for the dot product and weighted sum operation is:
[0086] ,
[0087] in, F represents the fused image, I A represents the source image A, DM represents the final decision graph, I B represents the source image B, Represents pixel-by-pixel multiplication.
[0088] Step S219, respectively calculating the L of the fused image and the source image A and the source image B 2 loss loss value, if L 2 loss value meets the requirement, then the network is an integrated multi-source image fusion model; if L 2 If the loss value does not meet the requirement, repeat steps S211-S218 until L 2 loss value meets the requirements, and the network is now an integrated multi-source image fusion model.
[0089] On the basis of the above technical solution, for a multi-focus image greatly affected by DSE, another technical solution of step S2 is to replace steps S218-S219 with the following steps on the basis of keeping steps S211-S217 unchanged in the above technical solution:
[0090] Step S2110, performing image multiplication operation on the boundary map with relatively uneven boundary areas in the two boundary maps with different thicknesses and the source image A to obtain an intermediate image;
[0091] Step S2111, performing a sharpening operation on the intermediate image to obtain a sharpened image, so as to improve the fusion quality of FDB in the output result;
[0092] Step S2112, performing a dot multiplication and a weighted sum operation on the final decision graph, the source image A, the source image B and the sharpened image to obtain a second fused image;
[0093] Step S2113, respectively calculating the L of the second fused image and the source image A and the source image B 2 loss loss value, if L 2 loss value meets the requirement, then the network is an integrated multi-source image fusion model; if L 2 If the loss value does not meet the requirement, repeat steps S211-S217, S2110-S2112 until L 2 If the loss value meets the requirements, the network at this time is an integrated multi-source image fusion model.
[0094] Furthermore, the feature extraction module is a dual-channel conjoined structure with the same structure and weights. The source image A and the source image B are respectively input into the two channels of the feature extraction module for feature extraction to obtain extracted features A and extracted features B respectively.
[0095] Each channel of the feature extraction module includes a downsampling module, a ViT module, an upsampling module and a skip-layer connection.
[0096] The downsampling module is used to extract deep local features at different resolutions by downsampling the source image layer by layer, and generate four deep local feature maps with different resolutions.
[0097] The downsampling module includes a first downsampling ResNeSt convolution block, a second downsampling ResNeSt convolution block, a third downsampling ResNeSt convolution block and twelve Transformer layers connected in sequence.
[0098] Furthermore, each of the down-sampling ResNeSt convolution blocks includes k feature map groups arranged in parallel, the output features of the k feature map groups are connected, and a 1×1 convolution is used to restore the original feature map size, and then the entire down-sampling ResNeSt convolution block is connected in a residual manner.
[0099] Furthermore, each of the feature map groups includes r segmentation convolution paths and a segmentation attention mechanism. The r segmentation convolution paths are arranged in parallel, and the output features of the r segmentation convolution paths are all input into the segmentation attention mechanism.
[0100] Furthermore, the segmentation convolution path is composed of a 1×1 convolutional layer and a 3×3 convolutional layer connected in sequence. In this embodiment, in order to maintain the size of the input feature map, the stride and padding of the 1×1 convolutional layer are set to 1 and 0 respectively, and the stride and padding of the 3×3 convolutional layer are both set to 1.
[0101] Preferably, in the present embodiment, each of the down-sampling ResNeSt convolution blocks consists of 16 feature map groups, each of which includes 32 segmentation convolution paths and one segmentation attention mechanism.
[0102] The ViT module includes a patch embedding module, n ViT layers and 1 1×1 convolution.
[0103] In order to input the feature map into the ViT layer to extract its global features, it is necessary to first use the patch embedding module to serialize and patch embed the feature map to match the input of the ViT layer. The specific process is as follows:
[0104] Given a feature map F M The spatial resolution is L × W , L Indicates length, W represents the width, and the number of channels is C , the serialization is to F M Reshape into a series of flattened 2D blocks ,in, i is the sequence number of the flattened two-dimensional block, N P is the total number of flattened two-dimensional blocks, and then a series of flattened two-dimensional blocks are linearly mapped to the potential E-dimensional embedding space; a specific position embedding is introduced and added to the patch embedding module to retain the position information to encode the spatial information of the flattened two-dimensional blocks. The output feature of the patch embedding module z 0 for:
[0105] ,
[0106] in, is a flattened two-dimensional block, i is the sequence number of the flattened two-dimensional block, N P is the total number of flattened 2D blocks, D For block embedding projection, D pos Embed for position.
[0107] Each of the ViT layers consists of h The first one is composed of a multi-head self-attention (MSA) and a multi-layer perceptron (MLP) module. h Output features of multi-head self-attention and multi-layer perceptron modules for:
[0108] ,
[0109] ,
[0110] in, For the h - output features of 1 multi-head self-attention and multi-layer perceptron module, LN is the layer normalization operator, MSA For long positions, MLP is a multi-layer perceptron, is an intermediate variable.
[0111] The 1×1 convolution is used to transform the n Output features of ViT layers Restore to original size, that is, output features The size of Reshape .
[0112] Preferably, in this embodiment, each of the ViT layers is composed of 4 multi-head self-attention and multi-layer perceptron modules.
[0113] The upsampling module is used to upsample the output features of the ViT module layer by layer, and in the upsampling process, the fusion with the downsampled features of the same resolution is realized through skip-layer connection to restore the resolution of the feature map to L × W ,in L Indicates length, W Indicates width.
[0114] The upsampling module includes a first upsampling ResNeSt convolution block, a second upsampling ResNeSt convolution block, a third upsampling ResNeSt convolution block and a fourth upsampling ResNeSt convolution block which are connected in sequence.
[0115] The skip-layer connection is used to fuse the outputs of the down-sampling module and the up-sampling module with the same resolution to form a U-shaped structure, so as to improve the classification accuracy of focused and out-of-focus pixels.
[0116] On the above basis, the source image A and the source image B are respectively input into the feature extraction module to extract the deep local features of the source image A and the source image B, and the extracted features A and the extracted features B are obtained, specifically:
[0117] Inputting the source image A and the source image B into the first downsampling ResNeSt convolution block for downsampling respectively, to obtain a first deep local feature A and a first deep local feature B;
[0118] Inputting the first deep local feature A and the first deep local feature B into the second downsampling ResNeSt convolution block for downsampling respectively, to obtain the second deep local feature A and the second deep local feature B;
[0119] Inputting the second deep local feature A and the second deep local feature B into the third downsampling ResNeSt convolution block for downsampling respectively, to obtain the third deep local feature A and the third deep local feature B;
[0120] Inputting the third deep local feature A and the third deep local feature B into twelve Transformer layers for downsampling respectively, to obtain a fourth deep local feature A and a fourth deep local feature B;
[0121] Inputting the fourth deep local feature A and the fourth deep local feature B into the ViT module respectively for global feature extraction to obtain global feature A and global feature B;
[0122] Inputting the global feature A and the global feature B into the first upsampling ResNeSt convolution block for upsampling respectively, to obtain the first upsampling feature A and the first upsampling feature B;
[0123] The first up-sampled feature A and the first up-sampled feature B are respectively fused with the third deep local feature A and the third deep local feature B, and the fused features are respectively input into the second up-sampled ResNeSt convolution block to obtain the second up-sampled feature A and the second up-sampled feature B;
[0124] The second up-sampled feature A and the second up-sampled feature B are respectively fused with the second deep local feature A and the second deep local feature B, and the fused features are respectively input into the third up-sampled ResNeSt convolution block to obtain the third up-sampled feature A and the third up-sampled feature B;
[0125] The third up-sampled feature A and the third up-sampled feature B are fused with the first deep local feature A and the first deep local feature B respectively, and the fused features are input into the fourth up-sampled ResNeSt convolution block respectively to obtain the extracted feature A and the extracted feature B respectively.
[0126] Based on the above embodiments, this embodiment provides a device for integrating multi-source image fusion. Figure 2 , is a structural block diagram of a device for integrated multi-source image fusion provided by an embodiment of the present invention. Figure 2 As shown, the device comprises:
[0127] An image acquisition module 201 is used to acquire a training multi-focus image set, wherein the training multi-focus image set includes a foreground focused source image A and a background focused source image B;
[0128] A model training module 202, used to train the network based on the training multi-focus image set to obtain an integrated multi-source image fusion model;
[0129] The fused image generation module 203 is used to input the actual multi-focus image set into the integrated multi-source image fusion model to generate a fused image.
[0130] Furthermore, a technical solution of the model training module specifically includes:
[0131] A feature extraction module, used for performing feature extraction on the source image A and the source image B to obtain extracted feature A and extracted feature B;
[0132] A feature merging module, used for merging the extracted feature A and the extracted feature B to obtain a merged feature map;
[0133] An initial decision graph acquisition module is used to perform sigmoid projection processing on the merged feature graph to obtain an initial decision graph of the process;
[0134] The first error calculation module is used to calculate the L between the initial decision diagram of the process and the reference true value 1 loss loss value, based on L 1 The network is repeatedly trained with loss value, and the network parameters are continuously optimized until a qualified initial decision graph is obtained;
[0135] A boundary map acquisition module, used for processing the initial decision map by using a Gaussian filter, and then performing threshold processing on the initial decision map to obtain two boundary maps with different thicknesses;
[0136] A smoothed boundary map acquisition module, used for smoothing the boundary of a boundary map having a smooth boundary region in two boundary maps with different thicknesses by using a guided filter to obtain a smoothed boundary map;
[0137] A final decision graph acquisition module, used for combining the smoothed boundary graph with the initial decision graph to obtain a final decision graph;
[0138] A fused image acquisition module, used for performing point multiplication and weighted sum operation on the final decision graph, the source image A and the source image B to obtain a fused image;
[0139] The second error calculation module is used to calculate the L error between the fused image and the source image A and the source image B respectively. 2 loss loss value, based on L 2The network is repeatedly trained based on the loss value, and the network parameters are continuously optimized until an integrated multi-source image fusion model is obtained.
[0140] On the basis of the above technical solution, for multi-focus images greatly affected by DSE, another technical solution of the model training module is to replace the fused image acquisition module and the second error calculation module with the following modules on the basis that the feature extraction module, feature merging module, initial decision map acquisition module, first error calculation module, boundary map acquisition module, smooth boundary map acquisition module and final decision map acquisition module in the above technical solution remain unchanged:
[0141] An intermediate image acquisition module is used to perform image multiplication operation on the boundary map with relatively uneven boundary areas in two boundary maps with different thicknesses and the source image A to obtain an intermediate image;
[0142] A sharpening module, used for performing a sharpening operation on the intermediate image to obtain a sharpened image;
[0143] A second fused image acquisition module, used for fusing the final decision graph with the source image A, the source image B and the sharpened image to obtain a second fused image;
[0144] The third error calculation module is used to calculate the L error between the second fused image and the source image A and the source image B respectively. 2 loss loss value, based on L 2 The network is repeatedly trained based on the loss value, and the network parameters are continuously optimized until an integrated multi-source image fusion model is obtained.
[0145] Corresponding to the above embodiment, an embodiment of the present invention further provides an electronic device.
[0146] See also Figure 3 , is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 3 As shown, the electronic device 300 may include: a processor 301, a memory 302 and a communication unit 303. These components communicate via one or more buses. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not limit the embodiments of the present invention. It may be a bus structure or a star structure, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0147] The communication unit 303 is used to establish a communication channel so that the electronic device can communicate with other devices.
[0148] The processor 301 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and / or modules stored in the memory 302, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of a plurality of packaged ICs with the same or different functions. For example, the processor 301 can include only a central processing unit (CPU). In an embodiment of the present invention, the CPU can be a single computing core or multiple computing cores.
[0149] The memory 302 is used to store the execution instructions of the processor 301. The memory 302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0150] When the execution instructions in the memory 302 are executed by the processor 301 , the electronic device 300 is enabled to execute part or all of the steps in the above method embodiment.
[0151] Corresponding to the above embodiment, the embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium may store a program, wherein when the program is running, the device where the computer-readable storage medium is located may be controlled to execute some or all of the steps in the above method embodiment. In a specific implementation, the computer-readable storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).
[0152] Corresponding to the above embodiment, an embodiment of the present invention further provides a computer program product, which includes executable instructions. When the executable instructions are executed on a computer, the computer executes part or all of the steps in the above method embodiment.
[0153] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.
[0154] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0155] In several embodiments provided by the present invention, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.
[0156] The above description is only a specific implementation of the present invention. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention.
Claims
1. A method for integrating multi-source image fusion, It is characterized in that The method comprises: Acquire a training multi-focus image set, wherein the training multi-focus image set includes two source images, namely a foreground focused source image A and a background focused source image B; Training the network based on the training multi-focus image set to obtain an integrated multi-source image fusion model; Inputting the actual multi-focus image set into the integrated multi-source image fusion model to generate a fused image; The network is trained based on the training multi-focus image set to obtain an integrated multi-source image fusion model, specifically: Step S211, inputting the source image A and the source image B into a feature extraction module respectively for feature extraction to obtain extracted features A and extracted features B; Step S212, the extracted feature A and the extracted feature B are combined by 1×1 convolution to obtain a combined feature map; Step S213, the merged feature map is processed by sigmoid projection to obtain an initial decision map of the process; Step S214, calculating the L between the initial decision diagram of the process and the reference true value 1 loss loss value, if L 1 loss value meets the requirement, then the initial decision diagram of the process is the initial decision diagram; if L 1 If the loss value does not meet the requirement, repeat steps S211-S213 until L 1 The loss value meets the requirement, and the initial decision diagram of the process at this time is the initial decision diagram; Step S215, after processing the initial decision graph using a Gaussian filter, threshold processing is performed on the initial decision graph to obtain two boundary graphs with different thicknesses; Step S216, using a guided filter to smooth the boundary of the boundary map having a smooth boundary area in the two boundary maps with different thicknesses to obtain a smooth boundary map; Step S217, combining the smoothed boundary graph with the initial decision graph to obtain a final decision graph; Step S218, performing point multiplication and weighted sum operation on the final decision graph, the source image A and the source image B to obtain a fused image; Step S219, respectively calculating the L of the fused image and the source image A and the source image B 2 loss loss value, if L 2 loss value meets the requirement, then the network is an integrated multi-source image fusion model; if L 2 If the loss value does not meet the requirement, repeat steps S211-S218 until L 2 The loss value meets the requirement, and the network is now an integrated multi-source image fusion model; The feature extraction module is a dual-channel conjoined structure with two identical structures and weights, and each channel of the feature extraction module includes a downsampling module, a ViT module, an upsampling module, and a skip-layer connection; The downsampling module includes a first downsampling ResNeSt convolution block, a second downsampling ResNeSt convolution block, a third downsampling ResNeSt convolution block and twelve Transformer layers connected in sequence; The ViT module includes a patch embedding module, n ViT layers and 1 1×1 convolution; The upsampling module includes a first upsampling ResNeSt convolution block, a second upsampling ResNeSt convolution block, a third upsampling ResNeSt convolution block and a fourth upsampling ResNeSt convolution block connected in sequence; Each of the down-sampling ResNeSt convolution blocks includes k feature map groups arranged in parallel, the output features of the k feature map groups are connected, 1×1 convolution is used to restore the original feature map size, and then the entire down-sampling ResNeSt convolution block is connected in a residual manner; Each of the feature map groups includes r segmentation convolution paths and a segmentation attention mechanism, the r segmentation convolution paths are set in parallel, and the output features of the r segmentation convolution paths are all input into the segmentation attention mechanism; The segmentation convolution path consists of a 1×1 convolutional layer and a 3×3 convolutional layer connected in sequence; Each of the ViT layers consists of h The first one consists of a multi-head self-attention and a multi-layer perceptron module; h Output features of multi-head self-attention and multi-layer perceptron modules for: , , in, For the h - output features of 1 multi-head self-attention and multi-layer perceptron module, LN is the layer normalization operator, MSA For long positions, MLP is a multi-layer perceptron, is an intermediate variable; The step S211 is specifically as follows: Inputting the source image A and the source image B into the first downsampling ResNeSt convolution block for downsampling respectively, to obtain a first deep local feature A and a first deep local feature B; Inputting the first deep local feature A and the first deep local feature B into the second downsampling ResNeSt convolution block for downsampling respectively, to obtain the second deep local feature A and the second deep local feature B; Inputting the second deep local feature A and the second deep local feature B into the third downsampling ResNeSt convolution block for downsampling respectively, to obtain the third deep local feature A and the third deep local feature B; Inputting the third deep local feature A and the third deep local feature B into twelve Transformer layers for downsampling respectively, to obtain a fourth deep local feature A and a fourth deep local feature B; Inputting the fourth deep local feature A and the fourth deep local feature B into the ViT module respectively for global feature extraction to obtain global feature A and global feature B; Inputting the global feature A and the global feature B into the first upsampling ResNeSt convolution block for upsampling respectively, to obtain the first upsampling feature A and the first upsampling feature B; The first up-sampled feature A and the first up-sampled feature B are respectively fused with the third deep local feature A and the third deep local feature B, and the fused features are respectively input into the second up-sampled ResNeSt convolution block to obtain the second up-sampled feature A and the second up-sampled feature B; The second up-sampled feature A and the second up-sampled feature B are respectively fused with the second deep local feature A and the second deep local feature B, and the fused features are respectively input into the third up-sampled ResNeSt convolution block to obtain the third up-sampled feature A and the third up-sampled feature B; The third up-sampled feature A and the third up-sampled feature B are fused with the first deep local feature A and the first deep local feature B respectively, and the fused features are input into the fourth up-sampled ResNeSt convolution block respectively to obtain the extracted feature A and the extracted feature B respectively.
2. A device for integrating multi-source image fusion, It is characterized in that The device comprises: An image acquisition module, used to acquire a training multi-focus image set, wherein the training multi-focus image set includes a foreground focused source image A and a background focused source image B; A model training module, used for training the network based on the training multi-focus image set to obtain an integrated multi-source image fusion model; A fused image generation module, used for inputting an actual multi-focus image set into the integrated multi-source image fusion model to generate a fused image; Wherein, the model training module includes: A feature extraction module, used for performing feature extraction on the source image A and the source image B to obtain extracted feature A and extracted feature B; A feature merging module, used for merging the extracted feature A and the extracted feature B to obtain a merged feature map; An initial decision graph acquisition module is used to perform sigmoid projection processing on the merged feature graph to obtain an initial decision graph of the process; The first error calculation module is used to calculate the L between the initial decision diagram of the process and the reference true value 1 loss loss value, based on L 1 The network is repeatedly trained with loss value, and the network parameters are continuously optimized until a qualified initial decision graph is obtained; A boundary map acquisition module, used for processing the initial decision map by using a Gaussian filter, and then performing threshold processing on the initial decision map to obtain two boundary maps with different thicknesses; A smoothed boundary map acquisition module, used for smoothing the boundary of a boundary map having a smooth boundary region in two boundary maps with different thicknesses by using a guided filter to obtain a smoothed boundary map; A final decision graph acquisition module, used for combining the smoothed boundary graph with the initial decision graph to obtain a final decision graph; A fused image acquisition module, used for performing point multiplication and weighted sum operation on the final decision graph, the source image A and the source image B to obtain a fused image; The second error calculation module is used to calculate the L error between the fused image and the source image A and the source image B respectively. 2 loss loss value, based on L 2 The network is repeatedly trained based on the loss value, and the network parameters are continuously optimized until an integrated multi-source image fusion model is obtained; The feature extraction module is a dual-channel conjoined structure with two identical structures and weights, and each channel of the feature extraction module includes a downsampling module, a ViT module, an upsampling module, and a skip-layer connection; The downsampling module includes a first downsampling ResNeSt convolution block, a second downsampling ResNeSt convolution block, a third downsampling ResNeSt convolution block and twelve Transformer layers connected in sequence; The ViT module includes a patch embedding module, n ViT layers and 1 1×1 convolution; The upsampling module includes a first upsampling ResNeSt convolution block, a second upsampling ResNeSt convolution block, a third upsampling ResNeSt convolution block and a fourth upsampling ResNeSt convolution block connected in sequence; Each of the down-sampling ResNeSt convolution blocks includes k feature map groups arranged in parallel, the output features of the k feature map groups are connected, 1×1 convolution is used to restore the original feature map size, and then the entire down-sampling ResNeSt convolution block is connected in a residual manner; Each of the feature map groups includes r segmentation convolution paths and a segmentation attention mechanism, the r segmentation convolution paths are set in parallel, and the output features of the r segmentation convolution paths are all input into the segmentation attention mechanism; The segmentation convolution path consists of a 1×1 convolutional layer and a 3×3 convolutional layer connected in sequence; Each of the ViT layers consists of h The first one consists of a multi-head self-attention and a multi-layer perceptron module; h Output features of multi-head self-attention and multi-layer perceptron modules for: , , in, For the h - output features of 1 multi-head self-attention and multi-layer perceptron module, LN is the layer normalization operator, MSA For long positions, MLP is a multi-layer perceptron, is an intermediate variable; The feature extraction module is used to extract features from the source image A and the source image B to obtain extracted features A and extracted features B, specifically: Inputting the source image A and the source image B into the first downsampling ResNeSt convolution block for downsampling respectively, to obtain a first deep local feature A and a first deep local feature B; Inputting the first deep local feature A and the first deep local feature B into the second downsampling ResNeSt convolution block for downsampling respectively, to obtain the second deep local feature A and the second deep local feature B; Inputting the second deep local feature A and the second deep local feature B into the third downsampling ResNeSt convolution block for downsampling respectively, to obtain the third deep local feature A and the third deep local feature B; Inputting the third deep local feature A and the third deep local feature B into twelve Transformer layers for downsampling respectively, to obtain a fourth deep local feature A and a fourth deep local feature B; Inputting the fourth deep local feature A and the fourth deep local feature B into the ViT module respectively for global feature extraction to obtain global feature A and global feature B; Inputting the global feature A and the global feature B into the first upsampling ResNeSt convolution block for upsampling respectively, to obtain the first upsampling feature A and the first upsampling feature B; The first up-sampled feature A and the first up-sampled feature B are respectively fused with the third deep local feature A and the third deep local feature B, and the fused features are respectively input into the second up-sampled ResNeSt convolution block to obtain the second up-sampled feature A and the second up-sampled feature B; The second up-sampled feature A and the second up-sampled feature B are respectively fused with the second deep local feature A and the second deep local feature B, and the fused features are respectively input into the third up-sampled ResNeSt convolution block to obtain the third up-sampled feature A and the third up-sampled feature B; The third up-sampled feature A and the third up-sampled feature B are fused with the first deep local feature A and the first deep local feature B respectively, and the fused features are input into the fourth up-sampled ResNeSt convolution block respectively to obtain the extracted feature A and the extracted feature B respectively.
