Multi-scale feature image fusion method in complex scene

Abstract

The invention relates to the technical field of image fusion, in particular to a multi-scale feature image fusion method in a complex scene, which comprises the following steps of: S1, performing multi-scale analysis on each input image by combining a multi-scale convolutional neural network with depth separable convolution, and identifying and strengthening key features between a focusing region and a defocusing region through a CA coordinate attention module; s2, screening and strengthening the feature maps through an attention gate module, and further fusing to obtain a comprehensive feature map; and S3, carrying out initial segmentation on the fused feature mapping, and generating a final segmentation graph or decision graph by adopting the strong segmentation capability of the U-Net model and combining morphological processing and watershed algorithm fine adjustment. The integrated processing strategy ensures the accuracy of the decision diagram and the high quality of the fused image.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to an image fusion method of multi-scale features in complex scenes. Background Art

[0002] The basic purpose of image fusion is to extract useful information from two or more images and combine them into a new image so that the new image can be more effectively used for human visual perception or computer processing programs. For example, in satellite imaging, fusing images from different times or different sensors can provide more comprehensive data on environmental changes; in medical imaging, by fusing images from different imaging devices (such as CT and MRI), doctors can obtain more information about the patient's condition.

[0003] Image fusion technology can be roughly divided into two categories: pixel-based fusion and feature-based fusion.

[0004] Pixel-based fusion: This method directly operates on the pixel values ​​of the source image, such as simple pixel averaging, selecting the maximum or minimum value, etc. This type of method is simple to operate, but may not be able to handle more complex scene changes because they ignore the relationship between pixels and the structural information of the image.

[0005] Feature-based fusion: This method first extracts features (such as edges, textures, colors, etc.) from each image, then fuses these features together according to certain rules, and finally reconstructs the image through some inverse transformations. This method can better preserve the structural information of the image, but the computational complexity is higher.

[0006] With the rapid development of deep learning technology, its application in the field of image processing is becoming more and more extensive, especially in the field of image fusion. Deep learning models, especially convolutional neural networks (CNNs), have become the preferred technology for image processing tasks due to their powerful feature extraction capabilities. These models can automatically learn and recognize complex patterns and features in images without the need to manually design feature extractors.

[0007] The basic concept of image fusion: First, image fusion is a process of merging multiple images with the goal of creating a new image that is better than any individual input image in a specific aspect (such as clarity). In multi-focus image fusion, multiple images of the same scene are usually processed, each with different focus depths in different areas. For example, the foreground in one image may be clear, while the background may be blurred; the background in another image may be very clear.

[0008] (1) Multi-scale Convolutional Neural Network (MSCNN) At the heart of this technology is a multi-scale convolutional neural network designed to extract features at different scales from each input image. In practice, the network automatically learns how to capture key information by considering different levels of detail in an image, from microscopic to macroscopic. For example, the network might capture details of edges and textures at one scale, while analyzing the overall shape and structure of an image region at another, more macroscopic scale. This multi-scale analysis approach ensures that the network can fully assess the image content and identify which areas are in focus and which are blurry.

[0009] (2) Generation of decision diagram Based on the features extracted from MSCNN, the next step is to generate a decision graph. The decision graph is a key intermediate product that indicates how to select pixels from multiple input images to generate the final fused image. Specifically, this image segmentation technique classifies each pixel in the feature map as "in focus" or "out of focus" and determines from which original image the corresponding pixel should be extracted. This process is similar to drawing dividing lines in the image, marking exactly which areas are in sharp focus and which are not.

[0010] (3) Post-processing steps Once the decision map is generated, it needs to be post-processed to ensure the quality of the fused image. This includes using morphological operations to optimize image boundaries, remove noise, strengthen boundaries, etc., and applying the watershed algorithm to fine-tune the boundaries of each region. These post-processing steps are necessary because they help clearly define each region and ensure that the pixel selection of the focus area in the final fused image is accurate.

[0011] Through this series of steps - from multi-scale feature extraction, to decision graph production, to post-processing optimization - the technical solution achieves efficient fusion of multi-focus images. The final fused image is not only visually clearer and more coherent, but also technically more accurate and reliable, fully demonstrating the powerful potential of deep learning in the field of image processing.

[0012] The existing MSCNN has the following problems: (1) Although MSCNN can process multi-scale image features, it lacks an effective mechanism to dynamically focus on and emphasize the most important feature areas in the image. In multi-focus image fusion, it is crucial to correctly identify which areas are clearly focused areas. MSCNN may not be able to make this distinction accurately, resulting in the loss of details in important areas of the fused image.

[0013] (2) Traditional MSCNN designs may contain a large number of convolution operations, which not only increases the computational burden but may also lead to inefficient operation of the model on edge devices. For applications that require real-time processing, such as real-time video surveillance image fusion, this high computational load is unacceptable.

[0014] (3) In deep network structures, information is often gradually lost during the transmission process, especially without the support of effective skip connections, which may lead to the loss of details from the original input to the final output.

[0015] (4) MSCNN usually lacks the ability to dynamically adjust its feature processing strategy, which may lead to unsatisfactory fusion effects when faced with images of various types and qualities.

[0016] To this end, this application designs an image fusion method of multi-scale features in complex scenes and solves the above problems. Summary of the invention

[0017] In order to make up for the deficiencies in the prior art, the present invention provides an image fusion method of multi-scale features in complex scenes.

[0018] An image fusion method for multi-scale features in complex scenes, characterized in that the task of detecting decision mapping is regarded as image segmentation between focused areas and defocused areas in a source image, and a CA coordinate attention module, a depth-separable convolution, a U-Net model and an attention gate module are combined to achieve efficient and accurate image segmentation, including the following steps: S1, a multi-scale convolutional neural network combined with a deep separable convolution performs multi-scale analysis of each input image, and identifies and enhances the key features between the focused and defocused areas through the CA coordinate attention module; S2, filters and strengthens these feature maps through the attention gate module, and further fuses them to obtain a comprehensive feature map; S3 performs initial segmentation on the fused feature map, using the powerful segmentation capability of the U-Net model, combined with morphological processing and watershed algorithm fine-tuning to generate the final segmentation map or decision map.

[0019] Furthermore, in order to better implement the present invention, the S1 is specifically: S11, the input layer receives image patches of size 16×16 pixels. This size is chosen because it can reduce computational complexity while maintaining sufficient details, making it suitable for fast feature extraction; S12, the first convolution layer: convolve the input image using a 3×3 depthwise separable filter and output 64 feature maps. The depthwise separable convolution first applies convolution independently on each input channel, and then combines the outputs of all channels using a 1×1 convolution with a stride of 1 to ensure that the spatial size of the feature map remains unchanged while extracting rich local features; The second convolution layer: Continue to use a 3×3 depth-separable filter and output 128 feature maps. This layer further deepens the feature extraction and provides more detailed information for subsequent feature analysis. S13, feature enhancement and focusing: The CA coordinate attention module is introduced after the second convolution layer. This module focuses on analyzing the spatial correlation of image features, identifying key areas such as focus areas, and improving the representation weights of these areas to optimize the expression of information in the feature map; S14, pooling and feature fusion: MaxPooling: Use a 2×2 filter and a MaxPooling layer with a stride of 2 to reduce feature dimensions while maintaining key features. This step effectively reduces the amount of data and prepares the data for deeper network structures. Further convolution processing: The obtained feature map is then passed through a 3×3 depth-wise separable convolution layer to output 256 feature maps. The purpose of this layer is to further refine and enhance the features and prepare more accurate data for the final decision map.

[0020] Further, in order to better implement the present invention, S2 is specifically: S21, Attention gating and feature selection: Before the feature map enters the fully connected layer, it is screened and enhanced by the attention gate module. These modules dynamically adjust the importance of feature channels to ensure that only the most useful features are used for the final image fusion decision, thereby optimizing the fusion effect; S22, decision and output: The feature map is converted into a two-dimensional vector through the fully connected layer, representing the decision state of the image, such as focus or non-focus; S23, uses the Softmax layer to process this two-dimensional vector and outputs the probability distribution of two categories, namely the probabilities of focusing and defocusing, for the final fusion decision.

[0021] Furthermore, in order to better implement the present invention, in S3, the output of Softmax is subjected to morphological processing and a watershed algorithm is applied to generate an accurate segmentation map or decision map. This step ensures clear boundaries during image fusion and avoids fusion errors.

[0022] The beneficial effects of the present invention are: 1. Make the model more suitable for running in environments with limited computing resources, such as mobile devices and real-time systems, while ensuring the quality and speed of image processing.

[0023] 2. Improved the accuracy and quality of image fusion, especially when processing images with complex backgrounds and multi-focus areas, and can more accurately identify and preserve the details of important areas.

[0024] 3. This combination improves detail preservation and boundary clarity in image fusion, ensuring the visual coherence and quality of the fused image, especially the significant performance in boundary processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of a process of extracting a multi-scale input patch from an input image according to the present invention; Figure 2 It is a work flow chart of the present invention; Figure 3 Schematic diagram of the CA attention mechanism of the present invention. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Generally, the components of the embodiments of the present invention described and shown in the accompanying drawings here can be arranged and designed in various different configurations.

[0027] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0028] Figure 1-Figure 3This is a specific embodiment of the present invention, which is a new multi-focus image fusion method based on image segmentation. The method regards the task of detecting decision maps as image segmentation between focused areas and defocused areas in the source image. The method combines the CA coordinate attention module, deep separable convolution, U-Net model and attention gate module to achieve efficient and accurate image segmentation. First, the multi-scale convolutional neural network combines with deep separable convolution to perform multi-scale analysis of each input image, and the key features between the focused area and the defocused area are identified and enhanced through the CA coordinate attention module. Next, these feature maps are screened and enhanced through the attention gate module, and further fused to obtain a comprehensive feature map. The fused feature map is initially segmented, and the powerful segmentation capability of the U-Net model is used, combined with morphological processing and watershed algorithm for fine adjustment to generate the final segmentation map / decision map. This integrated processing strategy ensures the accuracy of the decision map and the high quality of the fused image. The specific process is as follows: This network uses the CA coordinate attention module, depthwise separable convolution, U-Net model, and attention gate module to perform multi-focus image fusion. The specific process is as follows: (1) Input layer: The input layer receives image patches of size 16 × 16 pixels. This size was chosen because it can reduce computational complexity while maintaining sufficient details, making it suitable for fast feature extraction.

[0029] (2) Primary convolutional layer: First convolution layer: convolve the input image using a 3×3 depthwise separable filter, outputting 64 feature maps. Depthwise separable convolution first applies convolution to each input channel independently (depthwise convolution), and then combines the outputs of all channels using a 1×1 convolution (pointwise convolution). The stride is 1, which ensures that the spatial size of the feature map remains unchanged while extracting rich local features.

[0030] The second convolution layer: Continue to use a 3×3 depth-separable filter and output 128 feature maps. This layer further deepens the feature extraction and provides more detailed information for subsequent feature analysis.

[0031] (3) Feature enhancement and focusing module: CA coordinate attention module: The CA coordinate attention module is introduced after the second layer of convolution. This module focuses on analyzing the spatial correlation of image features, identifying key areas such as focus areas, and improving the representation weights of these areas to optimize the expression of information in the feature map.

[0032] (4) Pooling and feature fusion: MaxPooling: Use a 2×2 filter and a MaxPooling layer with a stride of 2 to reduce feature dimensions while maintaining key features. This step effectively reduces the amount of data and prepares the data to enter a deeper network structure.

[0033] Further convolution processing: The obtained feature map is then passed through a 3×3 depthwise separable convolution layer to output 256 feature maps. The purpose of this layer is to further refine and enhance the features and prepare more accurate data for the final decision map.

[0034] (5) Attention gating and feature selection: Attention gate module: Before the feature map enters the fully connected layer, it is filtered and strengthened by the attention gate module. These modules dynamically adjust the importance of feature channels to ensure that only the most useful features are used for the final image fusion decision, thus optimizing the fusion effect.

[0035] (6) Decision-making and output: Fully connected layer: The feature map is converted into a two-dimensional vector through the fully connected layer, which represents the decision state of the image (such as focus or non-focus).

[0036] (7) Softmax layer: Finally, a Softmax layer is used to process this two-dimensional vector and output the probability distribution of two categories, namely the probabilities of focus and defocus, for the final fusion decision.

[0037] (8) Post-processing steps: Morphological processing and watershed algorithm are applied to the output of Softmax to generate accurate segmentation map / decision map. This step ensures clear boundaries during image fusion and avoids fusion errors.

[0038] The design of this network structure takes into account the high sensitivity to details and the need for computational efficiency. By combining the depth-separable convolution, CA coordinate attention module and attention gate module, it can not only improve the accuracy of image fusion, but also handle complex image scenes while maintaining low computational costs. This structural design ensures that image areas with different focal points can be effectively distinguished and processed in multi-focus image fusion, achieving high-quality fusion output.

[0039] The present invention adopts depth-separable convolution in multiple layers of the model, which significantly reduces the computational complexity and the number of parameters. This convolution method separates the spatial and depth processing in traditional convolution, improves the computational efficiency, and maintains the efficient extraction of image features. The CA coordinate attention module is introduced to enhance the model's attention to important spatially related features in the image. This module can dynamically adjust the focus according to the spatial features of the image and prioritize the most critical parts of the image. The symmetric structure and jump connection of the U-Net model are adopted, and the attention gate module is combined to optimize the information flow and feature selection. The jump connection of U-Net helps to retain the detail information that may be lost in the deep network, and the attention gate module further strengthens the processing of key features on this basis. Morphological processing and watershed algorithm are applied to refine the decision graph after the model output. This post-processing strategy accurately processes and optimizes the final image segmentation result.

[0040] The present invention can solve the following problems: 1. Accurate feature extraction Problem: Existing techniques cannot adequately extract all important features in an image, especially in areas with rich details or insufficient contrast.

[0041] Solution: The CA coordinate attention module ensures that important features are not overlooked by automatically identifying key areas in the image and prioritizing them. This means that no matter how subtle or imperceptible a part of the image is, the system can identify and emphasize it.

[0042] 2. High-efficiency processing Problem: Traditional image fusion methods are computationally intensive and inefficient, and are not suitable for scenarios that require fast processing.

[0043] Solution: Depthwise separable convolution greatly reduces computational requirements and improves processing speed. This technology optimizes the convolution process so that image fusion tasks can run quickly on ordinary hardware, making it very suitable for real-time or mobile applications.

[0044] 3. Dynamic and flexible integration Problem: Existing fusion strategies are often too fixed and cannot adapt to actual changes in image content.

[0045] Solution: The attention gate module can dynamically adjust the weights of different features in the fusion process, making the fusion result more natural and realistic. Whether it is night scene, foggy day or other complex lighting conditions, the system can intelligently adjust and optimize the output results.

[0046] 4. Keep details and context Problem: Details are easily lost during the fusion process, especially in deep networks.

[0047] Solution: The special feature of the U-Net model is its symmetrical structure and skip connections, which ensure that the information of the image from input to output is preserved at each step. In this way, every detail from edge sharpness to color saturation can be accurately presented in the fused image.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Other modifications or equivalent substitutions made to the technical solution of the present invention by ordinary technicians in the field should be included in the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for image fusion of multi-scale features in complex scenes, characterized in that: The task of detecting decision mapping is regarded as image segmentation between focused and defocused areas in the source image. The CA coordinate attention module, depthwise separable convolution, U-Net model and attention gate module are combined to achieve efficient and accurate image segmentation, including the following steps: S1, a multi-scale convolutional neural network combined with a deep separable convolution performs multi-scale analysis of each input image, and identifies and enhances the key features between the focused and defocused areas through the CA coordinate attention module; S2, filters and strengthens these feature maps through the attention gate module, and further fuses them to obtain a comprehensive feature map; S3 performs initial segmentation on the fused feature map, using the powerful segmentation capability of the U-Net model, combined with morphological processing and watershed algorithm fine-tuning to generate the final segmentation map or decision map.

2. The image fusion method of multi-scale features in complex scenes according to claim 1, characterized in that: The S1 is specifically: S11, the input layer receives image patches of size 16×16 pixels. This size is chosen because it can reduce computational complexity while maintaining sufficient details, making it suitable for fast feature extraction; S12, the first convolution layer: convolve the input image using a 3×3 depthwise separable filter and output 64 feature maps. The depthwise separable convolution first applies convolution independently on each input channel, and then combines the outputs of all channels using a 1×1 convolution with a stride of 1 to ensure that the spatial size of the feature map remains unchanged while extracting rich local features; The second convolution layer: Continue to use a 3×3 depth-separable filter and output 128 feature maps. This layer further deepens the feature extraction and provides more detailed information for subsequent feature analysis. S13, feature enhancement and focusing: The CA coordinate attention module is introduced after the second convolution layer. This module focuses on analyzing the spatial correlation of image features, identifying key areas such as focus areas, and improving the representation weights of these areas to optimize the expression of information in the feature map; S14, pooling and feature fusion: MaxPooling: Use a 2×2 filter and a MaxPooling layer with a stride of 2 to reduce feature dimensions while maintaining key features. This step effectively reduces the amount of data and prepares the data for deeper network structures. Further convolution processing: The obtained feature map is then passed through a 3×3 depth-wise separable convolution layer to output 256 feature maps. The purpose of this layer is to further refine and enhance the features and prepare more accurate data for the final decision map.

3. The image fusion method of multi-scale features in complex scenes according to claim 1, characterized in that: The S2 is specifically: S21, Attention gating and feature selection: Before the feature map enters the fully connected layer, it is screened and enhanced by the attention gate module. These modules dynamically adjust the importance of feature channels to ensure that only the most useful features are used for the final image fusion decision, thereby optimizing the fusion effect; S22, decision and output: The feature map is converted into a two-dimensional vector through the fully connected layer, representing the decision state of the image, such as focus or non-focus; S23, uses the Softmax layer to process this two-dimensional vector and outputs the probability distribution of two categories, namely the probabilities of focusing and defocusing, for the final fusion decision.

4. The image fusion method of multi-scale features in complex scenes according to claim 1, characterized in that: In S3, morphological processing and watershed algorithm are performed on the output of Softmax to generate an accurate segmentation map or decision map. This step ensures clear boundaries during image fusion and avoids fusion errors.

Images