Polarization and intensity image fusion method based on saliency mechanism and multi-layer attention perception

Abstract

The present invention discloses a polarization and intensity image fusion method based on saliency mechanism and multi-layer attention perception, wherein polarization and intensity images are input into a densely connected feature extraction encoder to obtain corresponding source image features, and then input into a multi-layer progressive attention perception fusion module to perform image fusion operation and output fusion features; polarization images are fed into a polarization salient target highlighting mechanism, and high-brightness polarization salient targets in polarization images are mined based on intensity saliency and spatial compactness to obtain polarization polarization target saliency maps; fusion features and polarization polarization target saliency maps are spliced, and the spliced ​​features are input into an image reconstruction decoder, and a fused image is reconstructed through multi-layer convolution processing, and a fusion network is trained based on a loss function of a deep feature map, and real-time polarization and intensity image fusion is performed based on the trained fusion network. The ideal fusion effect of polarization polarization information and texture details can be achieved to obtain a high-quality fused image.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a polarization and intensity image fusion method based on saliency mechanism and multi-layer attention perception. Background Art

[0002] Intensity images are widely used today. By capturing the reflected and refracted light of an object, clear and rich scene information can be obtained. However, intensity images tend to describe the reflectivity and refractive index of an object, and their susceptibility to light often causes them to ignore some subtle information about the object, such as the surface roughness and texture of the object. Compared with intensity images, polarization images describe the object based on its inherent polarization properties and are not affected by light intensity. Polarization imaging technology captures multi-angle intensity information through polarization imaging devices. Comprehensive analysis of these multi-angle intensity information can obtain polarization information, which is often expressed by Stokes vectors. Stokes vectors are generally divided into three categories, respectively: , as well as Describe it. Further analysis and calculation of the three types of polarization information can further obtain the Degree of Linear Polarization (DoLP) image. The linear polarization image can describe the polarization characteristics of the object and capture subtle features such as the shape, roughness and texture of the object. In summary, polarization images and intensity images can obtain object information from two different angles, providing each other with the required complementary information. Based on this, the fusion of polarization images and intensity images can obtain images with richer information and stronger expressiveness, providing reliable help for improving the performance of various advanced visual tasks in complex scenes.

[0003] In the past few decades, scholars in the field of image fusion have proposed a variety of image fusion methods. These methods can be divided into traditional methods and deep learning methods according to the different mechanisms and networks adopted. Methods based on multi-scale transformation, sparse representation and subspace are three commonly used traditional fusion methods. Methods based on multi-scale transformation are widely used in the field of image fusion. They extract and operate the features of the source image by transforming decomposition and formulating fusion rules. The performance of this type of method is also mainly affected by the transformation method and fusion rules. The sparse representation-based method learns a complete dictionary from the image, and then uses the complete dictionary and specific encoding to generate the result image. The subspace-based method aims to map the input high-dimensional image to a low-dimensional subspace, eliminate redundant information in the source image, and thus more conducive to extracting feature information. Although these traditional methods can achieve relatively ideal fusion results in most cases, there are still some defects. Traditional image fusion methods often use the same processing method for source images of different modalities, which makes it impossible to fully extract complementary features in the source image. In addition, the manually designed fusion rules and the increasing complexity of feature transformation and feature representation in traditional methods make it difficult to meet the needs of today's image fusion.

[0004] Recently, deep learning-based methods have achieved outstanding performance in multiple computer vision fields due to their powerful feature representation and feature learning capabilities. Therefore, scholars in the field of image fusion have gradually applied deep learning to multiple steps such as feature extraction and feature fusion in image fusion. According to the different network frameworks adopted, deep learning-based image fusion methods can be mainly divided into the following three categories: methods based on convolutional neural network (CNN), methods based on auto-encoder (AE), and methods based on generative adversarial network (GAN). The powerful representation learning ability of convolutional neural network enables methods based on convolutional neural network to effectively extract and reconstruct feature information. The method based on autoencoder realizes feature extraction and reconstruction of fused image by training encoder and decoder. This type of method usually adopts specific fusion rules to fuse the extracted features. The method based on generative adversarial network can directly generate the required fused image through the adversarial game mechanism between its generator and discriminator. In order to improve the fusion performance, some scholars proposed a deep network composed of encoder layer, fusion layer and decoder layer, and adopted a self-learning strategy to optimize the fusion problem of polarization and intensity images. Some scholars have proposed a deep network based on the combination of Transformer and CNN to improve the performance of polarization and intensity image fusion. Subsequently, some scholars have further proposed an unsupervised polarization and intensity image fusion network based on pixel information guidance and attention mechanism.

[0005] Although image fusion has developed rapidly in recent years, most of the existing image fusion methods focus on the fusion between other modal images, such as multi-focus image fusion, multi-exposure image fusion, and infrared and visible light image fusion. There are relatively few fusion methods for polarization and intensity image fusion. Although the above-mentioned polarization and intensity image fusion methods can achieve a relatively ideal fusion effect to a certain extent, there are still problems that need to be overcome urgently. Existing fusion methods often ignore the differences between different modal images and use the same processing method or network to extract features from source images of different modalities, resulting in the inability to fully utilize the complementary information in the source image. Secondly, simple fusion operations cannot fully explore the meaningful information in the extracted source image features and may even introduce redundant information. For example, some low-intensity areas in the polarization image show very little information. These information-deficient areas will have a negative impact on the quality of the final generated fused image during the fusion process. In addition, the loss function currently used for fusion network training usually only performs mathematical operations based on the source image and the final fused image, while ignoring the influence of the different depths of features that can be extracted by the network on the loss function in the back propagation and guidance of the network training process, resulting in the trained network being insufficiently sensitive to feature information. At the same time, the loss function does not take into account the differences in intensity information and texture information of source images of different modalities, and does not make reasonable weight distribution for different losses during the calculation process, which affects the fusion effect of the network. Summary of the invention

[0006] In view of the above technical problems, the present invention provides a polarization and intensity image fusion method based on saliency mechanism and multi-layer attention perception.

[0007] The technical solution adopted by the present invention to solve the technical problem is:

[0008] A polarization and intensity image fusion method based on saliency mechanism and multi-layer attention perception, the method comprising the following steps:

[0009] S100: Obtain polarization and intensity image datasets and build a fusion network, which includes a densely connected feature extraction encoder, a multi-layer progressive attention perception fusion module, and an image reconstruction decoder;

[0010] S200: inputting the polarization and intensity images in the data set into the fusion network at the same time, and obtaining the corresponding polarization image features and intensity image features through the densely connected feature extraction encoder;

[0011] S300: Inputting the extracted polarization image features and intensity image features into a multi-layer progressive attention perception fusion module for image fusion operation, and outputting fusion features after being guided by three layers of continuous attention perception inside the module;

[0012] S400: feeding the polarization image into the polarization salient target highlighting mechanism, mining the polarization salient targets with high brightness in the polarization image based on intensity saliency and spatial compactness, and obtaining the polarization polarization target saliency map;

[0013] S500: performing a splicing operation on the fusion features output by the fusion module and the polarization polarization target saliency map, inputting the spliced ​​features into the image reconstruction decoder, and reconstructing the fused image through multi-layer convolution processing;

[0014] S600: The fusion network is trained based on the fusion features, fusion images, polarization images, intensity images, polarization image features and intensity image features and the loss function based on the depth feature map. When the loss function based on the depth feature map converges, a trained fusion network is obtained, and real-time polarization and intensity image fusion is performed based on the trained fusion network.

[0015] Preferably, the densely connected feature extraction encoder in S200 includes a convolutional layer and two dense blocks, the convolutional layer uses ReLu as its activation function, and the dense block includes three consecutive convolutional layers, wherein the input of each convolutional layer is the concatenation of the output of the previous convolution and the output of all previous convolutional layers, and the polarization image and intensity image are respectively input into the densely connected feature extraction encoder to obtain corresponding polarization image features and intensity image features.

[0016] Preferably, the multi-layer progressive attention perception fusion module in S300 is composed of three space-channel attention perception modules connected together. The polarization image features and the intensity image features are spliced ​​and input into the first space-channel attention perception module. The output of the first space-channel attention perception module is respectively multiplied element-by-element with the polarization image features and the intensity image features and then the features obtained are spliced ​​again and input into the second space-channel attention perception module. The output of the second space-channel attention perception module is respectively multiplied element-by-element with the polarization image features and the intensity image features and then the features obtained are spliced ​​again and input into the third space-channel attention perception module. The output of the third space-channel attention perception module is respectively multiplied element-by-element with the polarization image features and the intensity image features and then the features obtained are spliced ​​again to obtain the fusion features.

[0017] Preferably, each channel-spatial attention perception module includes a channel attention mechanism and a spatial attention mechanism. After the input features are input into the channel attention mechanism, the features obtained after maximum pooling and average pooling are respectively performed on the channel dimension and then fed into the multi-layer perceptron for learning. The features output by the multi-layer perceptron are added element by element, and then multiplied element by element with the input features to obtain the channel attention features. The channel attention features are input into the spatial attention mechanism, and the features obtained after maximum pooling and average pooling are respectively performed on the spatial dimension. After being spliced, a convolution operation is performed, and the obtained features are multiplied element by element with the channel attention features to obtain the channel-spatial attention features, and then converted into a channel-spatial attention weight map consistent with the input feature dimension through a Sigmoid activation function.

[0018] Preferably, the specific calculation process of the multi-layer progressive attention perception fusion module in S300 is expressed as:

[0019] ;

[0020] ;

[0021] ;

[0022] in, represents the feature concatenation operation, represents element-wise multiplication operation, represents the input features fed into the channel-spatial attention perception module, , the input features of the first layer are It is obtained by concatenating the extracted features of source images of different modalities. Represents the fusion features obtained at each layer, and the last layer Represents the fusion features finally output by the fusion module.

[0023] Preferably, S400 includes:

[0024] S410: Acquire an intensity saliency map of the polarization image by constructing a threshold truncation function;

[0025] S420: using a simple linear iterative clustering method to divide the saliency map into different regions, and calculating the center deviation of each region respectively;

[0026] S430: After obtaining the center deviation of each segmented area, quantify the center deviation of each segmented area using a half-Gaussian distribution;

[0027] S440: The spatial compactness map obtained by spatial compactness calculation is combined with the intensity saliency map to obtain a final polarization polarization target saliency map.

[0028] Preferably, S410 specifically includes:

[0029] ;

[0030] in, Represents the pixel intensity of the point at position (x,y) in the intensity saliency map, Represents the pixel intensity of the point at position (x,y) in the polarization image, It indicates the cutoff threshold;

[0031] S420 is specifically:

[0032] ;

[0033] ;

[0034] ;

[0035] ;

[0036] ;

[0037] in, represents the step length, and Respectively represent the total number of pixels and the number of cluster centers in the image. Represents the color distance between the pixel and the cluster center. Represents the spatial distance between the pixel and the cluster center point, represents the final distance metric, is a fixed trade-off parameter used to weigh the relative importance between color similarity and spatial proximity, Indicates the split The center deviation of the region, represents the centroid of the region in the x direction, represents the centroid of the region in the y direction, and Represent the height and width of the polarization image respectively;

[0038] S430 is specifically:

[0039] ;

[0040] in, Represents the spatial compact map of the current segmentation area, which represents the spatial structure information within the current segmentation area. Indicates the parameter for determining the spatial compactness;

[0041] S440 is specifically:

[0042] ;

[0043] in, Represents the final output polarization target saliency map, and Respectively indicate the corresponding Intensity saliency map and spatial compactness map of a region.

[0044] Preferably, the image reconstruction decoder in S500 includes four consecutive convolutional layers, each convolutional layer uses ReLu as the activation function, and the output channels of each convolution layer are 64, 32, 16, and 1 respectively. The fused features output from the multi-layer progressive attention perception fusion module are reconstructed into a single-channel fused image with significant polarization targets and rich and clear texture details via the image reconstruction decoder.

[0045] Preferably, the loss function based on the depth feature map in S600 includes an intensity-texture joint loss function and a perceptual loss function, specifically:

[0046] ;

[0047] in, is the hyperparameter used to balance the two non-class component loss functions, is the intensity-texture joint loss function, is the perceptual loss function.

[0048] Preferably, the intensity-texture joint loss function is specifically:

[0049] ;

[0050] ;

[0051] ;

[0052] ;

[0053] ;

[0054] ;

[0055] in, represents the exponential function operation, and represents the first polarization image extracted by the feature extraction encoder The features of the layer and the intensity image The characteristics of the layer, and represent polarization image and intensity image respectively, is a multiplication factor used to reduce or amplify the difference in pixel intensity of the source image. , is the balance parameter in texture loss, is the balance parameter in strength loss, and Represents the intermediate gradient information quantification index of polarization image and intensity image, , ,and Indicates the corresponding The length, width, and number of channels of the layer’s features, , and Indicates the length, width and number of channels of the corresponding source image. represents the sobel gradient operator, Hyperparameters to balance intensity loss and texture loss;

[0056] The perceptual loss function is specifically:

[0057] ;

[0058] in, is the fusion feature output by the multi-layer progressive attention perception fusion module, and N represents the total number of layers of polarization images and intensity images.

[0059] The above-mentioned polarization and intensity image fusion method based on saliency mechanism and multi-layer attention perception can fully mine the valuable information in the source image features by performing feature fusion based on more reasonable attention weights through the multi-layer progressive attention perception fusion module, while avoiding the negative impact of excessive redundant information on the fused image, so that the fused image retains ideal intensity information and rich and clear texture details; through the polarization salient target highlighting mechanism, the polarization salient information in the polarization image can be more fully mined and utilized, and the saliency of the polarization polarization target in the fusion result can be enhanced, thereby improving the overall contrast of the fused image; through the novel loss function based on the deep feature map to guide the fusion network training, the sensitivity of the fusion network to the information features in the source image is improved, and the loss of some feature information is avoided; based on the polarization salient target highlighting mechanism and multi-layer progressive attention perception, the ideal fusion effect of polarization polarization information and texture details can be achieved, and a high-quality fused image can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a flow chart of a polarization and intensity image fusion method based on a saliency mechanism and multi-layer attention perception in one embodiment of the present invention;

[0061] Figure 2 It is an overall framework diagram of a polarization and intensity image fusion method based on a saliency mechanism and multi-layer attention perception in one embodiment of the present invention;

[0062] Figure 3 It is a schematic diagram of the structure of a multi-layer progressive attention perception fusion module in one embodiment of the present invention;

[0063] Figure 4 Schematic diagram of the structure of a channel-spatial attention perception module in one embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0065] In one embodiment, Figure 1 As shown, a polarization and intensity image fusion method based on saliency mechanism and multi-layer attention perception comprises the following steps:

[0066] S100: Obtain polarization and intensity image datasets and build a fusion network, which includes a densely connected feature extraction encoder, a multi-layer progressive attention perception fusion module, and an image reconstruction decoder;

[0067] S200: inputting the polarization and intensity images in the data set into the fusion network at the same time, and obtaining the corresponding polarization image features and intensity image features through the densely connected feature extraction encoder;

[0068] S300: Inputting the extracted polarization image features and intensity image features into a multi-layer progressive attention perception fusion module for image fusion operation, and outputting fusion features after being guided by three layers of continuous attention perception inside the module;

[0069] S400: feeding the polarization image into the polarization salient target highlighting mechanism, mining the polarization salient targets with high brightness in the polarization image based on intensity saliency and spatial compactness, and obtaining the polarization polarization target saliency map;

[0070] S500: performing a splicing operation on the fusion features output by the fusion module and the polarization polarization target saliency map, inputting the spliced ​​features into the image reconstruction decoder, and reconstructing the fused image through multi-layer convolution processing;

[0071] S600: The fusion network is trained based on the fusion features, fusion images, polarization images, intensity images, polarization image features and intensity image features and the loss function based on the depth feature map. When the loss function based on the depth feature map converges, a trained fusion network is obtained, and real-time polarization and intensity image fusion is performed based on the trained fusion network.

[0072] Specifically, in order to solve the problems of the prior art, the present invention proposes a polarization and intensity image fusion method based on saliency mechanism and multi-layer attention perception. The network of the proposed polarization and intensity image fusion method is mainly composed of the following three parts: a densely connected feature extraction encoder, a multi-layer progressive attention perception fusion module and an image reconstruction decoder. In addition, in order to improve the sensitivity of the fusion network to the feature information of the source image, the present invention proposes a novel loss function based on a deep feature map, which consists of two parts: an intensity-texture joint loss function and a perceptual loss function. The following Figure 2 This is the overall framework of the polarization and intensity image fusion method proposed in the present invention, which mainly includes a specific network architecture and a loss function for network training. The specific process of the fusion method is as follows:

[0073] 1. First, the polarization and intensity images are simultaneously input into the fusion network, and the corresponding source image features are obtained through the densely connected feature extraction encoder.

[0074] 2. Secondly, the extracted polarization image features and intensity image features are input into the multi-layer progressive attention perception fusion module for image fusion operation, and the fusion features are output after three layers of continuous attention perception guidance within the module.

[0075] 3. Then, the polarization image is fed into the polarization salient target highlighting mechanism, and the high-brightness polarization salient targets in the polarization image are mined based on intensity saliency and spatial compactness to obtain the polarization polarization target saliency map.

[0076] 4. Finally, the fusion features and polarization target saliency map output by the fusion module are spliced. The spliced ​​features are input into the image reconstruction decoder, and the fused image is reconstructed through multi-layer convolution processing.

[0077] In one embodiment, the densely connected feature extraction encoder in S200 includes a convolutional layer and two dense blocks, the convolutional layer uses ReLu as its activation function, and the dense block includes three consecutive convolutional layers, wherein the input of each convolutional layer is the concatenation of the output of the previous convolution and the output of all previous convolutional layers, and the polarization image and the intensity image are respectively input into the densely connected feature extraction encoder to obtain corresponding polarization image features and intensity image features.

[0078] Specifically, in the fusion network, the present invention uses a densely connected network to extract the feature information of the source image. The encoder extracts and splices feature information of different levels by introducing dense blocks in a densely connected manner, thereby avoiding the loss of valuable information in the feature extraction process, while reducing the number of network parameters and avoiding the overall fusion network from being too complicated. The specific structure of the encoder is as follows Figure 2As shown in the figure, it consists of a convolutional layer and two dense blocks, where the convolutional layer uses ReLu as its activation function. The dense block is composed of three consecutive convolutional layers, where the input of each convolutional layer is the concatenation of the output of the previous convolution and the output of all previous convolutional layers. This dense connection method can make full use of the features extracted by each convolution layer, avoid the loss of meaningful information, and alleviate the gradient explosion problem that may occur during network training to a certain extent.

[0079] Furthermore, polarization imaging devices can simultaneously obtain intensity information of the same object at different angles. , , , Represents the intensity information captured at these four different angles. Based on the intensity information obtained at the four angles, the necessary Stokes polarization parameters required for the Stokes vector representation method can be obtained through analysis and calculation. The specific calculation formula is as follows:

[0080] ;

[0081] in, In Indicates that the intensity information is from the polarization device Direction obtained, The first parameter of the Stokes vector method Represents the total intensity information of the captured light, usually called the intensity image, and the other two Stokes parameters and In addition to the above three Stokes parameters, there is also a parameter that represents the difference in circular polarization. , which is usually regarded as 0 in natural environments and does not participate in the calculation of polarization information.

[0082] Based on the Stokes parameters obtained above, the Stokes vector method can further obtain polarization information through analysis and calculation. The degree of linear polarization (DoLP) can be expressed as follows:

[0083] ;

[0084] The pixel value in the DoLP image is a constant in the range of 0 to 1, which represents the proportion of polarized light in the total intensity information of light.

[0085] In one embodiment, the multi-layer progressive attention perception fusion module in S300 is composed of three space-channel attention perception modules connected in connection. The polarization image features and the intensity image features are spliced ​​and input into the first space-channel attention perception module. The output of the first space-channel attention perception module is respectively multiplied element-by-element with the polarization image features and the intensity image features, and the features obtained are spliced ​​again and input into the second space-channel attention perception module. The output of the second space-channel attention perception module is respectively multiplied element-by-element with the polarization image features and the intensity image features, and the features obtained are spliced ​​again and input into the third space-channel attention perception module. The output of the third space-channel attention perception module is respectively multiplied element-by-element with the polarization image features and the intensity image features, and the features obtained are spliced ​​again to obtain the fusion features.

[0086] Specifically, in the network of the present invention, in order to effectively integrate the source image features extracted by the encoder and avoid the problems of insufficient utilization of source image feature information and excessive redundant information of fused features caused by simple fusion operations, a novel multi-layer progressive attention perception fusion module is designed. This fusion module combines the spatial attention mechanism and the channel attention mechanism to more effectively and fully mine the valuable information in the source image features, while avoiding the negative impact of some meaningless redundant information in the features on the fused features. For example, the low-intensity area in the polarization image contains too little valuable information, which will affect the visual quality of the final fused image. The specific structure of the designed multi-layer progressive attention perception fusion module is as follows: Figure 3 As shown in the figure, the module is composed of three sub-modules, namely the spatial-channel attention perception module, which uses the spatial attention mechanism and channel attention mechanism in the sub-module to perceive the input features and improve the utilization rate of the valuable information in the features. At the same time, the multi-layer progressive form can further enhance the learning ability of the fusion module, so that the final attention weight map can be more focused on extracting meaningful information in the features.

[0087] In one embodiment, each channel-spatial attention perception module includes a channel attention mechanism and a spatial attention mechanism. After the input features are input into the channel attention mechanism, the features obtained after maximum pooling and average pooling are performed on the channel dimension and then fed into the multi-layer perceptron for learning. The features output by the multi-layer perceptron are added element by element, and then multiplied element by element with the input features to obtain the channel attention features. The channel attention features are input into the spatial attention mechanism, and the features obtained after maximum pooling and average pooling are performed on the spatial dimension are concatenated and then convolved. The obtained features are then multiplied element by element with the channel attention features to obtain the channel-spatial attention features, and then converted into a channel-spatial attention weight map consistent with the input feature dimension through a Sigmoid activation function.

[0088] Specifically, in the channel-spatial attention perception (CSAP) module, the spatial attention mechanism and the channel attention mechanism are connected in series to enhance the perception of the extracted features. This method can ensure the adequacy and completeness of the perception of feature information. The specific structure of this module is as follows: Figure 4 As shown. The channel attention mechanism performs a pooling operation on the features in the channel dimension, obtains the weights corresponding to different channels and performs weighted calculation on the features based on this, guiding the network to focus on the value information of the input features in the channel dimension. Unlike the channel attention mechanism, the spatial attention mechanism focuses on feature information in the spatial dimension, enhances the information representation ability of the features based on the spatial position information of the input features, and highlights the salient targets in the features. In this module, the channel attention feature map is first obtained based on the channel attention mechanism, and then the channel attention features are further generated into channel-spatial attention features using the spatial attention mechanism. Finally, the obtained channel-spatial attention features are converted into a channel-spatial attention weight map consistent with the input feature dimension through the Sigmoid activation function. The specific calculation method of this module can be expressed as follows:

[0089] ;

[0090] in, represents the attention weight map of the x-th layer output, , represents the Sigmoid activation function, and They represent the spatial attention mechanism and channel attention mechanism respectively.

[0091] Furthermore, in the overall multi-layer progressive attention perception fusion module, the channel-spatial attention perception module of the previous layer obtains and outputs the channel-spatial attention weight map through calculation, and the extracted source image features are calculated based on the attention weight map to generate new extracted features. The new extracted features generated by the previous layer are fed to the channel-spatial attention perception module of the next layer as new input features after splicing, and the same operation as the previous layer is performed. Finally, the fusion result after three layers of attention perception guidance is obtained.

[0092] In one embodiment, the specific calculation process of the multi-layer progressive attention perception fusion module in S300 is expressed as:

[0093] ;

[0094] ;

[0095] ;

[0096] in, represents the feature concatenation operation, represents element-wise multiplication operation, represents the input features fed into the channel-spatial attention perception module, , the input features of the first layer are It is obtained by concatenating the extracted features of source images of different modalities. Represents the fusion features obtained at each layer, and the last layer Represents the fusion features finally output by the fusion module.

[0097] In one embodiment, S400 includes:

[0098] S410: Acquire an intensity saliency map of the polarization image by constructing a threshold truncation function;

[0099] S420: using a simple linear iterative clustering method to divide the saliency map into different regions, and calculating the center deviation of each region respectively;

[0100] S430: After obtaining the center deviation of each segmented area, quantify the center deviation of each segmented area using a half-Gaussian distribution;

[0101] S440: The spatial compactness map obtained by spatial compactness calculation is combined with the intensity saliency map to obtain a final polarization polarization target saliency map.

[0102] Specifically, according to the characteristics of source images of different modes, in the fusion process of polarization intensity images, the high-brightness polarization salient information of polarization images and the rich and clear texture details in intensity images are complementary information to be extracted and integrated. Considering that a completely consistent extraction network will ignore the differences between source images of different modes and cannot fully mine the complementary information in the source images, the present invention designs a polarization salient target highlighting mechanism to mine the high-brightness polarization salient information in polarization images, enhance the saliency of polarization polarization targets in fused images, and improve the visual quality of fused images. The pixel intensity of high-brightness polarization salient targets in polarization images has obvious distinguishability compared with other regions. Therefore, the salient target highlighting mechanism first extracts salient targets by weighing the intensity information of pixels in polarization images, that is, utilizing the intensity saliency of pixels in the target area. In this mechanism, based on the above-mentioned prior knowledge about polarization salient targets, the intensity saliency map of the polarization image is obtained by constructing a threshold truncation function commonly used in masks. In addition to obtaining the intensity saliency map by weighing the intensity information of pixels in polarization images, the mechanism also utilizes the spatial compactness between intensity pixels to further obtain the corresponding spatial structure information as a supplement. According to the distribution of high-brightness polarization salient objects in polarization images, the intensity pixels are often compact. Therefore, the spatial structural information of the salient objects in the intensity salient map can be further obtained by evaluating the spatial compactness of the salient objects in the intensity salient map as additional information supplement. In this mechanism, the spatial compactness is evaluated by the central deviation principle. First, the salient objects are divided into different regions using the simple linear iterative clustering method (SLIC), and then the central deviation of each region is calculated respectively; secondly, after obtaining the central deviation of each segmented region, it is quantified using the half-Gaussian distribution; finally, the spatial compact map obtained by the spatial compactness calculation is combined with the intensity salient map obtained in the previous step to obtain the final polarization polarization target salient map.

[0103] In one embodiment, S410 specifically includes:

[0104] ;

[0105] in, Represents the pixel intensity of the point at position (x,y) in the intensity saliency map, Represents the pixel intensity of the point at position (x,y) in the polarization image, It indicates the cutoff threshold;

[0106] S420 is specifically:

[0107] ;

[0108] ;

[0109] ;

[0110] ;

[0111] ;

[0112] in, represents the step length, and Respectively represent the total number of pixels and the number of cluster centers in the image. Represents the color distance between the pixel and the cluster center. Represents the spatial distance between the pixel and the cluster center point, represents the final distance metric, is a fixed trade-off parameter used to weigh the relative importance between color similarity and spatial proximity, Indicates the split The center deviation of the region, represents the centroid of the region in the x direction, represents the centroid of the region in the y direction, and Represent the height and width of the polarization image respectively;

[0113] S430 is specifically:

[0114] ;

[0115] in, Represents the spatial compact map of the current segmentation area, which represents the spatial structure information within the current segmentation area. Indicates the parameter for determining the spatial compactness;

[0116] S440 is specifically:

[0117] ;

[0118] in, Represents the final output polarization target saliency map, and Respectively indicate the corresponding Intensity saliency map and spatial compactness map of a region.

[0119] In one embodiment, the image reconstruction decoder in S500 includes four consecutive convolutional layers, each convolutional layer uses ReLu as an activation function, and the output channels of each convolution layer are 64, 32, 16, and 1 respectively. The fused features output from the multi-layer progressive attention perception fusion module are reconstructed into a single-channel fused image with significant polarization targets and rich and clear texture details via the image reconstruction decoder.

[0120] In one embodiment, the loss function based on the depth feature map in S600 includes an intensity-texture joint loss function and a perceptual loss function, specifically:

[0121] ;

[0122] in, is the hyperparameter used to balance the two non-class component loss functions, is the intensity-texture joint loss function, is the perceptual loss function.

[0123] Specifically, the loss functions used in the image fusion method are mostly based on the error analysis calculation of the final fused image and the source image, which does not fully utilize the features of different depths extracted by the network, resulting in insufficient sensitivity of the trained network to the feature information in the source image. At the same time, the loss function does not consider the differences between the intensity information and texture information of the source images of different modalities, and does not make reasonable weight allocation for different losses. Based on this, the present invention designs a loss function based on a depth feature map, which is composed of an intensity-texture joint loss function and a perceptual loss function.

[0124] The intensity-texture joint loss function is used to guide the pixel intensity of the fused image generated by the network to be consistent with the source image and contain rich and clear texture details in the source image. The intensity-texture joint loss calculates the corresponding balance weights based on feature maps of different depths, constraining the intensity similarity and texture similarity between the fused image and the source images of two different modalities. Specifically, when the pixel intensity information of the polarization image is more prominent than that of the intensity image, the balance weight assigned to the polarization image should be larger when calculating the intensity loss, and the same is true when calculating the texture loss function. According to the structure of the feature extraction encoder, it is divided into three layers, and the source image features of three different depths are obtained and used to calculate the corresponding balance weights, such as Figure 2 As shown. In addition to the intensity-texture joint loss function, the other part of the loss function is the perceptual loss function. If the loss function used in training the fusion network only calculates the pixel-level loss, that is, the intensity and texture loss, it is impossible to fully obtain the perceptual difference between the fused image and the source image. In order to avoid the limitation of the defects of the loss function on improving the fusion effect, the loss function design process of the present invention proposes to evaluate the perceptual loss based on the extracted features of different depths, enhance the sensitivity of the trained network to the feature information of the source image, and improve the fusion effect of the network. In the present invention, features of three different depths are involved in the calculation of the perceptual loss function.

[0125] In one embodiment, the intensity-texture joint loss function is specifically:

[0126] ;

[0127] ;

[0128] ;

[0129] ;

[0130] ;

[0131] ;

[0132] in, represents the exponential function operation, and represents the first polarization image extracted by the feature extraction encoder The features of the layer and the intensity image The characteristics of the layer, and represent polarization image and intensity image respectively, is a multiplication factor used to reduce or amplify the difference in pixel intensity of the source image. , is the balance parameter in texture loss, is the balance parameter in strength loss, and Represents the intermediate gradient information quantification index of polarization image and intensity image, , ,and Indicates the corresponding The length, width, and number of channels of the layer’s features, , and Indicates the length, width and number of channels of the corresponding source image. represents the sobel gradient operator, Hyperparameters to balance intensity loss and texture loss;

[0133] Specifically, after obtaining the quantization index of the intermediate gradient information of the source image, use The function performs normalization operation on it to obtain the balance parameters involved in the texture loss calculation , .

[0134] The perceptual loss function is specifically:

[0135] ;

[0136] in, is the fusion feature output by the multi-layer progressive attention perception fusion module, N represents the total number of layers of polarization image and intensity image, and represents the first polarization image extracted by the feature extraction encoder The features of the layer and the intensity image Layer features, there are three extracted features of different depths corresponding to the source image of each modality.

[0137] The present invention proposes a polarization and intensity image fusion method based on saliency mechanism and multi-layer attention perception. At the same time, the existing polarization and intensity image fusion methods usually ignore the differences in the information contained in the source image, and only use the same processing method or network structure to extract features from source images of different modalities, resulting in the inability to fully utilize the complementary information in the source image, thereby affecting the fusion effect. Secondly, the simple fusion operations adopted by the existing fusion methods, such as superposition and splicing, often fail to fully utilize the valuable information in the extracted features, and there is a possibility of causing excessive redundant information in the fusion features. In addition, most of the loss functions currently used in fusion network training only focus on the source image and the fused image, while neglecting the influence of the extracted features of different depths that can be obtained on the loss function in the back propagation and guidance of the network training process, resulting in insufficient sensitivity of the fusion network to the feature information, and the designed loss function does not consider the differences between the intensity information and texture information of the polarization image and the intensity image, and no reasonable weight distribution is performed on different losses in the calculation process, all of which will affect the fusion effect of the network. The main advantages of the present invention are as follows:

[0138] (1) This paper proposes a novel polarization and intensity image fusion method that is currently lacking in the field of image fusion. Based on the polarization salient target highlighting mechanism and multi-layer progressive attention perception, this method can achieve an ideal fusion effect of polarization information and texture details, and obtain a high-quality fused image;

[0139] (2) The fusion network of the present invention is an end-to-end trainable network, which can learn operations such as feature extraction, feature fusion and image reconstruction through training.

[0140] (3) The present invention uses a multi-layer progressive attention perception fusion module to perform feature fusion based on a more reasonable attention weight, which can fully explore the valuable information in the source image features and avoid the negative impact of excessive redundant information on the fused image, so that the fused image retains ideal intensity information and rich and clear texture details.

[0141] (4) By designing a polarization salient target highlighting mechanism, the present invention can more fully explore and utilize the polarization salient information in the polarization image, enhance the saliency of the polarization polarization target in the fusion result, and thus improve the overall contrast of the fused image.

[0142] (5) The present invention guides the training of the fusion network by designing a novel loss function based on the depth feature map, which is composed of the intensity-texture joint loss function and the perceptual loss function. The former is based on the balance weight parameter calculated by analyzing the feature maps of source images at different depths, guiding the fusion network to more appropriately fuse the intensity information and texture details from source images of different modalities. The latter enables the fusion network to fully consider the extracted features from different depths, improves the sensitivity of the fusion network to the information features in the source image, and avoids the loss of some feature information. At the same time, reasonable weight distribution is performed in the calculation process of the loss function, so as to fuse the intensity information and texture details from the source image in a more reasonable way, thereby achieving excellent fusion effect.

[0143] The above is a detailed introduction to the polarization and intensity image fusion method based on saliency mechanism and multi-layer attention perception provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A polarization and intensity image fusion method based on saliency mechanism and multi-layer attention perception, characterized in that: The method comprises the following steps: S100: Obtain polarization and intensity image datasets and build a fusion network, which includes a densely connected feature extraction encoder, a multi-layer progressive attention perception fusion module, and an image reconstruction decoder; S200: inputting the polarization and intensity images in the data set into the fusion network at the same time, and obtaining the corresponding polarization image features and intensity image features through the densely connected feature extraction encoder; S300: Inputting the extracted polarization image features and intensity image features into a multi-layer progressive attention perception fusion module for image fusion operation, and outputting fusion features after being guided by three layers of continuous attention perception inside the module; S400: feeding the polarization image into the polarization salient target highlighting mechanism, mining the polarization salient targets with high brightness in the polarization image based on intensity saliency and spatial compactness, and obtaining the polarization polarization target saliency map; S400 includes: S410: Acquire an intensity saliency map of the polarization image by constructing a threshold truncation function; S420: using a simple linear iterative clustering method to divide the saliency map into different regions, and calculating the center deviation of each region respectively; S430: After obtaining the center deviation of each segmented area, quantify the center deviation of each segmented area using a half-Gaussian distribution; S440: combining the spatial compact map obtained by spatial compactness calculation with the intensity saliency map to obtain a final polarization target saliency map; S410 is specifically: ； in, Represents the pixel intensity of the point at position (x,y) in the intensity saliency map, Represents the pixel intensity of the point at position (x,y) in the polarization image, It represents the cutoff threshold; S420 is specifically: ； ； ； ； ； in, represents the step length, Respectively represent the total number of pixels and the number of cluster centers in the image, Represents the color distance between the pixel and the cluster center. Represents the spatial distance between the pixel and the cluster center point, represents the final distance metric, is a fixed trade-off parameter used to weigh the relative importance between color similarity and spatial proximity, Indicates the split The center deviation of the region, represents the centroid of the region in the x direction, represents the centroid of the region in the y direction, Represent the height and width of the polarization image respectively; S430 is specifically: ； in, Represents the spatial compact map of the current segmentation area, which represents the spatial structure information within the current segmentation area. Indicates the parameter for determining the spatial compactness; S440 is specifically: ； in, Represents the final output polarization target saliency map, Respectively indicate the corresponding Intensity saliency map and spatial compactness map of regions; S500: performing a splicing operation on the fusion features output by the fusion module and the polarization polarization target saliency map, inputting the spliced ​​features into the image reconstruction decoder, and reconstructing the fused image through multi-layer convolution processing; S600: training a fusion network based on fusion features, fusion images, polarization images, intensity images, polarization image features, intensity image features and loss functions. When the loss function converges, a trained fusion network is obtained. Real-time polarization and intensity image fusion is performed based on the trained fusion network.

2. The method according to claim 1, characterized in that: The densely connected feature extraction encoder in S200 includes a convolutional layer and two dense blocks. The convolutional layer uses ReLu as its activation function. The dense block includes three consecutive convolutional layers, where the input of each convolutional layer is the concatenation of the output of the previous convolution and the output of all previous convolutional layers. The polarization image and intensity image are respectively input into the densely connected feature extraction encoder to obtain the corresponding polarization image features and intensity image features.

3. The method according to claim 2, characterized in that The multi-layer progressive attention perception fusion module in S300 is composed of three space-channel attention perception modules connected in series. The polarization image features and the intensity image features are spliced ​​and input into the first space-channel attention perception module. The output of the first space-channel attention perception module is respectively multiplied element-by-element with the polarization image features and the intensity image features and then spliced ​​and input into the second space-channel attention perception module. The output of the second space-channel attention perception module is respectively multiplied element-by-element with the polarization image features and the intensity image features and then spliced ​​and input into the third space-channel attention perception module. The output of the third space-channel attention perception module is respectively multiplied element-by-element with the polarization image features and the intensity image features and then spliced ​​to obtain the fused features.

4. The method according to claim 3, characterized in that Each channel-spatial attention perception module includes a channel attention mechanism and a spatial attention mechanism. After the input features are input into the channel attention mechanism, the features obtained after maximum pooling and average pooling are performed on the channel dimension and then fed into the multi-layer perceptron for learning. The features output by the multi-layer perceptron are added element by element, and then multiplied element by element with the input features to obtain the channel attention features. The channel attention features are input into the spatial attention mechanism, and the features obtained after maximum pooling and average pooling are performed on the spatial dimension. After concatenation, a convolution operation is performed, and the obtained features are multiplied element by element with the channel attention features to obtain the channel-spatial attention features, which are then converted into a channel-spatial attention weight map consistent with the input feature dimension through the Sigmoid activation function.

5. The method according to claim 3, characterized in that: The specific calculation process of the multi-layer progressive attention perception fusion module in S300 is expressed as: ； ； ； in, represents the feature concatenation operation, represents element-wise multiplication operation, represents the input features fed into the channel-spatial attention perception module, , the input features of the first layer are It is obtained by concatenating the extracted features of source images of different modalities. Represents the fusion features obtained at each layer, and the last layer Represents the fusion features finally output by the fusion module.

6. The method according to claim 5, characterized in that The image reconstruction decoder in S500 includes four consecutive convolutional layers, each of which uses ReLu as the activation function. The output channels of each convolution layer are 64, 32, 16, and 1 respectively. The fused features output from the multi-layer progressive attention perception fusion module are reconstructed into a single-channel fused image with significant polarization targets and rich and clear texture details through the image reconstruction decoder.

7. The method according to claim 6, characterized in that The loss function in S600 includes the intensity-texture joint loss function and the perceptual loss function, which are: ； in, is the hyperparameter used to balance the two non-class component loss functions, is the intensity-texture joint loss function, is the perceptual loss function.

8. The method according to claim 7, characterized in that The intensity-texture joint loss function is specifically: ； ； ； ； ； ； in, represents the exponential function operation, and represents the first polarization image extracted by the feature extraction encoder The features of the layer and the intensity image The characteristics of the layer, and represent polarization image and intensity image respectively, is a multiplication factor used to reduce or amplify the difference in pixel intensity of the source image. , is the balance parameter in texture loss, , is the balance parameter in strength loss, and Represents the intermediate gradient information quantification index of polarization image and intensity image, , and Indicates the corresponding The length, width, and number of channels of the layer’s features, , and Indicates the length, width and number of channels of the corresponding source image. represents the sobel gradient operator, Hyperparameters to balance intensity loss and texture loss; The perceptual loss function is specifically: ； in, is the fusion feature output by the multi-layer progressive attention perception fusion module, and N represents the total number of layers of polarization images and intensity images.

Images