SAR and visible light fusion method based on color reconstruction

Abstract

The invention relates to the field of image processing, in particular to an SAR and visible light fusion method based on color reconstruction. The technical problems are as follows: as the SAR image and the visible light image have essential differences in color expression and optical characteristics, direct pixel-level image fusion can cause serious pollution of color details in the visible light image by the gray value of the SAR image, and as the SAR image and the visible light image have differences in spatial resolution, acquisition viewing angle and the like, the color details in the visible light image can be seriously polluted by the gray value of the SAR image. Distortion is generated when fusion is carried out; according to the technical scheme, the SAR and visible light fusion method based on color reconstruction comprises the steps that an image fusion network used for fusing different-source images is constructed; according to the color reconstruction method based on the convolutional neural network, the color authenticity of the fused image is ensured, the color deviation caused by mismatching of color spaces is avoided, and the color and structure information is independently processed, so that the image quality is improved. And the definition of the structural information and the authenticity of the color information are well reserved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a SAR and visible light fusion method based on color reconstruction. Background Art

[0002] Image fusion is a technology in the field of computer vision and image processing, which aims to synthesize a new image from multiple source images so as to be better used in human-computer interface or subsequent computing processing; through fusion, the newly generated image is usually more representative or contains complementary information in the source images; image fusion can be used for homologous images (i.e., from the same type of sensor or imaging mode) or heterogeneous images (i.e., from different types of sensors or imaging modes); for heterogeneous images, due to the different types of sensors used to acquire the images, they are mainly divided into visible light sensors, infrared sensors, and synthetic aperture radar sensors; the fusion of visible light images and SAR images provides us with both color and detail and High-resolution geographic information images can even capture information outside the visible light range, which is attributed to the strong penetration ability of SAR sensors. When fusing SAR images and visible light images, a pixel-level image fusion method is used. Due to the essential differences between SAR and visible light images in color expression and optical characteristics, direct pixel-level image fusion will cause the color details in the visible light image to be seriously contaminated by the grayscale value of the SAR image, resulting in obvious color distortion. In addition, due to the differences between SAR and visible light images in spatial resolution and acquisition angle, fusion through simple registration homomorphism will cause the edge contour of the SAR to distort the local part of the visible light image. Summary of the invention

[0003] In order to overcome the problem that when fusing SAR images and visible light images, a pixel-level image fusion method is used, due to the essential differences between SAR and visible light images in color expression and optical characteristics, direct pixel-level image fusion will cause the color details in the visible light image to be seriously contaminated by the grayscale value of the SAR image, resulting in obvious color distortion. In addition, due to the differences between SAR and visible light images in spatial resolution and acquisition viewing angle, fusion through simple registration homomorphism will cause the edge contour of SAR to distort the local part of the visible light image.

[0004] The technical solution of the present invention is: a SAR and visible light fusion method based on color reconstruction, including constructing an image fusion network for fusing heterogeneous images, wherein the image fusion network includes a feature extraction layer, a feature fusion layer, a color reconstruction layer and a feature reconstruction layer, wherein the feature extraction layer is used to extract important feature information from the input visible light image and SAR image, the feature fusion layer is used to integrate the feature information extracted from the visible light image and the SAR image into a comprehensive representation, the color reconstruction layer is used to combine the converted color information with the fused feature map to reconstruct an output image with correct color information, and the feature reconstruction layer is used to reconstruct the fused feature map with color information.

[0005] Preferably, the structural information features of the visible light image and the SAR image are first extracted, and these features are integrated through concatenation and convolution operations. Subsequently, the color reconstruction layer first performs color space conversion, and then reconstructs the extracted color information features and the structural information feature map. Finally, the reconstruction layer is responsible for reconstructing the reconstructed feature map into the final output and saved color fusion image, which not only ensures the color authenticity of the fused image, but also maintains the structural details of the SAR image.

[0006] Preferably, by constructing a color reconstruction module, color features are directly extracted from the original visible light image, and the color features are combined with the fusion output through a convolutional neural network to obtain a color-rich fused image. This method not only ensures the color authenticity of the fused image, but also avoids color deviation caused by mismatch of color space. In addition, by independently processing color and structural information, it ensures that the clarity of structural information and the authenticity of color information are well preserved, effectively retaining the image color features and avoiding color distortion and distortion during the fusion process.

[0007] Preferably, the feature extraction layer includes a feature extraction convolution layer, a first residual block and a second residual block, the convolution kernel size of the feature extraction convolution layer is 3×3, the first residual block contains two convolution layers with a convolution kernel size of 3×3, and the second convolution layer contains two convolution layers with a convolution kernel size of 3×3, wherein the first layer of the two convolution layers in the first residual block and the second residual block is a batch normalization and ReLU layer, and the second layer of the two convolution layers in the first residual block and the second residual block is the output added to the input of ResBlock.

[0008] Preferably, the feature extraction layer includes the following steps when extracting important feature information from the input visible light image and SAR image:

[0009] S11: Use the feature extraction convolution layer to perform convolution operations on the input visible light image and SAR image, so as to enhance the features of the image and highlight the edge and texture information of the image;

[0010] S12: The visible light image and the SAR image processed by the feature extraction convolution layer are processed by two residual blocks, and the visible light image and the SAR image processed by the feature convolution layer are added to the original input to generate a feature map containing important feature information of the input image.

[0011] Preferably, the main task of the feature extraction layer is to extract important feature information from the input visible light image and SAR image to provide a basis for subsequent image fusion and processing; in this layer, we first use the convolution layer to perform a convolution operation on the image, and the convolution kernel size of this convolution layer is 3×3; the convolution operation can enhance the features of the image and highlight the edge and texture information of the image; next, we use two residual blocks ResBlocks to process the convolutional feature map; each residual block contains two convolution layers, the convolution kernel size is 3×3, the first layer is batch normalization and a ReLU layer, and the second layer is the output added to the input of ResBlock; the residual block is designed to prevent network degradation and improve the training speed and performance of the network; in the processing of the residual block, we first process the input feature map through the convolution layer and the activation function, and then add the processed feature map to the original input feature map, so that the original feature information can be retained and the loss of information can be prevented; finally, the output of the feature extraction layer is 16 feature maps generated after processing by the two residual blocks, which contain important feature information of the input image.

[0012] Preferably, the feature fusion layer includes a feature fusion convolution layer, the feature fusion convolution layer includes 16 filters of size 5×5, and the feature fusion layer includes the following steps when integrating the feature information extracted from the visible light image and the SAR image into a comprehensive representation:

[0013] S21: All feature maps extracted from the feature extraction layer are combined in series into one feature map;

[0014] S22: Use the feature fusion convolution layer to perform convolution processing on the feature map.

[0015] Preferably, the color reconstruction layer includes the following steps when combining the converted color information with the fusion feature map to reconstruct an output image with correct color information:

[0016] S31: color space conversion, converting the color space of the input image into another color space;

[0017] S32: Color reconstruction: the visible light image is converted through a color space conversion layer, and the obtained image is used as an input of a color reconstruction operation for color reconstruction.

[0018] Preferably, when converting the color space of the input image into another color space, the color reconstruction layer converts the input image in the RGB color space into an image in the HSV color space, wherein the specific steps are as follows:

[0019] S41: Scale the values ​​of the three primary colors of each pixel in the RGB color space from [0, 255] to [0, 1]:

[0020]

[0021] S42: Calculate the parameter value of each pixel in the HSV color space:

[0022]

[0023] V=max(R′,G′,B′).

[0024] Preferably, when the visible light image is converted through the color space conversion layer and the obtained image is used as the input of the color reconstruction operation for color reconstruction, three color reconstruction convolution layers are used for color reconstruction, and each of the three color reconstruction convolution layers is followed by a specific activation function. The function of the three color reconstruction convolution layers is to extract and process the information in the feature map, wherein the three color reconstruction convolution layers are respectively a first color reconstruction convolution layer, a second color reconstruction convolution layer and a third color reconstruction convolution layer.

[0025] Preferably, the calculation principle formula of the first color reconstruction convolutional layer is:

[0026] O1 = LeakyReLU((W1*I)+b1);

[0027] The calculation principle formula of the second color reconstruction convolutional layer is:

[0028] O2=LeakyReLU((W2*O1)+b2)

[0029] The calculation principle formula of the third color reconstruction convolutional layer is:

[0030] O3 = Sigmoid ((W3*O2) + b3);

[0031] Among them, 01, 02, 03 are the outputs of the first, second, and third convolutional layers respectively; W1, W2, W3 are the convolution kernels of the first, second, and third convolutional layers respectively; b1, b2, b3 are the biases of the first, second, and third convolutional layers respectively; * represents the convolution operation; I is the feature map of the visible light image after passing through the color conversion layer.

[0032] Preferably, the feature reconstruction layer includes three groups of independent reconstruction convolution blocks and a group of separate convolution layers. The main task of the feature reconstruction layer is to restore and reconstruct the output feature map of the feature fusion layer and the output feature map of the color reconstruction module, and reconstruct the final fused image step by step. The three groups of reconstruction convolution blocks are composed of convolution layers, normalization and ReLU activation functions.

[0033] As a preference, the fusion loss function of the image fusion network is:

[0034] L total =αL SSIM +βL color ;

[0035] Among them, α and β are weight parameters; L SSIM =(1-SSIM(I fusion , I SAR ))+(1-SSIM(I fusion , I VIS )), where I fusion is the fused image, I SAR is the SAR image, I VIS It is a VIS image; C fusion is the fused image, C visible It is a visible light image.

[0036] Beneficial effects of the present invention:

[0037] 1. Compared with the prior art, when fusing SAR images and visible light images, a pixel-level image fusion method is used for fusion. Since there are essential differences between SAR and visible light images in color expression and optical characteristics, direct pixel-level image fusion will cause the color details in the visible light image to be seriously polluted by the grayscale value of the SAR image, resulting in obvious color distortion. In addition, since there are differences between SAR and visible light images in terms of spatial resolution and acquisition viewing angle, fusion through simple registration homomorphism will cause the edge contour of the SAR to distort the local part of the visible light image. This fusion method directly extracts color features from the original visible light image by setting a color reconstruction method based on a convolutional neural network, and combines the color features with the fusion output through a convolutional neural network, thereby ensuring the color authenticity of the fused image and avoiding color deviation caused by mismatching of color space. In addition, by independently processing color and structural information, it is ensured that the clarity of structural information and the authenticity of color information are well preserved.

[0038] 2. By constructing a color reconstruction module, color features are directly extracted from the original visible light image, and the color features are combined with the fusion output through a convolutional neural network to obtain a colorful fusion image. This method not only ensures the color authenticity of the fusion image, but also avoids the color deviation caused by the mismatch of the color space. In addition, by independently processing the color and structure information, the clarity of the structure information and the authenticity of the color information are well preserved, the image color features are effectively preserved, and the color distortion and distortion in the fusion process are avoided;

[0039] 3. First, the structural information features of the visible light image and the SAR image are extracted, and these features are integrated through concatenation and convolution operations. Then, the color reconstruction layer performs color space conversion, and then reconstructs the extracted color information features and structural information feature maps. Finally, the reconstruction layer is responsible for reconstructing the reconstructed feature maps into the final output and saved color fusion image, which not only ensures the color authenticity of the fused image, but also maintains the structural details of the SAR image. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 What is shown is a schematic diagram of the construction of the image fusion network in the first stereoscopic construction of the SAR and visible light fusion method based on color reconstruction of the present invention;

[0041] Figure 2 Shown is a schematic diagram of the working steps of the color reconstruction layer in the SAR and visible light fusion method based on color reconstruction of the present invention;

[0042] Figure 3 Shown is a schematic diagram of some working steps of the color reconstruction layer in the SAR and visible light fusion method based on color reconstruction of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0044] See also Figure 1 The present invention provides an embodiment: a SAR and visible light fusion method based on color reconstruction, including constructing an image fusion network for fusing heterogeneous images, wherein the image fusion network includes a feature extraction layer, a feature fusion layer, a color reconstruction layer and a feature reconstruction layer, wherein the feature extraction layer is used to extract important feature information from the input visible light image and SAR image, the feature fusion layer is used to integrate the feature information extracted from the visible light image and the SAR image into a comprehensive representation, the color reconstruction layer is used to combine the converted color information with the fused feature map to reconstruct an output image with correct color information, and the feature reconstruction layer is used to reconstruct the fused feature map with color information.

[0045] Preferably, the structural information features of the visible light image and the SAR image are first extracted, and these features are integrated through concatenation and convolution operations. Subsequently, the color reconstruction layer first performs color space conversion, and then reconstructs the extracted color information features and the structural information feature map. Finally, the reconstruction layer is responsible for reconstructing the reconstructed feature map into the final output and saved color fusion image, which not only ensures the color authenticity of the fused image, but also maintains the structural details of the SAR image.

[0046] Preferably, by constructing a color reconstruction module, color features are directly extracted from the original visible light image, and the color features are combined with the fusion output through a convolutional neural network to obtain a color-rich fused image. This method not only ensures the color authenticity of the fused image, but also avoids color deviation caused by mismatch of color space. In addition, by independently processing color and structural information, it ensures that the clarity of structural information and the authenticity of color information are well preserved, effectively retaining the image color features and avoiding color distortion and distortion during the fusion process.

[0047] Preferably, the feature extraction layer includes a feature extraction convolution layer, a first residual block and a second residual block, the convolution kernel size of the feature extraction convolution layer is 3×3, the first residual block contains two convolution layers with a convolution kernel size of 3×3, and the second convolution layer contains two convolution layers with a convolution kernel size of 3×3, wherein the first layer of the two convolution layers in the first residual block and the second residual block is a batch normalization and ReLU layer, and the second layer of the two convolution layers in the first residual block and the second residual block is the output added to the input of ResBlock.

[0048] Preferably, the feature extraction layer includes the following steps when extracting important feature information from the input visible light image and SAR image:

[0049] S11: Use the feature extraction convolution layer to perform convolution operations on the input visible light image and SAR image, so as to enhance the features of the image and highlight the edge and texture information of the image;

[0050] S12: The visible light image and the SAR image processed by the feature extraction convolution layer are processed by two residual blocks, and the visible light image and the SAR image processed by the feature convolution layer are added to the original input to generate a feature map containing important feature information of the input image.

[0051] Preferably, the main task of the feature extraction layer is to extract important feature information from the input visible light image and SAR image to provide a basis for subsequent image fusion and processing; in this layer, we first use the convolution layer to perform a convolution operation on the image, and the convolution kernel size of this convolution layer is 3×3; the convolution operation can enhance the features of the image and highlight the edge and texture information of the image; next, we use two residual blocks ResBlocks to process the convolutional feature map; each residual block contains two convolution layers, the convolution kernel size is 3×3, the first layer is batch normalization and a ReLU layer, and the second layer is the output added to the input of ResBlock; the residual block is designed to prevent network degradation and improve the training speed and performance of the network; in the processing of the residual block, we first process the input feature map through the convolution layer and the activation function, and then add the processed feature map to the original input feature map, so that the original feature information can be retained and the loss of information can be prevented; finally, the output of the feature extraction layer is 16 feature maps generated after processing by the two residual blocks, which contain important feature information of the input image.

[0052] Preferably, the feature fusion layer includes a feature fusion convolution layer, the feature fusion convolution layer includes 16 filters of size 5×5, and the feature fusion layer includes the following steps when integrating the feature information extracted from the visible light image and the SAR image into a comprehensive representation:

[0053] S21: All feature maps extracted from the feature extraction layer are combined in series into one feature map;

[0054] S22: Use the feature fusion convolution layer to perform convolution processing on the feature map.

[0055] See also Figure 2-3 In this embodiment, the color reconstruction layer includes the following steps when combining the converted color information with the fusion feature map to reconstruct an output image with correct color information:

[0056] S31: color space conversion, converting the color space of the input image into another color space;

[0057] S32: Color reconstruction: the visible light image is converted through a color space conversion layer, and the obtained image is used as an input of a color reconstruction operation for color reconstruction.

[0058] Preferably, when converting the color space of the input image into another color space, the color reconstruction layer converts the input image in the RGB color space into an image in the HSV color space, wherein the specific steps are as follows:

[0059] S41: Scale the values ​​of the three primary colors of each pixel in the RGB color space from [0, 255] to [0, 1]:

[0060]

[0061] S42: Calculate the parameter value of each pixel in the HSV color space:

[0062]

[0063] V=max(R', G', B').

[0064] Preferably, when the visible light image is converted through the color space conversion layer and the obtained image is used as the input of the color reconstruction operation for color reconstruction, three color reconstruction convolution layers are used for color reconstruction, and each of the three color reconstruction convolution layers is followed by a specific activation function. The function of the three color reconstruction convolution layers is to extract and process the information in the feature map, wherein the three color reconstruction convolution layers are respectively a first color reconstruction convolution layer, a second color reconstruction convolution layer and a third color reconstruction convolution layer.

[0065] Preferably, the calculation principle formula of the first color reconstruction convolutional layer is:

[0066] O1 = LeakyReLU((W1*I)+b1);

[0067] The calculation principle formula of the second color reconstruction convolutional layer is:

[0068] O2=LeakyReLU((W2*O1)+b2)

[0069] The calculation principle formula of the third color reconstruction convolutional layer is:

[0070] O3 = Sigmoid ((W3*O2) + b3);

[0071] Among them, 01, 02, 03 are the outputs of the first, second, and third convolutional layers respectively; W1, W2, W3 are the convolution kernels of the first, second, and third convolutional layers respectively; b1, b2, b3 are the biases of the first, second, and third convolutional layers respectively; * represents the convolution operation; I is the feature map of the visible light image after passing through the color conversion layer.

[0072] Preferably, the feature reconstruction layer includes three groups of independent reconstruction convolution blocks and a group of separate convolution layers. The main task of the feature reconstruction layer is to restore and reconstruct the output feature map of the feature fusion layer and the output feature map of the color reconstruction module, and reconstruct the final fused image step by step. The three groups of reconstruction convolution blocks are composed of convolution layers, normalization and ReLU activation functions.

[0073] As a preference, the fusion loss function of the image fusion network is:

[0074] L total =αL SSIM +βL color ;

[0075] Among them, α and β are weight parameters; L SSIM =(1-SSIM(I fusion , I SAR ))+(1-SSIM(I fusion , I VIS )), where I fusion is the fused image, I SAR is the SAR image, I VIS It is a VIS image; C fusion is the fused image, C visible It is a visible light image.

[0076] Through the above steps, compared with the prior art, when fusing SAR images and visible light images, a pixel-level image fusion method is used for fusion. Since there are essential differences between SAR and visible light images in color expression and optical characteristics, direct pixel-level image fusion will cause the color details in the visible light image to be seriously polluted by the grayscale value of the SAR image, resulting in obvious color distortion. Moreover, since there are differences between SAR and visible light images in terms of spatial resolution and acquisition viewing angle, fusion through simple registration homomorphism will cause the edge contour of the SAR to distort the local part of the visible light image. The fusion method directly extracts color features from the original visible light image by setting a color reconstruction method based on a convolutional neural network, and combines the color features with the fusion output through a convolutional neural network, thereby ensuring the color authenticity of the fused image and avoiding the color deviation caused by the mismatch of the color space. Moreover, by independently processing the color and structural information, it is ensured that the clarity of the structural information and the authenticity of the color information are well preserved.

[0077] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of those skilled in the art without departing from the spirit of the present invention.

Claims

1. SAR and visible light fusion method based on color reconstruction; characterized by: The invention comprises constructing an image fusion network for fusing heterogeneous images, wherein the image fusion network comprises a feature extraction layer, a feature fusion layer, a color reconstruction layer and a feature reconstruction layer, wherein the feature extraction layer is used to extract important feature information from the input visible light image and SAR image, the feature fusion layer is used to integrate the feature information extracted from the visible light image and the SAR image into a comprehensive representation, the color reconstruction layer is used to combine the converted color information with the fused feature map to reconstruct an output image with correct color information, and the feature reconstruction layer is used to reconstruct the fused feature map with color information.

2. The SAR and visible light fusion method based on color reconstruction according to claim 1, characterized in that: The feature extraction layer includes a feature extraction convolution layer, a first residual block, and a second residual block. The convolution kernel size of the feature extraction convolution layer is 3×3. The first residual block contains two convolution layers with a convolution kernel size of 3×3. The second convolution layer contains two convolution layers with a convolution kernel size of 3×3. Among them, the first layer of the two convolution layers in the first residual block and the second residual block is a batch normalization and ReLU layer, and the second layer of the two convolution layers in the first residual block and the second residual block is the output added to the input of ResBlock.

3. The SAR and visible light fusion method based on color reconstruction according to claim 2, characterized in that: The feature extraction layer extracts important feature information from the input visible light image and SAR image, including the following steps: S11: Use the feature extraction convolution layer to perform convolution operations on the input visible light image and SAR image, so as to enhance the features of the image and highlight the edge and texture information of the image; S12: The visible light image and the SAR image processed by the feature extraction convolution layer are processed by two residual blocks, and the visible light image and the SAR image processed by the feature convolution layer are added to the original input to generate a feature map containing important feature information of the input image.

4. The SAR and visible light fusion method based on color reconstruction according to claim 3, characterized in that: The feature fusion layer includes a feature fusion convolution layer, which contains 16 filters of size 5×5. The feature fusion layer integrates the feature information extracted from the visible light image and the SAR image into a comprehensive representation, including the following steps: S21: All feature maps extracted from the feature extraction layer are combined in series into one feature map; S22: Use the feature fusion convolution layer to perform convolution processing on the feature map.

5. The SAR and visible light fusion method based on color reconstruction according to claim 4, characterized in that: The color reconstruction layer combines the converted color information with the fusion feature map to reconstruct an output image with correct color information, including the following steps: S31: color space conversion, converting the color space of the input image into another color space; S32: Color reconstruction: the visible light image is converted through a color space conversion layer, and the obtained image is used as an input of a color reconstruction operation for color reconstruction.

6. The SAR and visible light fusion method based on color reconstruction according to claim 5, characterized in that: When the color reconstruction layer converts the color space of the input image into another color space, the image in the input RGB color space is converted into an image in the HSV color space, wherein the specific steps are as follows: S41: Scale the values ​​of the three primary colors of each pixel in the RGB color space from [0, 255] to [0, 1]: S42: Calculate the parameter value of each pixel in the HSV color space:

7. The SAR and visible light fusion method based on color reconstruction according to claim 6, characterized in that: When the visible light image is converted through the color space conversion layer and the obtained image is used as the input of the color reconstruction operation for color reconstruction, three color reconstruction convolution layers are used for color reconstruction, and each of the three color reconstruction convolution layers is followed by a specific activation function. The function of the three color reconstruction convolution layers is to extract and process the information in the feature map, wherein the three color reconstruction convolution layers are respectively the first color reconstruction convolution layer, the second color reconstruction convolution layer and the third color reconstruction convolution layer.

8. The SAR and visible light fusion method based on color reconstruction according to claim 7, characterized in that: The calculation principle formula of the first color reconstruction convolutional layer is: O1 = LeakyReLU((W1*I)+b1); The calculation principle formula of the second color reconstruction convolutional layer is: O2=LeakyReLU((W2*O1)+b2) The calculation principle formula of the third color reconstruction convolutional layer is: O3 = Sigmoid ((W3*O2) + b3); Among them, 01, 02, 03 are the outputs of the first, second, and third convolutional layers respectively; W1, W2, W3 are the convolution kernels of the first, second, and third convolutional layers respectively; b1, b2, b3 are the biases of the first, second, and third convolutional layers respectively; * represents the convolution operation; I is the feature map of the visible light image after passing through the color conversion layer.

9. The SAR and visible light fusion method based on color reconstruction according to claim 8, characterized in that: The feature reconstruction layer includes three groups of independent reconstruction convolution blocks and a group of separate convolution layers. The main task of the feature reconstruction layer is to restore and reconstruct the output feature map of the feature fusion layer and the output feature map of the color reconstruction module, and reconstruct the final fused image step by step. The three groups of reconstruction convolution blocks are composed of convolution layers, normalization and ReLU activation functions.

10. The SAR and visible light fusion method based on color reconstruction according to claim 9, characterized in that: The fusion loss function of the image fusion network is: L total =αL SSIM +βL color ; Among them, α and β are weight parameters; L SSIM =(1-SSIM(I fusion , I SAR ))+(1-SSIM(I fusion , I VIS )), where I fusion is the fused image, I SAR is the SAR image, I VIS It is a VIS image; C fusion is the fused image, C visible It is a visible light image.

Images