SAR and visible light fusion method based on color reconstruction

Abstract

The present invention relates to the field of image processing, and in particular to a SAR and visible light fusion method based on color reconstruction. Technical problem: Due to the essential differences between SAR and visible light images in color expression and optical characteristics, direct pixel-level image fusion will cause the color details in the visible light image to be seriously contaminated by the grayscale value of the SAR image. Moreover, due to the differences between SAR and visible light images in spatial resolution, acquisition viewing angle, etc., fusion will produce distortion. Technical solution: A SAR and visible light fusion method based on color reconstruction includes constructing an image fusion network for fusing heterogeneous images. The present invention ensures the color authenticity of the fused image by providing a color reconstruction method based on a convolutional neural network, avoids color deviation caused by color space mismatch, and ensures that both the clarity of the structural information and the authenticity of the color information are well preserved by independently processing the color and structural information.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a SAR and visible light fusion method based on color reconstruction. Background Art

[0002] Image fusion is a technology in the field of computer vision and image processing, which aims to synthesize a new image from multiple source images so as to be better used in human-computer interface or subsequent computational processing. Through fusion, the newly generated image is usually more representative or contains complementary information in the source images. Image fusion can be used for homologous images (i.e., from the same type of sensor or imaging mode) or heterologous images (i.e., from different types of sensors or imaging modes). For heterologous images, due to the different types of sensors that acquire the images, they are mainly divided into visible light sensors, infrared sensors, and synthetic aperture radar sensors. The fusion of visible light images and SAR images provides us with both color and details. Images with high-resolution geographic information can even capture information beyond the visible light range, which is attributed to the strong penetration ability of SAR sensors. When fusing SAR images and visible light images, a pixel-level image fusion method is used. Due to the essential differences between SAR and visible light images in color expression and optical characteristics, direct pixel-level image fusion will cause the color details in the visible light image to be seriously contaminated by the grayscale value of the SAR image, resulting in obvious color distortion. In addition, due to the differences between SAR and visible light images in spatial resolution, acquisition angle, etc., fusion through simple registration homomorphism will cause the edge contour of the SAR to distort the local part of the visible light image. Summary of the Invention

[0003] In order to overcome the problem of pixel-level image fusion method being used to fuse SAR images and visible light images, due to the essential differences between SAR and visible light images in color expression and optical characteristics, direct pixel-level image fusion will cause the color details in the visible light image to be seriously contaminated by the grayscale value of the SAR image, resulting in obvious color distortion. In addition, due to the differences between SAR and visible light images in spatial resolution, acquisition angle of view, etc., fusion through simple registration homomorphism will cause the edge contour of the SAR to distort the local part of the visible light image.

[0004] The technical solution of the present invention is: a SAR and visible light fusion method based on color reconstruction, including constructing an image fusion network for fusing heterogeneous images, wherein the image fusion network includes a feature extraction layer, a feature fusion layer, a color reconstruction layer and a feature reconstruction layer, wherein the feature extraction layer is used to extract important feature information from the input visible light image and SAR image, the feature fusion layer is used to integrate the feature information extracted from the visible light image and the SAR image into a comprehensive representation, the color reconstruction layer is used to combine the converted color information with the fusion feature map to reconstruct an output image with correct color information, and the feature reconstruction layer is used to reconstruct the fusion feature map with color information.

[0005] Preferably, the structural information features of the visible light image and the SAR image are first extracted, and these features are integrated through concatenation and convolution operations. Subsequently, the color reconstruction layer first performs color space conversion, and then reconstructs the extracted color information features and the structural information feature map. Finally, the reconstruction layer is responsible for reconstructing the reconstructed feature map into the final output and saved color fusion image, which not only ensures the color authenticity of the fusion image, but also maintains the structural details of the SAR image.

[0006] Preferably, by constructing a color reconstruction module, color features are directly extracted from the original visible light image, and the color features are combined with the fusion output through a convolutional neural network to obtain a color-rich fusion image. This method not only ensures the color authenticity of the fusion image, but also avoids color deviation caused by color space mismatch. In addition, by independently processing color and structural information, the clarity of the structural information and the authenticity of the color information are well preserved, the image color features are effectively preserved, and the distortion and distortion of the color during the fusion process are avoided.

[0007] Preferably, the feature extraction layer includes a feature extraction convolution layer, a first residual block and a second residual block, the convolution kernel size of the feature extraction convolution layer is 3×3, the first residual block contains two convolution layers with a convolution kernel size of 3×3, and the second convolution layer contains two convolution layers with a convolution kernel size of 3×3, wherein the first layer of the two convolution layers in the first residual block and the second residual block is a batch normalization and ReLU layer, and the second layer of the two convolution layers in the first residual block and the second residual block is the output added to the input of ResBlock.

[0008] Preferably, the feature extraction layer includes the following steps when extracting important feature information from the input visible light image and SAR image:

[0009] S11: Use the feature extraction convolution layer to perform convolution operations on the input visible light image and SAR image to enhance the image features and highlight the image edges and texture information;

[0010] S12: The visible light image and the SAR image processed by the feature extraction convolution layer are processed using two residual blocks, and the visible light image and the SAR image processed by the feature convolution layer are added to the original input to generate a feature map containing important feature information of the input image.

[0011] Preferably, the main task of the feature extraction layer is to extract important feature information from the input visible light image and SAR image to provide a basis for subsequent image fusion and processing; in this layer, we first use the convolution layer to perform a convolution operation on the image, and the convolution kernel size of this convolution layer is 3×3; the convolution operation can enhance the features of the image and highlight the edge and texture information of the image; next, we use two residual blocks ResBlocks to process the convolved feature map; each residual block contains two convolution layers with a convolution kernel size of 3×3. The first layer is batch normalization and a ReLU layer, and the second layer is the output added to the input of ResBlock; the purpose of the design of the residual block is to prevent network degradation and improve the training speed and performance of the network; in the processing of the residual block, we first process the input feature map through the convolution layer and activation function, and then add the processed feature map to the original input feature map, so that the original feature information can be retained and information loss can be prevented; finally, the output of the feature extraction layer is 16 feature maps generated after processing by the two residual blocks, which contain important feature information of the input image.

[0012] Preferably, the feature fusion layer includes a feature fusion convolution layer, the feature fusion convolution layer includes 16 filters of size 5×5, and the feature fusion layer includes the following steps when integrating the feature information extracted from the visible light image and the SAR image into a comprehensive representation:

[0013] S21: All feature maps extracted from the feature extraction layer are serially merged into one feature map;

[0014] S22: Use the feature fusion convolution layer to perform convolution processing on the feature map.

[0015] Preferably, the color reconstruction layer comprises the following steps when combining the converted color information with the fusion feature map to reconstruct an output image with correct color information:

[0016] S31: color space conversion, converting the color space of the input image into another color space;

[0017] S32: Color reconstruction: the visible light image is converted through a color space conversion layer, and the obtained image is used as the input of the color reconstruction operation for color reconstruction.

[0018] Preferably, when converting the color space of the input image into another color space, the color reconstruction layer converts the input RGB color space image into an HSV color space image, wherein the specific steps are as follows:

[0019] S41: Scale the values ​​of the three primary colors of each pixel in the RGB color space from [0, 255] to [0, 1]:

[0020]

[0021] S42: Calculate the parameter value of each pixel in the HSV color space:

[0022]

[0023] V=max(R′, G′, B′).

[0024] Preferably, when the visible light image is converted through the color space conversion layer and the obtained image is used as the input of the color reconstruction operation for color reconstruction, three color reconstruction convolution layers are used for color reconstruction, and each of the three color reconstruction convolution layers is followed by a specific activation function. The function of the three color reconstruction convolution layers is to extract and process the information in the feature map, wherein the three color reconstruction convolution layers are respectively the first color reconstruction convolution layer, the second color reconstruction convolution layer and the third color reconstruction convolution layer.

[0025] Preferably, the calculation principle formula of the first color reconstruction convolutional layer is:

[0026] O1 = LeakyReLU((W1*I)+b1);

[0027] The calculation principle formula of the second color reconstruction convolutional layer is:

[0028] O2=LeakyReLU((W2*O1)+b2)

[0029] The calculation principle formula of the third color reconstruction convolutional layer is:

[0030] O3=Sigmoid((W3*O2)+b3);

[0031] Among them, 01, 02, and 03 are the outputs of the first, second, and third convolutional layers respectively; W1, W2, and W3 are the convolution kernels of the first, second, and third convolutional layers respectively; b1, b2, and b3 are the biases of the first, second, and third convolutional layers respectively; * represents the convolution operation; I is the feature map of the visible light image after passing through the color conversion layer.

[0032] Preferably, the feature reconstruction layer includes three groups of independent reconstruction convolution blocks and a group of separate convolution layers. The main task of the feature reconstruction layer is to restore and reconstruct the output feature map of the feature fusion layer and the output feature map of the color reconstruction module, and reconstruct the final fusion image step by step. The three groups of reconstruction convolution blocks are composed of convolution layers, normalization and ReLU activation functions.

[0033] As a preference, the fusion loss function of the image fusion network is:

[0034] L total =αL SSIM +βL color ;

[0035] Among them, α and β are weight parameters; L SSIM =(1-SSIM(I fusion , I SAR ))+(1-SSIM(I fusion , I VIS )), where I fusion is the fused image, I SAR is the SAR image, I VIS It is a VIS image; C fusion is the fused image, C visible For visible light images.

[0036] Beneficial effects of the present invention:

[0037] 1. Compared with the existing technology, when fusing SAR images and visible light images, pixel-level image fusion methods are used. Due to the essential differences between SAR and visible light images in color expression and optical characteristics, direct pixel-level image fusion will cause the color details in the visible light image to be seriously contaminated by the grayscale values ​​of the SAR image, resulting in obvious color distortion. In addition, due to the differences between SAR and visible light images in spatial resolution and acquisition angle, simple registration homomorphic fusion will cause the edge contours of the SAR to distort the local part of the visible light image. This fusion method directly extracts color features from the original visible light image by setting a color reconstruction method based on a convolutional neural network, and combines the color features with the fusion output through a convolutional neural network, thereby ensuring the color authenticity of the fused image and avoiding color deviation caused by color space mismatch. By independently processing color and structural information, the clarity of structural information and the authenticity of color information are well preserved.

[0038] 2. By building a color reconstruction module, we extract color features directly from the original visible light image and combine the color features with the fusion output through a convolutional neural network to obtain a colorful fused image. This method not only ensures the color authenticity of the fused image, but also avoids color deviation caused by color space mismatch. In addition, by independently processing color and structural information, the clarity of structural information and the authenticity of color information are well preserved, effectively preserving the image color characteristics and avoiding color distortion and distortion during the fusion process.

[0039] 3. First, the structural information features of the visible light image and the SAR image are extracted, and these features are integrated through concatenation and convolution operations. Subsequently, the color reconstruction layer performs color space conversion and then reconstructs the extracted color information features and the structural information feature map. Finally, the reconstruction layer is responsible for reconstructing the reconstructed feature map into the final output and saved color fusion image. This not only ensures the color authenticity of the fused image, but also maintains the structural details of the SAR image. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Shown is a schematic diagram of the structure of the image fusion network in the first stereoscopic structure of the SAR and visible light fusion method based on color reconstruction of the present invention;

[0041] Figure 2 Shown is a schematic diagram of the working steps of the color reconstruction layer in the SAR and visible light fusion method based on color reconstruction of the present invention;

[0042] Figure 3 Shown is a schematic diagram of some working steps of the color reconstruction layer in the SAR and visible light fusion method based on color reconstruction of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below with reference to the accompanying drawings and examples.

[0044] See also Figure 1 The present invention provides an embodiment: a SAR and visible light fusion method based on color reconstruction, including constructing an image fusion network for fusing heterogeneous images, wherein the image fusion network includes a feature extraction layer, a feature fusion layer, a color reconstruction layer and a feature reconstruction layer, wherein the feature extraction layer is used to extract important feature information from the input visible light image and SAR image, the feature fusion layer is used to integrate the feature information extracted from the visible light image and the SAR image into a comprehensive representation, the color reconstruction layer is used to combine the converted color information with the fused feature map to reconstruct an output image with correct color information, and the feature reconstruction layer is used to reconstruct the fused feature map with color information.

[0045] Preferably, the structural information features of the visible light image and the SAR image are first extracted, and these features are integrated through concatenation and convolution operations. Subsequently, the color reconstruction layer first performs color space conversion, and then reconstructs the extracted color information features and the structural information feature map. Finally, the reconstruction layer is responsible for reconstructing the reconstructed feature map into the final output and saved color fusion image, which not only ensures the color authenticity of the fusion image, but also maintains the structural details of the SAR image.

[0046] Preferably, by constructing a color reconstruction module, color features are directly extracted from the original visible light image, and the color features are combined with the fusion output through a convolutional neural network to obtain a color-rich fusion image. This method not only ensures the color authenticity of the fusion image, but also avoids color deviation caused by color space mismatch. In addition, by independently processing color and structural information, the clarity of the structural information and the authenticity of the color information are well preserved, the image color features are effectively preserved, and the distortion and distortion of the color during the fusion process are avoided.

[0047] Preferably, the feature extraction layer includes a feature extraction convolution layer, a first residual block and a second residual block, the convolution kernel size of the feature extraction convolution layer is 3×3, the first residual block contains two convolution layers with a convolution kernel size of 3×3, and the second convolution layer contains two convolution layers with a convolution kernel size of 3×3, wherein the first layer of the two convolution layers in the first residual block and the second residual block is a batch normalization and ReLU layer, and the second layer of the two convolution layers in the first residual block and the second residual block is the output added to the input of ResBlock.

[0048] Preferably, the feature extraction layer includes the following steps when extracting important feature information from the input visible light image and SAR image:

[0049] S11: Use the feature extraction convolution layer to perform convolution operations on the input visible light image and SAR image to enhance the image features and highlight the image edges and texture information;

[0050] S12: The visible light image and the SAR image processed by the feature extraction convolution layer are processed using two residual blocks, and the visible light image and the SAR image processed by the feature convolution layer are added to the original input to generate a feature map containing important feature information of the input image.

[0051] Preferably, the main task of the feature extraction layer is to extract important feature information from the input visible light image and SAR image to provide a basis for subsequent image fusion and processing; in this layer, we first use the convolution layer to perform a convolution operation on the image, and the convolution kernel size of this convolution layer is 3×3; the convolution operation can enhance the features of the image and highlight the edge and texture information of the image; next, we use two residual blocks ResBlocks to process the convolved feature map; each residual block contains two convolution layers with a convolution kernel size of 3×3. The first layer is batch normalization and a ReLU layer, and the second layer is the output added to the input of ResBlock; the purpose of the design of the residual block is to prevent network degradation and improve the training speed and performance of the network; in the processing of the residual block, we first process the input feature map through the convolution layer and activation function, and then add the processed feature map to the original input feature map, so that the original feature information can be retained and information loss can be prevented; finally, the output of the feature extraction layer is 16 feature maps generated after processing by the two residual blocks, which contain important feature information of the input image.

[0052] Preferably, the feature fusion layer includes a feature fusion convolution layer, the feature fusion convolution layer includes 16 filters of size 5×5, and the feature fusion layer includes the following steps when integrating the feature information extracted from the visible light image and the SAR image into a comprehensive representation:

[0053] S21: All feature maps extracted from the feature extraction layer are serially merged into one feature map;

[0054] S22: Use the feature fusion convolution layer to perform convolution processing on the feature map.

[0055] See also Figure 2-3 In this embodiment, the color reconstruction layer combines the converted color information with the fusion feature map to reconstruct an output image with correct color information, including the following steps:

[0056] S31: color space conversion, converting the color space of the input image into another color space;

[0057] S32: Color reconstruction: the visible light image is converted through a color space conversion layer, and the obtained image is used as the input of the color reconstruction operation for color reconstruction.

[0058] Preferably, when converting the color space of the input image into another color space, the color reconstruction layer converts the input RGB color space image into an HSV color space image, wherein the specific steps are as follows:

[0059] S41: Scale the values ​​of the three primary colors of each pixel in the RGB color space from [0, 255] to [0, 1]:

[0060]

[0061] S42: Calculate the parameter value of each pixel in the HSV color space:

[0062]

[0063] V=max(R', G', B').

[0064] Preferably, when the visible light image is converted through the color space conversion layer and the obtained image is used as the input of the color reconstruction operation for color reconstruction, three color reconstruction convolution layers are used for color reconstruction, and each of the three color reconstruction convolution layers is followed by a specific activation function. The function of the three color reconstruction convolution layers is to extract and process the information in the feature map, wherein the three color reconstruction convolution layers are respectively the first color reconstruction convolution layer, the second color reconstruction convolution layer and the third color reconstruction convolution layer.

[0065] Preferably, the calculation principle formula of the first color reconstruction convolutional layer is:

[0066] O1 = LeakyReLU((W1*I)+b1);

[0067] The calculation principle formula of the second color reconstruction convolutional layer is:

[0068] O2=LeakyReLU((W2*O1)+b2)

[0069] The calculation principle formula of the third color reconstruction convolutional layer is:

[0070] O3=Sigmoid((W3*O2)+b3);

[0071] Among them, 01, 02, and 03 are the outputs of the first, second, and third convolutional layers respectively; W1, W2, and W3 are the convolution kernels of the first, second, and third convolutional layers respectively; b1, b2, and b3 are the biases of the first, second, and third convolutional layers respectively; * represents the convolution operation; I is the feature map of the visible light image after passing through the color conversion layer.

[0072] Preferably, the feature reconstruction layer includes three groups of independent reconstruction convolution blocks and a group of separate convolution layers. The main task of the feature reconstruction layer is to restore and reconstruct the output feature map of the feature fusion layer and the output feature map of the color reconstruction module, and reconstruct the final fusion image step by step. The three groups of reconstruction convolution blocks are composed of convolution layers, normalization and ReLU activation functions.

[0073] As a preference, the fusion loss function of the image fusion network is:

[0074] L total =αL SSIM +βL color ;

[0075] Among them, α and β are weight parameters; L SSIM =(1-SSIM(I fusion , I SAR ))+(1-SSIM(I fusion , I VIS )), where I fusion is the fused image, I SAR is the SAR image, I VIS It is a VIS image; C fusion is the fused image, C visible For visible light images.

[0076] Through the above steps, compared with the prior art, when fusing SAR images and visible light images, a pixel-level image fusion method is used. Due to the essential differences between SAR and visible light images in color expression and optical characteristics, direct pixel-level image fusion will cause the color details in the visible light image to be severely contaminated by the grayscale values ​​of the SAR image, resulting in obvious color distortion. In addition, due to the differences between SAR and visible light images in spatial resolution and acquisition angle, fusion through simple registration homomorphism will cause the edge contours of the SAR to distort parts of the visible light image. This fusion method directly extracts color features from the original visible light image by setting a color reconstruction method based on a convolutional neural network, and combines the color features with the fusion output through a convolutional neural network, thereby ensuring the color authenticity of the fused image and avoiding color deviation caused by color space mismatch. By independently processing color and structural information, both the clarity of structural information and the authenticity of color information are well preserved.

[0077] The embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the scope of knowledge of those skilled in the art without departing from the spirit of the present invention.

Claims

1. SAR and visible light fusion method based on color reconstruction; characterized by: The invention comprises constructing an image fusion network for fusing heterogeneous images, wherein the image fusion network comprises a feature extraction layer, a feature fusion layer, a color reconstruction layer and a feature reconstruction layer, wherein the feature extraction layer is used to extract important feature information from the input visible light image and the SAR image, the feature fusion layer is used to integrate the feature information extracted from the visible light image and the SAR image into a comprehensive representation, the color reconstruction layer is used to combine the converted color information with the fusion feature map to reconstruct an output image with correct color information, and the feature reconstruction layer is used to reconstruct the fusion feature map with color information; The color reconstruction layer combines the converted color information with the fused feature map to reconstruct an output image with correct color information, including the following steps: S31: color space conversion, converting the color space of the input image into another color space; S32: Color reconstruction: the visible light image is converted through the color space conversion layer, and the obtained image is used as the input of the color reconstruction operation for color reconstruction; When the color reconstruction layer converts the color space of the input image into another color space, it converts the input RGB color space image into the HSV color space image. The specific steps are as follows: S41: Scale the values ​​of the three primary colors of each pixel in the RGB color space from [0,255] to [0,1]: S42: Calculate the parameter value of each pixel in the HSV color space: When the visible light image is converted through the color space conversion layer and the obtained image is used as the input of the color reconstruction operation for color reconstruction, three color reconstruction convolution layers are used for color reconstruction, and each of the three color reconstruction convolution layers is followed by an activation function. The function of the three color reconstruction convolution layers is to extract and process the information in the feature map. Among them, the three color reconstruction convolution layers are the first color reconstruction convolution layer, the second color reconstruction convolution layer, and the third color reconstruction convolution layer. The calculation principle formula of the first color reconstruction convolutional layer is: O1 = LeakyReLU((W1*I)+b1); The calculation principle formula of the second color reconstruction convolutional layer is: O2=LeakyReLU((W2*O1)+b2) The calculation principle formula of the third color reconstruction convolutional layer is: O3=Sigmoid((W3*O2)+b3); Among them, O1, O2, O3 are the outputs of the first, second, and third convolutional layers respectively; W1, W2, W3 are the convolution kernels of the first, second, and third convolutional layers respectively; b1, b2, b3 are the biases of the first, second, and third convolutional layers respectively; * represents the convolution operation; I is the feature map of the visible light image after passing through the color conversion layer.

2. The SAR and visible light fusion method based on color reconstruction according to claim 1, characterized in that: The feature extraction layer includes a feature extraction convolution layer, a first residual block, and a second residual block. The convolution kernel size of the feature extraction convolution layer is 3×3. The first residual block contains two convolution layers with a convolution kernel size of 3×3. The second convolution layer contains two convolution layers with a convolution kernel size of 3×3. Among them, the first layer of the two convolution layers in the first residual block and the second residual block is a batch normalization and ReLU layer, and the second layer of the two convolution layers in the first residual block and the second residual block is the output added to the input of ResBlock.

3. The SAR and visible light fusion method based on color reconstruction according to claim 2, characterized in that: The feature extraction layer extracts important feature information from the input visible light image and SAR image, including the following steps: S11: Use the feature extraction convolution layer to perform convolution operations on the input visible light image and SAR image to enhance the image features and highlight the image edges and texture information; S12: The visible light image and the SAR image processed by the feature extraction convolution layer are processed using two residual blocks, and the visible light image and the SAR image processed by the feature convolution layer are added to the original input to generate a feature map containing important feature information of the input image.

4. The SAR and visible light fusion method based on color reconstruction according to claim 3, characterized in that: The feature fusion layer includes a feature fusion convolution layer, which contains 16 filters of size 5×5. The feature fusion layer integrates the feature information extracted from the visible light image and the SAR image into a comprehensive representation, including the following steps: S21: All feature maps extracted from the feature extraction layer are serially combined into one feature map; S22: Use the feature fusion convolution layer to perform convolution processing on the feature map.

5. The SAR and visible light fusion method based on color reconstruction according to claim 4, characterized in that: The feature reconstruction layer includes three sets of independent reconstruction convolution blocks and a set of separate convolution layers. The main task of the feature reconstruction layer is to restore and reconstruct the output feature map of the feature fusion layer and the output feature map of the color reconstruction module, and reconstruct the final fused image step by step. The three sets of reconstruction convolution blocks are composed of convolution layers, normalization and ReLU activation functions.

6. The SAR and visible light fusion method based on color reconstruction according to claim 5, characterized in that: The fusion loss function of the image fusion network is: L total =αL SSIM +βL color ; Among them, α and β are weight parameters; L SSIM =(1-SSIM(I fusion ,I SAR ))+(1-SSIM(I fusion ,I VIS )), where I fusion is the fused image, I SAR is the SAR image, I VIS It is a VIS image; C fusion is the fused image, C visible For visible light images.

Images