Spatial quality evaluation method for multispectral and panchromatic image fusion based on regional convolutional network

Abstract

The application relates to a multispectral and panchromatic image fusion spatial quality evaluation method based on a region convolution network, which improves the processing process of a fused image and a panchromatic image in a traditional region-based multispectral image and panchromatic image fusion spatial quality evaluation method. The spatial features of region images in the original fused image and the panchromatic image are directly extracted by using a GoogLeNet network, instead of performing gray image conversion and discrete cosine transformation on the fused image. The spatial information loss of the image is effectively avoided, and meanwhile, the deep features with stronger representation ability and generalization ability are extracted, so that the spatial quality of a fusion result is evaluated at a feature level. Finally, an intuitive spatial quality distribution map is obtained to represent the spatial quality of local regions of the fusion result.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of information processing, and relates to a quality evaluation method for a remote sensing image fusion result, in particular to a multi-spectral image and panchromatic image fusion spatial quality evaluation method based on a regional convolution network. BACKGROUND

[0002] Remote sensing image information obtained through image fusion technology has a large amount of information and has high resolution in the spatial domain and the spectral domain, and is more and more widely applied to the field of remote sensing. Remote sensing image fusion quality evaluation plays an important role in these applications, and is not only a benchmark for measuring remote sensing image fusion methods, but also an important tool for improving remote sensing image fusion methods.

[0003] A traditional method is to perform spatial quality evaluation on the fusion result of the multi-spectral and panchromatic images at the pixel level, and finally only one quality evaluation score is used to represent the spatial quality of the fusion result, and the local area spatial quality of the fusion result is not concerned. Therefore, a spatial quality evaluation method for the fusion result of the multi-spectral image and the panchromatic image based on a region is proposed, the spatial quality of the fusion result can be evaluated in units of regions through region extraction of the fusion image, and finally the spatial quality evaluation scores of each region of the fusion result are obtained. The traditional spatial quality evaluation method based on a region converts the fusion image into a gray image before performing spatial quality evaluation on the original panchromatic image and the fusion image, then performs discrete cosine transformation on the gray images of the panchromatic image and the fusion image, and finally performs correlation calculation on the results obtained through the discrete cosine transformation to obtain the spatial quality evaluation result. The parameters required for the gray image conversion of the fusion image from different sensors are different, so that the generalization ability of the method itself is poor, and the process of gray image conversion causes information loss of the fusion image, which brings more uncertainty to the spatial quality evaluation result. SUMMARY

[0004] Technical problems to be solved

[0005] The purpose of the present application is to overcome the problems of poor generalization ability and information loss in the evaluation process of the traditional spatial quality evaluation method for the fusion result of the multi-spectral image and the panchromatic image based on a region, and to perform spatial quality evaluation of the fusion result in units of regions at the feature level. The present application replaces the gray conversion process of the fusion image and the discrete cosine transformation process of the panchromatic image and the gray image in the traditional method, and further proposes a multi-spectral image and panchromatic image fusion spatial quality evaluation method based on a regional convolution network.

[0006] Technical scheme

[0007] A multi-spectral and panchromatic image fusion spatial quality evaluation method based on a regional convolution network, characterized by the following steps:

[0008] Step 1: The fusion image has 4 bands or 8 bands, for the 4-band fusion image, the 3rd, 2nd and 1st bands are R, G and B bands in turn, and for the 8-band fusion image, the 5th, 3rd and 2nd bands are R, G and B bands in turn; the R, G and B bands of the fusion image are extracted as input, SLIC superpixel segmentation is performed, and segmentation information of the segmentation image is obtained, the segmentation information includes the bounding rectangle coordinates (x i ,y i ), size h i ×w i and region image label stat i , the region image label is 1, otherwise it is 0;

[0009] Step 2: According to the bounding rectangle coordinates (x i ,y i ) and size h i ×w i in the segmentation information obtained in step 1, the fusion image is rectangularly cropped to obtain an image block FUScrop i containing the i-th region image object i , the same operation is performed on the panchromatic image to obtain a PAN image block PANcrop i with the same size as FUScrop i ;

[0010] Step 3: The FUScrop i obtained in step 2 and the region image label stat i contained in the segmentation information in step 1 are multiplied to obtain a rectangle FUSobj i containing the region image and replacing the pixels other than object i with 0, and the same operation is performed on PANcrop i to obtain PANobj i ;

[0011] Step 4: A set of rectangular images with the same size and containing the same shape of region images, i.e. FUSobj i and PANobj i , are obtained by step 3, the set of rectangular images is input into a pre-trained GoogLeNet network model for feature extraction, the input data of the last fully connected layer of the network model is extracted as a feature vector to obtain two feature vectors ff i and pf i , ff i represents the feature vector of FUSobj i , and pfi representing the feature vector of PANobj i ;

[0012] Step 5: Calculate the Pearson linear correlation coefficient PCC between ff i and pf i obtained in step 4 as the spatial quality score Q i of the current region, which reflects the local spatial quality of FUS;

[0013] Step 6: If the spatial quality scores of all regions are obtained, the global quality map Q global in the region unit is obtained, and the process ends; otherwise, return to step 2 to extract the next pair of image blocks FUScrop i+1 and PANcrop i+1 from the fused image and the panchromatic image.

[0014] Advantages

[0015] The present application proposes a multi-spectral and panchromatic image fusion spatial quality evaluation method based on regional convolution network, which improves the processing process of the fused image and the panchromatic image in the traditional multi-spectral image and panchromatic image fusion spatial quality evaluation method based on regions. The spatial features of the region image in the original fused image and the panchromatic image are directly extracted using the GoogLeNet network, instead of first converting the fused image into a gray image and performing discrete cosine transform. This effectively avoids the loss of spatial information of the image while extracting deep features with stronger representation ability and generalization ability, and performs spatial quality evaluation at the feature level on the fusion result. Finally, an intuitive spatial quality distribution map is obtained to represent the spatial quality of the local region of the fusion result. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 : Flowchart of the multi-spectral and panchromatic image fusion spatial quality evaluation method based on regional convolution network DETAILED DESCRIPTION

[0017] The present application will be further described in conjunction with the embodiments and drawings:

[0018] A multi-spectral image and panchromatic image fusion spatial quality evaluation method based on regional convolution network, the steps are as follows:

[0019] Step 1: Extract the R, G, B bands of the fused image as input, and perform SLIC superpixel segmentation to obtain the segmentation information of the segmentation map, i.e. the information of each region. Including the circumscribed rectangle coordinates, size and region image label of each region.

[0020] Step 2 uses the coordinates and dimensions of the bounding rectangle of the region image obtained in Step 1 to perform rectangular cropping on the fused image. The same operation is repeated on the panchromatic image.

[0021] Step 3 performs a dot product operation on the image patches of the fused image obtained in Step 2 and the region image labels contained in the segmentation information in Step 1. Pixels outside the region are replaced with 0, resulting in a bounding rectangle containing only the region pixels, which is the region image of the fused image. The same operation is repeated on the panchromatic image to obtain paired regions of the fused image and the panchromatic image.

[0022] Step 4 takes the pair of rectangular region images of the fused image and the panchromatic image obtained in Step 3 as input, and uses the pre-trained GoogLeNet model to extract spatial features respectively, and obtains the feature vectors of the two.

[0023] Step 5: Calculate the Pearson linear correlation coefficient between the two eigenvectors from Step 4, and use it as the spatial quality score for this region.

[0024] Step 6: If the spatial quality scores for all regions are obtained, the spatial quality distribution map of the final fused image is obtained, and the process ends. Otherwise, return to step 3 to process the next pair of image patches.

[0025] The specific steps are as follows:

[0026] Step 1: The fused image has 4 or 8 bands. For a 4-band fused image, bands 3, 2, and 1 are R, G, and B bands respectively. For an 8-band fused image, bands 5, 3, and 2 are R, G, and B bands respectively. Extract the R, G, and B bands from the fused image as input for SLIC superpixel segmentation. For a 1600×1600 fused image, n is typically chosen. object =100~120 as the number of image regions after segmentation, object i (1≤i≤n) represents the image of the i-th region. The segmentation information output in step 1 includes the bounding rectangle coordinates (x, y, y) of each region. i ,y i ), size h i ×w i and regional image markers stat i In the region image labeling, those belonging to the region are labeled as 1, and those not belonging to the region are labeled as 0.

[0027] Step 2: Based on the circumscribed rectangle coordinates (x, y) obtained from the segmentation information in Step 1 i ,y i ) and size h i ×w i Perform rectangular cropping on the fused image to obtain the image containing the object. iFUScrop i , the same operation is performed on the panchromatic image to get a rectangular image block PANcrop i with the same size as FUScrop i .

[0028] Step 3, the point multiplication operation is performed on FUScrop i and stat i included in the segmentation information in step 1 to get a rectangular image block FUSobj i except that the pixels outside object i are replaced by 0 and only contains the bounding rectangle of the region image. i Step 4, the same operation is performed on PANcrop i .

[0029] Step 4, a set of rectangular image blocks with the size of h i × w i and containing the same shape region are obtained from the fused image and the panchromatic image in step 3, namely FUSobj i and PANobj i . The set of rectangular images is input into the pre-trained GoogLeNet network model for feature extraction, and the input data of the last fully connected layer of the network model is extracted as the feature vector, that is, two feature vectors with a length of 1024 are obtained, ff i represents the feature vector of FUSobj i , and pf i represents the feature vector of PANobj i .

[0030] Step 5, the Pearson Correlation Coefficient (PCC) between ff i and pf i obtained in step 4 is calculated as the spatial quality score Q i of the current region, which reflects the spatial quality of the local region of the fused image.

[0031] Step 6, if the spatial quality scores of all regions are obtained, the global spatial quality distribution map Q global in the unit of region is obtained, and the process ends. Otherwise, return to step 2 to extract the next pair of image blocks FUScrop i+1 and PANcrop i+1 from the fused image and the panchromatic image.

Claims

1. A method for evaluating the spatial quality of multispectral and panchromatic image fusion based on a region convolutional network, characterized in that The steps are as follows: Step 1: the fused image has 4 bands or 8 bands, for the fused image with 4 bands, the 3rd, 2nd and 1st bands are R, G and B bands in turn, for the fused image with 8 bands, the 5th, 3rd and 2nd bands are R, G and B bands in turn; taking the R, G and B bands of the fused image as input, SLIC superpixel segmentation is carried out to obtain segmentation information of a segmentation map, the segmentation information includes the circumscribed rectangle coordinates, size and region image label of each region The region image label is 1, otherwise 0.​​ Step 2: Based on the circumscribed rectangle coordinates obtained in Step 1 and size Perform rectangular cropping on the fused image to obtain the image containing the first... Image of each region Image patches of fused images Perform the same operation on the panchromatic image to obtain the same result. PAN image blocks of uniform size ; Step 3: multiply the region image contained in the split information in Step 2 by the region image contained in the split information in Step 1 and the region image contained in the split information in Step 1 Step 4: multiply the region image contained in the split information in Step 3 by the region image contained in the split information in Step 2 Step 5: multiply the region image contained in the split information in Step 4 by the region image contained in the split information in Step 1 Step 6: multiply the region image contained in the split information in Step 5 by the region image contained in the split information in Step 2 Step 7: multiply the region image contained in the split information in Step 6 by the region image contained in the split information in Step 1 Step 8: multiply Step 4: a set of fused images and panchromatic images of the same size and containing the same image shape of the region are obtained by step 3, that is, a rectangular image of the same size and containing the same image shape of the region is obtained by step 3 and , the set of rectangular images is input into a pre-trained GoogLeNet network model for feature extraction, the input data of the last fully connected layer of the network model are extracted as feature vectors, and two feature vectors and , represent the feature vectors of , represent the feature vectors of ; Step 5: Calculate the Pearson linear correlation coefficient PCC between the results of step 4 and as the spatial quality score of the current region which reflects the local spatial quality of the FUS. Step 6: If the spatial quality score of all regions is obtained, the global quality map in the unit of region is obtained , end; otherwise, return to step 2 to extract the next pair of image blocks from the fused image and the panchromatic image and ; The spatial features of the region images in the original fused image and the panchromatic image are directly extracted by using the GoogLeNet network instead of performing gray image conversion and discrete cosine transformation on the fused image; the spatial information loss of the image is effectively avoided, and the deep features with stronger representation ability and generalization ability are extracted, and the spatial quality evaluation of the fusion result is performed at the feature level.

Images