Sub-pixel positioning method for land cover categories based on multi-source remote sensing data fusion

Through multi-source remote sensing data fusion technology, combined with full-color sharpening and LiDAR elevation information, the accuracy restriction problem caused by the single auxiliary data type in the existing subpixel positioning method is solved, and the spatial distribution information of land cover categories is achieved with higher accuracy Get it.

CN112883823BActive Publication Date: 2024-11-01NANJING UNIV OF AERONAUTICS & ASTRONAUTICS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110116628.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-01-21
Publication Date
2024-11-01
Estimated Expiration
2041-01-21

AI Technical Summary

Technical Problem

When processing mixed pixels, the existing subpixel positioning method cannot fully utilize multiple auxiliary information due to the single type of auxiliary remote sensing data, resulting in limited results and cannot effectively improve the spatial distribution information accuracy of hyperspectral images.

Method used

Using a method based on multi-source remote sensing data fusion, the hyperspectral image and full-color image are fused through full-color sharpening technology, combined with LiDAR elevation information, and feature fusion is performed to generate multi-source remote sensing fusion data containing spatial, spectral and elevation information. And demixed by normalization processing and linear spectral mixing model to obtain more accurate subpixel proportional fraction images.

Benefits of technology

By making full use of multiple auxiliary information in multi-source remote sensing data, the accuracy of subpixel positioning results is significantly improved, which is better than traditional methods, and provides more accurate spatial distribution information for land cover categories.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112883823B_ABST
    Figure CN112883823B_ABST
Patent Text Reader

Abstract

The present invention proposes a sub-pixel positioning (Sup-pixel Mapping, SPM) method based on MRSFD by utilizing the spatial, spectral and elevation information of multi-source remote sensing fusion data (MRSFD). The method comprises the following steps: (1) fusing the original hyperspectral image and the panchromatic image, and obtaining a fused image with high spatial resolution and spectral resolution by using the panchromatic sharpening technology. (2) fusing the obtained fused image containing spatial and spectral information with a laser radar (Light Detection And Ranging, LiDAR) digital surface model (Digital Surface Model, DSM) with elevation information, and obtaining the MRSFD containing spatial, spectral and elevation information by feature fusion. (3) normalizing the MRSFD to obtain a sub-pixel proportional fractional image belonging to the land cover category, and then assigning the class label to the sub-pixel by the class unit-based class assignment method, and obtaining the SPM result according to the proportional information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing image processing, and is a technology for performing sub-pixel positioning on an image obtained by fusing multi-source remote sensing data. Background Art

[0002] The spatial distribution information of land cover categories is one of the important basic data for urban environmental and ecological research. Hyperspectral images are widely used in the observation of land cover categories due to their rich spectral information. However, with the continuous improvement of the spectral resolution of hyperspectral images, the spatial resolution of hyperspectral images will be limited by the hardware imaging equipment, and a large number of mixed pixels will appear in low-resolution hyperspectral images. These mixed pixels hinder the acquisition of accurate spatial distribution information of land cover categories. Although spectral separation technology can process mixed pixels and obtain fractional-order images with mixed pixel ratios belonging to land cover categories, the specific spatial distribution information of land cover categories is still uncertain.

[0003] In order to process mixed pixels and obtain the spatial distribution of land cover categories, sub-pixel mapping (SPM) is proposed as a subsequent spectral decomposition process. SPM first improves the fractional image and divides each mixed pixel into sub-pixels according to the proportional scale. The improved fractional image contains the proportion of sub-pixels belonging to the land cover category. According to this proportional information, the class label is assigned to the sub-pixel to obtain the sub-pixel proportional fractional image belonging to the land cover category. In recent decades, people have developed a variety of SPM methods, such as back propagation neural network, Hopfield neural network (HNN), interpolation algorithm, indicator co-kriging method, spatial gravity model, etc. In recent years, with the continuous development of remote sensing technology, in order to improve the SPM results of hyperspectral images, another high spatial resolution remote sensing data of the same area is selected as auxiliary data. For example, Nguyen et al. introduced the elevation information obtained by LiDAR into the HNN model to improve the final mapping results; Wang et al. used panchromatic images with rich spatial information and adopted panchromatic sharpening technology to fuse the original hyperspectral images to obtain better SPM results.

[0004] However, the types of auxiliary remote sensing data used in SPM are relatively single and cannot provide multiple types of auxiliary information at the same time, which hinders the further improvement of the accuracy of SPM results. In order to make full use of the various auxiliary information in multi-source remote sensing data, a sub-pixel positioning method based on multi-source remote sensing data fusion (SPM-MRSFD) is proposed. Firstly, the original hyperspectral image containing spectral information is fused with the auxiliary panchromatic image containing spatial information using the pan-sharpening technology to obtain a fused image containing spatial and spectral information. Secondly, the fused image containing spatial and spectral information and the LiDAR digital surface model (Digital Surface Model, DSM) with elevation information are feature fused to obtain Multi-Source Remote Sensing Fusion Data (MRSFD) containing spatial, spectral and elevation information. Finally, the MRSFD is normalized to obtain a sub-pixel scale score image belonging to the land cover category, and then the class label is assigned to the sub-pixel through the class unit-based class assignment method, and the SPM result is obtained according to these scale information. By making full use of the various auxiliary information of MRSFD (i.e., spatial, spectral, and elevation information), the accuracy of the SPM results is ultimately improved. Experimental results show that the SPM-MRSFD method is superior to the existing SPM methods. Ultimately, the mapping results of SPM are improved. Summary of the invention

[0005] In order to process mixed pixels and obtain the spatial distribution of land cover categories, SPM is proposed as a subsequent spectral decomposition process. However, the existing SPM method selects another high spatial resolution remote sensing data of the same area as auxiliary data. Although this improves the SPM results of hyperspectral images, the type of auxiliary remote sensing data used in SPM is relatively single and cannot provide multiple types of auxiliary information at the same time, which hinders the further improvement of the accuracy of SPM results. In order to solve the above problems and make full use of the various auxiliary information in MRSFD, a land cover category sub-pixel positioning method based on multi-source remote sensing data fusion is proposed. It includes the following steps:

[0006] (1) The original hyperspectral image and panchromatic image are fused, and the panchromatic sharpening technique is used to obtain a fused image with high spatial resolution and spectral resolution.

[0007] (2) The obtained fused image containing spatial and spectral information is fused with a DSM of LiDAR with elevation information, and the MRSFD containing spatial, spectral and elevation information is obtained through feature fusion.

[0008] (3) The MRSFD is normalized to obtain a sub-pixel scale score image belonging to the land cover category. The class labels are then assigned to the sub-pixels through a class unit-based class assignment method, and the SPM results are obtained based on this scale information.

[0009] Let S be the ratio between the original hyperspectral image and the auxiliary panchromatic image or LiDAR DSM, that is, a mixed pixel in the hyperspectral image is considered to be divided into S on the auxiliary panchromatic image or LiDAR DSM. 2 The original hyperspectral image Y consists of M mixed pixels and B band. H Including mixed pixels P J (J=1,2,...,M). Auxiliary full-color image Y P and the DSM assisted by LiDAR with elevation information contain MS 2 Sub-pixel.

[0010] In step (1), the specific steps of obtaining the fused image using the panchromatic sharpening technique are as follows: the hyperspectral image is converted into principal components (PCs), which contain the spatial information of the hyperspectral image. According to formula (1), the first principal component is replaced with the rich spatial information of the panchromatic image, and the panchromatic image and the first principal component are histogram matched before the replacement to reduce the distortion match. Finally, the panchromatic sharpening is completed by inverse transformation to obtain a fused image with rich spatial and spectral information.

[0011]

[0012] where Y F is the fused image after pan-sharpening, Y b F (b=1,2,...,B) represents the bth band of the fused image, represents the bth band of the hyperspectral image, Y H By panchromatic image Y P The ratio of g is interpolated. b is the vector of injection gain, w b is the weight vector for calculating the overlapping spectra between the hyperspectral image and the panchromatic image.

[0013] In step (2), the specific steps of obtaining MRSFD by feature fusion are as follows: first, spatial and spectral features are extracted using morphologically open or closed structural elements with predetermined sizes and shapes. (n is the number of features), where the B band comes from the fused image. The elevation feature is Among them B *The bands come from the LiDAR DSM. Because the fused image has more bands, the number of bands for spatial and spectral features is greater than that for elevation features (e.g., B > B * ). Then, in order to facilitate feature fusion, the number of frequency bands of spatial and spectral features needs to be normalized to B through kernel principal component analysis (KPCA). * Assumptions and The vector representing the superposition of spatial spectral features and elevation features. Finally, FFG is used to obtain the fusion feature The goal of FFG is to find a solution that can achieve The transformation matrix R. The matrix R is defined as formula (2).

[0014]

[0015] where z xy ∈{0,1} comes from the matrix Z defined as the edge of the image. Using the K-Nearest Neighbors (KNN) algorithm based on Euclidean distance, we find the features close to the center. Features if and Close to z xy =1, and and Distance z xy = 0. In addition, we use the constraint in equation (3) to avoid degeneracy.

[0016] R T (F S-E )D(F S-E ) T R=I (3)

[0017] Where D is a diagonal matrix, and the diagonal D x,x The value of I is the identity matrix. We get R = (r1, r2, ..., r o ), which is composed of the eigenvalues ​​λ1≤λ2≤···≤λ of the minimum eigenvector o of the generalized problem in equation (4) o The relevant feature vectors.

[0018] (F S-E )W(F S-E ) T r=λ(F S-E )D(F S-E ) T r (4)W=DZ is the Laplacian operator matrix.

[0019] In step (3), the feature fusion is used to obtain the MS 2 Multi-source remote sensing fusion data Y of mixed pixels M Then, the mixed spectral decomposition technique based on the linear spectral mixture model (LSMM) is used to unmix the MRSFD to obtain the sub-pixel p containing the land cover category. j (j=1,2,...,MS 2 ) is the proportional fraction image L(p j ). Proportion L(p j ) is defined as

[0020] D b =EL + n (5)

[0021] in is the spectrum vector of MRSFD; L = [L(p1), L(p2), ..., L(p N )] T is the scale vector; E is the matrix of endmember spectra, and n is random noise. When the random noise is minimized, the LSMM algorithm is used for optimal estimation.

[0022] Next, according to these ratios L(p j ), the class labels are assigned to sub-pixels using a class unit-based class assignment method to obtain the final SPM result.

[0023] The beneficial effects of the present invention are as follows: the method of the present invention improves the existing SPM method, fully utilizes the spatial, spectral and elevation information of MRSFD, and utilizes the new method SPM-MRSFD to improve the accuracy of the mapping results, which is superior to the existing latest methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Flowchart of SPM-MRSFD

[0025] Figure 2(a) is a hyperspectral image taken by the Airborne Laser Mapping Center on June 23, 2012

[0026] Figure 2(b) shows the DSM of the LiDAR taken by the Airborne Laser Mapping Center on June 22, 2012.

[0027] Figure 3(a) is the reference image of the SPM mapping result.

[0028] Figure 3(b) shows the SPM mapping result based on intra-and inter-pixel dependency (IID).

[0029] Figure 3(c) shows the SPM mapping result of the radial basis function interpolation by reducing point spread function effect (RBF-PSF).

[0030] Figure 3(d) shows the SPM mapping result based on HNN with LiDAR image (HNN-LiDAR)

[0031] Figure 3(e) shows the SPM (SPM-PAN) mapping result based on full sharpening technology.

[0032] Figure 3(f) shows the mapping result of SPM-MRSFD

[0033] Figure 4(a) is the sub-region reference image of the SPM result.

[0034] Figure 4(b) shows the sub-region mapping result based on IID

[0035] Figure 4(c) shows the sub-region mapping result based on RBF-PSF

[0036] Figure 4(d) shows the sub-region mapping result based on HNN-LiDAR

[0037] Figure 4(e) shows the sub-area mapping result of SPM-PAN

[0038] Figure 4(f) shows the sub-region mapping result of SPM-MRSFD DETAILED DESCRIPTION

[0039] In order to process mixed pixels and obtain the spatial distribution of land cover categories, a sub-pixel localization method for land cover categories based on multi-source remote sensing data fusion is proposed. We tested and compared five SPM methods, including IID-based SPM, RBF-PSF-based SPM, HNN-LiDAR-based SPM, SPM-PAN, and the proposed SPM-MRSFD. The steps are as follows:

[0040] (1) Downsample the fine hyperspectral image through a mean filter to obtain a simulated low-resolution hyperspectral image. Fuse the original hyperspectral image with the panchromatic image, and use the panchromatic sharpening technique to obtain a fused image with high spatial resolution and spectral resolution. Specifically:

[0041] The hyperspectral image is converted into principal components (PCs), which contain the spatial information of the hyperspectral image. According to formula (1), the first principal component is replaced with the rich spatial information of the panchromatic image, and the histogram matching of the panchromatic image and the first principal component is performed before the replacement to reduce the distortion matching. Finally, the inverse transform is used to complete the panchromatic sharpening to obtain a fused image with rich spatial and spectral information.

[0042]

[0043] where Y F is the fused image after pan-sharpening. represents the bth band of the fused image, represents the bth band of the hyperspectral image, Y H By panchromatic image Y P The ratio of g is interpolated. b is the vector of injection gain, w b is the weight vector for calculating the overlapping spectra between the hyperspectral image and the panchromatic image.

[0044] It should be noted that the spatial distribution of land cover classes cannot be determined due to a large number of mixed pixels. In addition, we only consider the full sharpening technique and avoid the influence of errors caused by the acquisition of panchromatic images. The spectral response function of the IKONOS satellite is applied to the refined hyperspectral image to create a proper panchromatic image. By fusing the sharpened fused image, a low-resolution hyperspectral image and a panchromatic image are simulated. Because the spatial information of the panchromatic image is provided to the low-resolution hyperspectral image, the fused image is clearer.

[0045] (2) The obtained fused image containing spatial and spectral information is fused with a DSM of LiDAR with elevation information, and the MRSFD containing spatial, spectral and elevation information is obtained through feature fusion. Specifically:

[0046] First, spatial and spectral features are extracted using morphologically open or closed structural elements of predetermined size and shape. (n is the number of features), where the B band comes from the fused image. The elevation feature is Among them B * The bands come from the LiDAR DSM. Because the fused image has more bands, the number of bands for spatial and spectral features is greater than that for elevation features (e.g., B > B * ). Then, in order to facilitate feature fusion, the number of frequency bands of spatial and spectral features needs to be normalized to B through Kernel Principal Component Analysis (KPCA)* Assumptions and The vector representing the superposition of spatial spectral features and elevation features. Finally, FFG is used to obtain the fusion feature The goal of FFG is to find a solution that can achieve The transformation matrix R. The matrix R is defined as formula (2).

[0047]

[0048] where z xy ∈{0,1} comes from the matrix Z defined as the edge of the image. Using the K-Nearest Neighbors (KNN) algorithm based on Euclidean distance, we find the features close to the center. Features if and Close to z xy =1, and and Distance z xy = 0. In addition, we use the constraint in equation (3) to avoid degeneracy.

[0049] R T (F S-E )D(F S-E ) T R=I (3)

[0050] Where D is a diagonal matrix, and the diagonal D x,x The value of I is the identity matrix. We get R = (r1, r2, ..., r o ), which is composed of the eigenvalues ​​λ1≤λ2≤···≤λ of the minimum eigenvector o of the generalized problem in equation (4) o The relevant feature vectors.

[0051] (F S-E )W(F S-E ) T r=λ(F S-E )D(F S-E ) T r (4)W=DZ is the Laplacian operator matrix.

[0052] (3) Normalize the MRSFD to obtain a sub-pixel ratio score image belonging to the land cover category, and then assign the class label to the sub-pixel through the class unit-based class assignment method, and obtain the SPM result based on this ratio information. Specifically:

[0053] Through feature fusion, we can obtain2 Multi-source remote sensing fusion data Y of mixed pixels M Then, the mixed spectral decomposition technique based on the linear spectral mixture model (LSMM) is used to unmix the MRSFD to obtain the sub-pixel p containing the land cover category. j (j=1,2,...,MS 2 ) is the proportional fraction image L(p j ). Proportion L(p j ) is defined as

[0054] D b =EL + n (5)

[0055] in is the spectrum vector of MRSFD; L = [L(p1), L(p2), ..., L(p N )] T is the scale vector; E is the matrix of endmember spectra, and n is random noise. When the random noise is minimized, the LSMM algorithm is used for optimal estimation.

[0056] Next, according to these ratios L(p j ), the class labels are assigned to sub-pixels using a class unit-based class assignment method to obtain the final SPM result.

[0057] The principle flow chart of the land cover category sub-pixel positioning method based on multi-source remote sensing data fusion proposed in the present invention is as follows: Figure 1 shown.

[0058] Figure 2(a) and Figure 2(b) are the datasets used by this method, where Figure 2(a) is a pseudo-color image of a hyperspectral image, which is obtained by downsampling a fine hyperspectral image using an S×S mean filter. Figure 2(b) is the DSM of LiDAR, which is fused with the sharpened fusion image to obtain MRSFD. By utilizing MRSFD, the proposed SPM-MRSFD can produce better mapping results. The pixels of this dataset are 320×1800 and the spatial resolution is 2.5m.

[0059] Figure 3(a) also comes from the dataset of Figure 2(a), which contains 15 types of scenes and serves as the benchmark reference image for the SPM results. The mapping results of the four SPM methods are shown in Figures 3(b) to (f).

[0060] Figure 4(a) is an enlarged sub-region of the pixel marked in Figure 3(a). By visual comparison, the mapping result of SPM-MRSFD is closer to the reference image. Comparing the other four methods, we can find that the SPM results of IID, RBF-PSF, HNN-LiDAR and SPM-PAN are not ideal. For example, observing Table 1, the 15th category (running track) has many prominent burr boundaries and discontinuous shape areas in Figures 4(b) to (e). The above phenomenon is alleviated in the proposed SPM-MRSFD in Figure 4(f).

[0061] The mapping accuracy of each class (%), OA (%), and Kappa of the five SPM methods are listed in Table I. Although the mapping accuracy of class 4 (synthetic turf), class 13 (parking lot 2), and class 14 (tennis court) in HNN-LiDAR is higher than that in SPM-MRSFD, the mapping accuracy of other classes, especially small target classes such as class 4 (tree), class 7 (residential), and class 9 (road), is very low, resulting in the lowest OA (%) and Kappa among the five SPM methods. Compared with SPM-PAN, the OA (%) and Kappa of SPM-MRSFD increased by about 2.6% and 0.027. According to the definition of OA (%), because there are pixels in the experimental data, the increase of about 2.6% means that SPM-MRSFD can produce 14976 more correctly mapped pixels. From the evaluation data, it can be seen that the performance of SPM-MRSFD is better than that of the other four SPM methods.

[0062] Table 1. Performance evaluation of different methods (S=8)

[0063]

Claims

1. A land cover category sub-pixel positioning method based on multi-source remote sensing data fusion, characterized in that: The steps include: (1) Fusing the original hyperspectral image and panchromatic image, and using panchromatic sharpening technology to obtain a fused image with high spatial resolution and spectral resolution; (2) fusing the obtained fused image containing spatial and spectral information with a lidar digital surface model with elevation information, and obtaining multi-source remote sensing fusion data containing spatial, spectral and elevation information through feature fusion; (3) Normalize the multi-source remote sensing fusion data to obtain the sub-pixel scale fraction image belonging to the land cover category, and then assign the class label to the sub-pixel through the class unit-based class assignment method, and obtain the sub-pixel positioning result based on this scale information; In step (2), spatial and spectral features are first extracted using morphologically open or closed structural elements of predetermined size and shape. n is the number of features, where the B band comes from the fused image; and the elevation feature is Among them B * The bands come from the digital surface model of the lidar with elevation information; because the fused image has more bands, the number of bands of spatial and spectral features is greater than the number of bands of elevation features, B>B * ; Then, in order to facilitate feature fusion, the number of frequency bands of spatial and spectral features needs to be normalized to B through kernel principal component analysis * Assumptions A vector representing the superposition of spatial spectral features and elevation features, Finally, the feature fusion method based on the graph is used to obtain Multi-source remote sensing fusion data; The goal of the graph-based feature fusion method is to find a The transformation matrix R of , the matrix R is defined as formula (2): where z xy ∈{0,1} comes from the matrix Z defined as the edge of the image; the K nearest neighbor algorithm based on Euclidean distance is used to find the features close to the center Features if and Close to z xy =1, and and Distance z xy = 0; In addition, the constraint in equation (3) is used to avoid degeneration: R T (F S-E )D(F S-E ) T R=I(3) Where D is a diagonal matrix, and the diagonal D x,x The value of I is the identity matrix; we get R = (r1, r2, ..., r o ), which is composed of the eigenvalues ​​λ1≤λ2≤···≤λ of the minimum eigenvector o of the generalized problem in equation (4) o The relevant eigenvectors consist of: Wherein, W=DZ is the Laplacian operator matrix.

2. The land cover category sub-pixel positioning method based on multi-source remote sensing data fusion as claimed in claim 1, characterized in that: In step (1), the hyperspectral image is converted into principal components, which contain the spatial information of the hyperspectral image; according to formula (1), the first principal component is replaced with the rich spatial information of the panchromatic image, and the panchromatic image and the first principal component are histogram matched before the replacement to reduce the distortion match; finally, the inverse transform is used to complete the panchromatic sharpening to obtain a fused image with rich spatial and spectral information; Among them, Y b F represents the bth band of the fused image, b=1,2,...,B, B represents the total number of bands, represents the bth band of the hyperspectral image, Y H By panchromatic image Y P The ratio of g is interpolated. b is the vector of injection gain, w b is the weight vector for calculating the overlapping spectra between the hyperspectral image and the panchromatic image.

3. The land cover category sub-pixel positioning method based on multi-source remote sensing data fusion as claimed in claim 1, characterized in that: In step (3), the feature fusion is used to obtain the MS 2 Multi-source remote sensing fusion data Y of mixed pixels M Finally, the mixed spectral decomposition technology based on the linear spectral mixture model is used to unmix the multi-source remote sensing fusion data to obtain the sub-pixel p containing the land cover category. j The proportional fraction image L(p j ), j = 1, 2, ..., MS 2 , the ratio L(p j ) is defined as D b =EL+n(5) Where D b =[D1,D2,...,D B* ] T is the spectrum vector of multi-source remote sensing fusion data; L = [L(p1), L(p2), ..., L(p N )] T is a scale vector; E is the matrix of endmember spectra, and n is random noise; when the random noise is minimal, the linear spectral mixture model algorithm is used for optimal estimation; Next, according to these ratios L(p j ), the class labels are assigned to sub-pixels using a class unit-based class assignment method to obtain the final sub-pixel positioning result.