Small sample imbalanced hyperspectral image classification method based on graph convolutional network and convolutional network fusion

Abstract

The present invention discloses a small sample unbalanced hyperspectral image classification method based on the fusion of graph convolutional network and convolutional network. The method proposes a multi-network hybrid architecture. In this framework, the graph network uses the long-range dependency in the hyperspectral image to learn the spectral-spatial feature representation in the image. In order to avoid the transition smoothing problem of the traditional deep graph network, the present invention provides an enhanced graph network model based on the image signal processing theoretical method. In order to make full use of the spectral domain information, spatial domain information and the complementary information of the two in the image, three graph network branches are designed to extract useful information. At the same time, the present invention also uses the SAM segmentation model based on the large model to segment the hyperspectral image to obtain superpixels, thereby greatly reducing the problem of large amount of calculation. On the other hand, the present invention adopts a convolutional network mixed with 1D convolution and 2D convolution to extract local dependency information in the image, thereby reducing the problem of computational complexity and parameter overfitting.

Description

Technical Field

[0001] The present invention belongs to the field of remote sensing technology and relates to a hyperspectral image classification method, and specifically relates to a small sample imbalanced hyperspectral image classification method based on the fusion of a graph convolutional network and a convolutional network. Background Art

[0002] Hyperspectral images contain a large number of bands with rich spectral information. With reliable hyperspectral classification methods, they can be widely used in various scenarios, such as precision agriculture, military, mineral exploration, etc. In real applications, it is a time-consuming and laborious task to label a large number of samples from hyperspectral images, and there is often an extremely imbalanced class problem in these labeled samples. This makes it difficult for traditional hyperspectral classification methods to avoid overfitting and difficult to train effective classification models, resulting in low classification accuracy and limiting the effective use of hyperspectral images. Summary of the invention

[0003] To solve the above problems, the present invention provides a small sample imbalanced hyperspectral image classification method based on the fusion of graph convolutional network and convolutional network, which is particularly suitable for classification scenarios with only a few labeled samples and extremely imbalanced categories, and can be effectively applied to hyperspectral classification tasks with small samples and severe category imbalance.

[0004] The objective of the present invention is achieved through the following technical solutions:

[0005] A small sample unbalanced hyperspectral image classification method based on graph convolutional network and convolutional network fusion, comprising the following steps:

[0006] Step 1: Design a CNN denoiser

[0007] A 2-layer 1×1 convolutional layer module is used as the CNN denoiser, with X l Represents the input of the lth layer, and the convolutional layer model of the CNN denoiser is expressed as:

[0008]

[0009] Where: f 1×1 is a 1×1 convolution kernel, BN(·) is a data normalization operation, b l is the offset, ReLU(·) is the nonlinear activation function, and “*” indicates the convolution operation;

[0010] Step 2: Design a spectral-spatial feature learning method for superpixel representation

[0011] Step 2.1: Obtain SAM superpixel representation

[0012] (1) From multiple bands of the hyperspectral image, three bands within the visible light range are selected to obtain a pseudo-color image of the hyperspectral image;

[0013] (2) uniformly select q pixels in the pseudo-color image as revealing information, and input them together with the pseudo-color image into the SAM segmentation model to obtain a series of mask matrices;

[0014] (3) rearranging the obtained series of mask matrices into a superpixel indication matrix H, where the non-zero elements in each column indicate the set of superpixels;

[0015] (4) Using the obtained superpixel indicator matrix H, the matrix-form hyperspectral image X is encoded into a superpixel representation:

[0016] S=D -1 H T X

[0017] Where: D is a diagonal matrix, and its diagonal element value is the sum of each column of the indicator matrix H;

[0018] Step 2.2: Spatial feature extraction from superpixel representation

[0019] The l-th spatial feature representation of the superpixel of a given hyperspectral image Its spatial domain graph convolution output is:

[0020]

[0021] Among them: A s is the spatial adjacency matrix of superpixel features, is the spectral domain feature transformation matrix of the lth layer;

[0022] Step 2.3: Spectral Domain Feature Extraction of Superpixel Representation

[0023] The lth layer spectral domain feature representation of the superpixel of a given hyperspectral image Its spectral domain graph convolution output is:

[0024]

[0025] Among them: A f is the spectral domain adjacency matrix of superpixel features, is the spectral domain feature transformation matrix of the lth layer; Step 2.4: Spatial-spectral domain feature extraction of superpixel representation

[0026] The l-th spatial-spectral feature representation of the superpixel of a given hyperspectral image Its spatial-spectral domain graph convolution output is:

[0027]

[0028] Among them: A c =αA s +(1-α)A f is the spatial-spectral domain adjacency matrix of superpixel features, α is the weighting parameter, is the spatial-spectral domain feature transformation matrix of the lth layer;

[0029] Step 3: Spectral-spatial feature extraction of pixel representation

[0030] 1D convolution kernel and 2D convolution kernel are used to extract pixel-level spatial-spectral feature representation in hyperspectral images. Represents the hyperspectral image of the lth layer, and the output of its spectrum-space feature representation is expressed as:

[0031]

[0032] Where: f 1×1 is a 1D convolution kernel, f 2D Represents a 2D convolution kernel;

[0033] Step 4: Fusion of superpixel representation and pixel-level feature representation

[0034] Let X p is the matrix representation of the pixel-level features output from step 3, let S s , S f and S c They are the superpixel-level output representations obtained by the spectral-spatial feature learning method of superpixel representation in step 2, and are decoded into pixel-level features using the superpixel indicator matrix H: s =HS s , X f =HS f , X c =HS c , then, the fusion of pixel-level and super-pixel-level features is written as:

[0035] Z=MLP(X s ‖X f ‖X c ‖X p )

[0036] Among them: MLP (.) is a standard multi-layer perception network, "||" represents the data concatenation operation in the column direction of the matrix;

[0037] Step 5: Hyperspectral pixel classification

[0038] The superpixel-level representation and the pixel-level fusion feature representation Z are represented and the corresponding category prediction results are generated using the softmax(.) function:

[0039]

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] The method of the present invention uses graph networks (GCNs) and convolutional networks (CNNs) at the same time, and proposes a multi-network hybrid architecture to jointly learn the spectral domain information and spatial information in the hyperspectral image, and fuse them to predict the classification results of the hyperspectral image. In this classification framework, the graph network uses the long-range dependency relationship in the hyperspectral image to learn the spectral-spatial feature representation in the image. In order to avoid the transition smoothing problem of the traditional deep graph network, the present invention provides an enhanced graph network model based on the image signal processing theory method. In order to make full use of the spectral domain information, spatial domain information and the complementary information of the two in the image, three graph network branches are designed to extract useful information. At the same time, the present invention also uses the SAM segmentation model based on the large model to segment the hyperspectral image to obtain superpixels, thereby greatly reducing the problem of large amount of calculation. On the other hand, the present invention uses a convolutional network mixed with 1D convolution and 2D convolution to extract local dependency information in the image, thereby reducing the problem of computational complexity and parameter overfitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a hyperspectral classification method framework based on the fusion of graph convolutional network and convolutional network.

[0043] Figure 2 Pseudo-color and annotated images of the Indian Pines dataset.

[0044] Figure 3 The predicted label results of the method of the present invention and other methods on the "Indian Pines" dataset. DETAILED DESCRIPTION

[0045] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.

[0046] The present invention provides a small sample unbalanced hyperspectral image classification method based on graph convolutional network and convolutional network fusion, such as Figure 1 As shown, the following steps are included:

[0047] Step 1: Design a CNN denoiser

[0048] Since the original hyperspectral image is susceptible to noise pollution, directly classifying this image will reduce the classification performance. To this end, the present invention uses a 2-layer 1×1 convolutional layer module as a CNN denoiser, which can perform denoising on each pixel of the hyperspectral image in the spectral domain using the similarity between adjacent bands. represents the input of the lth layer, where l = 1, 2, ..., the convolutional layer model of the CNN denoiser can be expressed as:

[0049]

[0050] Where: f 1×1 is a 1×1 convolution kernel, BN(·) is a data normalization operation, b l is the offset, ReLU(·) is the nonlinear activation function, and “*” represents the convolution operation.

[0051] Step 2: Spectral-spatial feature learning of superpixel representation

[0052] Step 2.1: Obtain SAM superpixel representation

[0053] First, from the multiple bands of the hyperspectral image, three bands in the visible light range are selected, and the red, green, and blue light wavelengths (in nanometers) range from 760 to 622, 577 to 492, and 450 to 435, respectively, so as to obtain a pseudo-color image of the hyperspectral image. Next, q pixels are uniformly selected in the pseudo-color image as revealing information, and they are input into the SAM segmentation model together with the pseudo-color image to obtain a series of mask matrices. Then, these mask matrices are rearranged into a superpixel indicator matrix H, in which the non-zero elements in each column indicate the set in the superpixel. Using the obtained superpixel indicator matrix H, the matrix-formed hyperspectral image X can be encoded into its superpixel representation as:

[0054] S=D -1 H T X

[0055] Where: D is a diagonal matrix, and its diagonal element value is the sum of each column of the indicator matrix H.

[0056] Step 2.2: Spatial feature extraction from superpixel representation

[0057] The l-th spatial feature representation of the superpixel of a given hyperspectral image Its spatial domain graph convolution output is:

[0058]

[0059] Among them: A s is the spatial adjacency matrix of superpixel features, is the spectral domain feature transformation matrix of the lth layer.

[0060] Step 2.3: Spectral Domain Feature Extraction of Superpixel Representation

[0061] The lth layer spectral domain feature representation of the superpixel of a given hyperspectral image Its spectral domain graph convolution output is:

[0062]

[0063] Among them: A f is the spectral domain adjacency matrix of superpixel features, is the spectral domain feature transformation matrix of the lth layer.

[0064] Step 2.4: Spatial-spectral feature extraction of superpixel representation

[0065] The l-th spatial-spectral feature representation of the superpixel of a given hyperspectral image Its spatial-spectral domain graph convolution output is:

[0066]

[0067] Among them: A c =αA s +(1-α)A f is the spatial-spectral domain adjacency matrix of superpixel features, α is the weighting parameter, is the spatial-spectral domain feature transformation matrix of the lth layer.

[0068] Step 3: Spectral-spatial feature extraction of pixel representation

[0069] Mix 1D convolution kernels and 2D convolution kernels to extract pixel-level spatial-spectral feature representations in hyperspectral images. Representing the hyperspectral image of the lth layer, the output of its spectrum-space feature representation can be expressed as:

[0070]

[0071] Where: f 1×1 is a 1D convolution kernel, f 2D Represents a 2D convolution kernel.

[0072] Step 4: Fusion of superpixel representation and pixel-level feature representation

[0073] Let X p is the matrix representation of the pixel-level features output from step 3, let S s , S f and S c They are the superpixel-level output representations obtained by the spectral-spatial feature learning method of superpixel representation in step 2, and can be decoded into pixel-level features using the superpixel indicator matrix H: s =HSs , X f =HS f , X c =HS c Then, the fusion representation of pixel-level and super-pixel-level features can be written as:

[0074] Z=MLP(X s ‖X f ‖X c ‖X p )

[0075] Among them: MLP(.) is a standard multi-layer perception network, and "||" represents the data concatenation operation of the matrix in the column direction.

[0076] Step 5: Hyperspectral pixel classification

[0077] The superpixel-level representation and the pixel-level fusion feature representation Z are represented and the corresponding category prediction results are generated using the softmax(.) function:

[0078]

[0079] In order to verify the effectiveness of the hyperspectral image classification method of the present invention, the public "Indian Pines" dataset is used to compare the classification results of the method of the present invention with those of several current mainstream classification methods. The hyperspectral image classification methods used for comparison include: SSRN, HybridSN, A2S2K, SSFTT, CEGCN, CNCMN, AMGCFN. The network parameters in the method of the present invention are shown in Table 1. The "Indian Pines" dataset contains 16 unbalanced categories, such as Figure 2 All these methods use only a small number of sample sets for the Indian Pines dataset during the training phase, and the sample ratios of the training set, validation set, and test set used are 1%, 1%, and 98%, as shown in Table 2.

[0080] Hyperspectral image classification is measured by overall accuracy (OA), average accuracy (AA) and Kappa coefficient. The classification performance of the method of the present invention and other methods is shown in Table 3. The prediction results are shown in Figure 3 Compared with the current mainstream hyperspectral classification method, this experimental result verifies that the method of the present invention is an effective pixel classification method under a small label data set.

[0081] Table 1 Network parameters in the hyperspectral image classification method of the present invention

[0082]

[0083] Table 2 The number of labeled samples in the “Indian Pines” dataset and the number of samples used in the training phase

[0084]

[0085] Table 3 Classification results of the method of the present invention and other methods

[0086]

Claims

1. A small sample unbalanced hyperspectral image classification method based on graph convolutional network and convolutional network fusion, characterized in that The method comprises the following steps: Step 1: Design a CNN denoiser A 2-layer 1×1 convolutional layer module is used as the CNN denoiser. Represents the input of the lth layer, and the convolutional layer model of the CNN denoiser is expressed as: Where: f 1×1 is a 1×1 convolution kernel, BN(·) is a data normalization operation, b l is the offset, ReLU(·) is the nonlinear activation function, and "*" indicates the convolution operation; Step 2: Design a spectral-spatial feature learning method for superpixel representation Step 2.1: Obtain SAM superpixel representation; Step 2.2: Spatial feature extraction from superpixel representation The l-th spatial feature representation of the superpixel of a given hyperspectral image Its spatial domain graph convolution output is: Among them: A s is the spatial adjacency matrix of superpixel features, is the spectral domain feature transformation matrix of the lth layer; Step 2.3: Spectral Domain Feature Extraction of Superpixel Representation The lth layer spectral domain feature representation of the superpixel of a given hyperspectral image Its spectral domain graph convolution output is: Among them: A f is the spectral domain adjacency matrix of superpixel features, is the spectral domain feature transformation matrix of the lth layer; Step 2.4: Spatial-spectral feature extraction of superpixel representation The l-th spatial-spectral feature representation of the superpixel of a given hyperspectral image Its spatial-spectral domain graph convolution output is: Among them: A c is the spatial-spectral domain adjacency matrix of superpixel features, is the spatial-spectral domain feature transformation matrix of the lth layer; Step 3: Spectral-spatial feature extraction of pixel representation 1D convolution kernel and 2D convolution kernel are used to extract pixel-level spatial-spectral feature representation in hyperspectral images. Represents the hyperspectral image of the lth layer, and the output of its spectrum-space feature representation is expressed as: Where: f 1×1 is a 1D convolution kernel, f 2D Represents a 2D convolution kernel; Step 4: Fusion of superpixel representation and pixel-level feature representation Let X p is the matrix representation of the pixel-level features output from step 3, let S s , S f and S c They are the superpixel-level output representations obtained by the spectral-spatial feature learning method of superpixel representation in step 2, and are decoded into pixel-level features using the superpixel indicator matrix H: s =HS s , X f =HS f , X c =HS c , then, the fusion of pixel-level and super-pixel-level features is written as: Z=MLP(X s ‖X f ‖X c ‖X p ) Among them: MLP (.) is a standard multi-layer perception network, "||" represents the data concatenation operation in the column direction of the matrix; Step 5: Hyperspectral pixel classification The superpixel-level representation and the pixel-level fusion feature representation Z are represented and the corresponding category prediction results are generated using the softmax(.) function:

2. The method for classifying small sample imbalanced hyperspectral images based on the fusion of graph convolutional network and convolutional network according to claim 1 is characterized in that The specific steps of step 2.1 are as follows: (1) From multiple bands of the hyperspectral image, three bands within the visible light range are selected to obtain a pseudo-color image of the hyperspectral image; (2) uniformly select q pixels in the pseudo-color image as revealing information, and input them together with the pseudo-color image into the SAM segmentation model to obtain a series of mask matrices; (3) Rearranging the obtained series of mask matrices into a superpixel indication matrix H, where the non-zero elements in each column indicate the set of superpixels; (4) Using the obtained superpixel indicator matrix H, the matrix-form hyperspectral image X is encoded into a superpixel representation: S=D -1 H T X Where: D is a diagonal matrix, and its diagonal element value is the sum of each column of the indicator matrix H.

3. The method for classifying small sample imbalanced hyperspectral images based on the fusion of graph convolutional network and convolutional network according to claim 2 is characterized in that In (1), the wavelength ranges of the three visible lights, red, green and blue, are 760-622 nanometers, 577-492 nanometers and 450-435 nanometers respectively.

4. The method for classifying small sample imbalanced hyperspectral images based on the fusion of graph convolutional network and convolutional network according to claim 1 is characterized in that In step 2.4, A c =αA s +(1-α)A f , α is the weighting parameter.

Images