Hyperspectral image classification method based on graph convolutional network

Abstract

The present invention discloses a hyperspectral image classification method based on a graph convolutional network, the method comprising: obtaining a target spectral image, dividing the target spectral image into a number of superpixel blocks; determining local structure graphs corresponding to the several superpixel blocks and a global structure graph corresponding to the target spectral image; and determining the target image category corresponding to the target spectral image according to the several local structure graphs and the global structure graph. The present invention retains the excellent homogeneous information within the superpixel by determining the local structure graph of each superpixel block, and comprehensively determines the image category of the target spectral image together with the global structure graph of the target spectral image, thereby improving the accuracy of classification. The method solves the problem that the existing target spectral image classification method based on graph convolution uses the mean or maximum value statistical variables within the superpixel block when characterizing superpixels, which seriously ignores the excellent homogeneous information within the superpixel, resulting in inaccurate classification results.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a hyperspectral image classification method based on a graph convolutional network. Background Art

[0002] In recent years, the development of hyperspectral imaging technology has significantly improved the spectral resolution of target spectral images. Fine spectral resolution is conducive to better expression of the spectral characteristics of objects in images and more accurate pixel-level classification. With the maturity of UAV technology and the deployment of hyperspectral satellites, there are more and more channels for obtaining target spectral image data. However, in the field of target spectral image classification, data labeling requires annotators to have certain prior knowledge, which is time-consuming and laborious. Thanks to the rapid development of hardware facilities such as graphics processing units (GPUs) and massive data, deep learning technology has made significant breakthroughs in the fields of computer vision and natural language processing, and has greatly surpassed traditional machine learning methods in performance. Compared with traditional machine learning, deep learning can adaptively learn the mapping relationship between massive data and label pairs without too much prior knowledge and manual settings. Therefore, deep learning technology has also been introduced into the field of target spectral image classification to better extract the spatial and spectral information of objects and further improve the accuracy of classification.

[0003] Although deep learning methods are fully capable of processing Euclidean spatial data such as images or sequences, they are difficult to implement in non-Euclidean data fields, such as graph structured data. At present, many methods for target spectral image classification based on graph convolution have been proposed, which use graph representation methods to better extract the neighboring information of pixels. In this type of method, it is usually necessary to use a superpixel segmentation method to convert the target spectral image into several superpixel blocks. Then, the superpixels are regarded as graph nodes, and their neighboring relationships are regarded as the edges of the nodes, so as to construct a graph representation of the target spectral image. However, the existing target spectral image classification methods based on graph convolution generally use the mean or maximum statistical variables within the superpixel block when characterizing superpixels. Although this method is simple, it seriously ignores the extremely excellent homogeneous information within the superpixel, resulting in the model achieving a suboptimal solution.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0005] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a hyperspectral image classification method based on graph convolutional network is provided, aiming to solve the problem that the existing target spectral image classification method based on graph convolution uses the mean or maximum value statistical variables within the superpixel block when characterizing superpixels, which seriously ignores the extremely excellent homogeneous information within the superpixel, resulting in inaccurate classification results.

[0006] The technical solution adopted by the present invention to solve the problem is as follows:

[0007] In a first aspect, an embodiment of the present invention provides a hyperspectral image classification method based on a graph convolutional network in a graph, wherein the method comprises:

[0008] Acquire a target spectral image, and divide the target spectral image into a plurality of super-pixel blocks;

[0009] Determine local structure graphs corresponding to the plurality of superpixel blocks and a global structure graph corresponding to the target spectral image;

[0010] According to the local structure graphs and the global structure graph, a target image category corresponding to the target spectral image is determined.

[0011] In one embodiment, dividing the target spectral image into a plurality of superpixel blocks comprises:

[0012] The target spectral image is input into a superpixel segmentation algorithm to obtain a plurality of superpixel blocks, wherein each superpixel block corresponds to a homogeneous region.

[0013] In one implementation, each of the superpixel blocks is used as a target superpixel block, and the process of determining the local structure graph includes:

[0014] Taking each pixel point in the target superpixel block as a first node, obtaining a first node matrix;

[0015] Obtaining a first adjacency matrix according to the adjacency relationships between all the first nodes;

[0016] The local structure graph corresponding to the target superpixel block is generated according to the first node matrix and the first adjacency matrix.

[0017] In one embodiment, the process of determining the global structure diagram includes:

[0018] Taking each of the super pixel blocks as a second node, obtaining a second node matrix;

[0019] Obtaining a second adjacency matrix according to the adjacency relationship between all the second nodes;

[0020] The global structure graph is generated according to the second node matrix and the second adjacency matrix.

[0021] In one implementation, determining the target image category corresponding to the target spectral image according to the plurality of local structure images and the global structure image includes:

[0022] Embedding a plurality of the local structure graphs into the global structure graph to obtain a target structure graph;

[0023] According to the target structure diagram, the target image category is determined.

[0024] In one implementation, embedding the plurality of local structure graphs into the global structure graph to obtain a target structure graph includes:

[0025] Determine, according to the superpixel block corresponding to each of the local structure graphs, an embedding position corresponding to each of the local structure graphs in the global structure graph;

[0026] According to the embedding position corresponding to each of the local structure graphs, each of the local structure graphs is embedded in the global structure graph to obtain the target structure graph.

[0027] In one implementation, determining the target image category according to the target structure diagram includes:

[0028] Inputting the target structure graph into an image classification model to obtain the target image category, wherein the image classification model includes a first convolution module, a second convolution module, a fusion module and a classification module;

[0029] The first convolution module is used to perform feature extraction according to the local structure graphs to obtain first feature vectors corresponding to the super pixel blocks respectively, and determine a first feature matrix according to all the first feature vectors;

[0030] The second convolution module is used to perform feature extraction according to the first feature matrix and the global structure graph to obtain second feature vectors corresponding to the plurality of super pixel blocks respectively;

[0031] The fusion module is used to fuse the first feature vector and the second feature vector corresponding to each super pixel block to obtain a fused feature vector corresponding to each super pixel block;

[0032] The classification module is used to generate probability distribution information according to the fusion feature vector corresponding to each super pixel block, and determine the target image category according to the probability distribution information.

[0033] In one implementation, the target structure graph includes a plurality of target structure graphs, each of which corresponds to a different hyperparameter; the image classification model includes a plurality of image classification models, each of which corresponds to a different hyperparameter; the hyperparameter is used to reflect the number of pixels in a unit superpixel block; and inputting the target structure graph into the image classification model to obtain the target image category includes:

[0034] The target structures Figure 1 Inputting a plurality of the image classification models one by one to obtain a plurality of classification results, wherein a pair of the target structure graphs and the image classification models having a corresponding relationship correspond to the same hyperparameters;

[0035] The target image category is determined according to the plurality of classification results, wherein the target image category is the image category with the largest number of votes among the plurality of classification results.

[0036] In a second aspect, an embodiment of the present invention further provides a hyperspectral image classification device based on a graph convolutional network in a graph, wherein the device comprises:

[0037] An image segmentation module is used to obtain a target spectral image and segment the target spectral image into a plurality of super-pixel blocks;

[0038] A structure determination module, used to determine local structure graphs corresponding to the plurality of superpixel blocks and a global structure graph corresponding to the target spectral image;

[0039] An image classification module is used to determine the target image category corresponding to the target spectral image according to the local structure images and the global structure image.

[0040] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a plurality of instructions are stored, wherein the instructions are suitable for being loaded and executed by a processor to implement any of the steps of the above-mentioned hyperspectral image classification method based on graph convolutional network in the graph.

[0041] Beneficial effects of the present invention: The embodiment of the present invention obtains a target spectral image, divides the target spectral image into a number of super-pixel blocks; determines the local structure graphs corresponding to the several super-pixel blocks and the global structure graph corresponding to the target spectral image; and determines the target image category corresponding to the target spectral image based on the several local structure graphs and the global structure graph. The present invention retains excellent homogeneous information within the super-pixel by determining the local structure graph of each super-pixel block and compares it with the global structure graph of the target spectral image. Figure 1 The image category of the target spectral image is comprehensively determined, which improves the classification accuracy. The existing target spectral image classification method based on graph convolution solves the problem that when representing superpixels, the mean or maximum statistical variables in the superpixel block are used, which seriously ignores the extremely excellent homogeneous information in the superpixel, resulting in inaccurate classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 It is a flowchart of a hyperspectral image classification method based on a graph convolutional network provided in an embodiment of the present invention.

[0044] Figure 2 It is a working principle diagram of the image classification model provided by an embodiment of the present invention.

[0045] Figure 3 It is a flow chart of an integrated learning algorithm provided by an embodiment of the present invention.

[0046] Figure 4 It is a schematic diagram of the internal modules of the hyperspectral image classification device based on the graph convolutional network in the figure provided by an embodiment of the present invention.

[0047] Figure 5 It is a principle block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention discloses a hyperspectral image classification based on a graph convolutional network. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0049] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0050] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0051] In recent years, the development of hyperspectral imaging technology has significantly improved the spectral resolution of target spectral images. Fine spectral resolution is conducive to better expression of the spectral characteristics of objects in images and more accurate pixel-level classification. With the maturity of UAV technology and the deployment of hyperspectral satellites, there are more and more channels for obtaining target spectral image data. However, in the field of target spectral image classification, data labeling requires annotators to have certain prior knowledge, which is time-consuming and laborious. Thanks to the rapid development of hardware facilities such as graphics processing units (GPUs) and massive data, deep learning technology has made significant breakthroughs in the fields of computer vision and natural language processing, and has greatly surpassed traditional machine learning methods in performance. Compared with traditional machine learning, deep learning can adaptively learn the mapping relationship between massive data and label pairs without too much prior knowledge and manual settings. Therefore, deep learning technology has also been introduced into the field of target spectral image classification to better extract the spatial and spectral information of objects and further improve the accuracy of classification.

[0052] Although deep learning methods are fully capable of processing Euclidean spatial data such as images or sequences, they are difficult to implement in non-Euclidean data fields, such as graph structured data. At present, many methods for target spectral image classification based on graph convolution have been proposed, which use graph representation methods to better extract the neighboring information of pixels. In this type of method, it is usually necessary to use a superpixel segmentation method to convert the target spectral image into several superpixel blocks. Then, the superpixels are regarded as graph nodes, and their neighboring relationships are regarded as the edges of the nodes, so as to construct a graph representation of the target spectral image. However, the existing target spectral image classification methods based on graph convolution generally use the mean or maximum statistical variables within the superpixel block when characterizing superpixels. Although this method is simple, it seriously ignores the extremely excellent homogeneous information within the superpixel, resulting in the model achieving a suboptimal solution.

[0053] In view of the above-mentioned defects of the prior art, the present invention provides a hyperspectral image classification method based on graph-in-graph convolutional network, the method comprising: obtaining a target spectral image, dividing the target spectral image into a number of super-pixel blocks; determining the local structure graphs corresponding to the several super-pixel blocks and the global structure graph corresponding to the target spectral image; and determining the target image category corresponding to the target spectral image based on the several local structure graphs and the global structure graph. The present invention retains the excellent homogeneous information within the super-pixel by determining the local structure graph of each super-pixel block, and compares it with the global structure graph of the target spectral image. Figure 1 The image category of the target spectral image is comprehensively determined, which improves the classification accuracy. The existing target spectral image classification method based on graph convolution solves the problem that when representing superpixels, the mean or maximum statistical variables in the superpixel block are used, which seriously ignores the extremely excellent homogeneous information in the superpixel, resulting in inaccurate classification results.

[0054] like Figure 1 As shown, the method comprises the following steps:

[0055] Step S100: Acquire a target spectral image, and divide the target spectral image into a plurality of super-pixel blocks.

[0056] Specifically, the target spectral image in this embodiment can be any hyperspectral image with a spectral resolution within the range of 10λ. In order to better extract the neighboring information between pixels, this embodiment needs to first convert the target spectral image into a number of superpixel blocks. Among them, a superpixel block is a small area composed of a series of pixels with adjacent positions and similar characteristics such as color, brightness, and texture. Most of these small areas retain effective information for further image segmentation, and generally do not destroy the boundary information of objects in the image.

[0057] In one implementation, step S100 specifically includes the following steps:

[0058] Step S101: input the target spectral image into a superpixel segmentation algorithm to obtain a plurality of superpixel blocks, wherein each superpixel block corresponds to a homogeneous region.

[0059] Specifically, the superpixel segmentation algorithm is a simple and efficient method for constructing superpixels. The target spectral image is input into the superpixel segmentation algorithm, which will identify all pixels in the target spectral image and cluster pixels with similar spatial distances and similar colors into a superpixel block. Therefore, each superpixel block is a homogeneous area.

[0060] In one implementation, the superpixel segmentation algorithm may be a SLIC superpixel segmentation algorithm, wherein the working principle of the SLIC superpixel segmentation algorithm is as follows:

[0061] Assume that the target spectral image is Where H, W, and D represent the height, width, and spectral dimension of the target spectral image, respectively, and each pixel in the image After superpixel segmentation, a target spectral image I is segmented into a superpixel set B = {B 1 , ..., B S}. The superpixel set contains a total of s superpixels, and the sth superpixel Covers n s hyperspectral pixels. In addition, f σ (x i )=s is a mapping function from hyperspectral pixels to superpixel blocks.

[0062] In another implementation, the superpixel segmentation algorithm may also be a fast shift image segmentation algorithm or a watershed segmentation algorithm.

[0063] like Figure 1 As shown, the method further comprises the following steps:

[0064] Step S200: determining local structure graphs corresponding to a plurality of superpixel blocks and a global structure graph corresponding to the target spectral image.

[0065] Specifically, since the excellent homogeneous information in the superpixel block can improve the accuracy of image classification, this embodiment needs to determine the image structure in each superpixel block, that is, retain the local structure map corresponding to each superpixel block. At the same time, in order to improve the representation ability and discriminability of features in the target spectral image, this embodiment also needs to mine the image structure between superpixels, that is, retain the global structure map corresponding to the target spectral image. Accurate image classification can be achieved through the local structure map and global structure map of each superpixel block.

[0066] In one implementation, each of the superpixel blocks is used as a target superpixel block, and the process of determining the local structure graph specifically includes the following steps:

[0067] Step S201, taking each pixel point in the target superpixel block as a first node to obtain a first node matrix;

[0068] Step S202: obtaining a first adjacency matrix according to the adjacency relations between all the first nodes;

[0069] Step S203: Generate the local structure graph corresponding to the target superpixel block according to the first node matrix and the first adjacency matrix.

[0070] Specifically, the operation of determining the local structure diagram of each super pixel block in this embodiment is the same, so this embodiment takes a super pixel block as an example to clearly explain how to determine the local structure diagram, and the super pixel block is the target super pixel block. First, each pixel point in the target super pixel block is taken as a node, wherein the node corresponding to the pixel point is defined as the first node in this embodiment, and the first node matrix can be formed by the first nodes corresponding to all the pixel points. Then the adjacency relationship between all the first nodes is obtained. For example, taking four neighborhoods as an example, each first node has an adjacency relationship with its four first nodes in the east, south, west and north; taking eight neighborhoods as an example, each first node has an adjacency relationship with its eight first nodes in the east, south, west, north, southeast, northeast, southwest and northwest. And the first adjacency matrix is ​​generated based on the adjacency relationship of each first node. Since the first node matrix and the first adjacency matrix can reflect the node features in the target super pixel block and the adjacency information between nodes, the first node matrix and the first adjacency matrix are used as homogeneous information in the target super pixel block, i.e., the local structure diagram.

[0071] For example, in superpixel block B s In the example, for each pixel x s，i is a node, and its four neighborhoods (upper, lower, left, and right) are neighbor nodes to construct the corresponding local structure graph representation Gi ＝ {X S , ε s},in is the node matrix of the graph, and ε s ={(x s，i , x s，j )} is the edge set of the graph. Then, the corresponding adjacency matrix is ​​constructed based on the edge set

[0072] In one implementation, the process of determining the global structure diagram specifically includes the following steps:

[0073] Step S204, taking each of the super pixel blocks as a second node to obtain a second node matrix;

[0074] Step S205: obtaining a second adjacency matrix according to the adjacency relationship between all the second nodes;

[0075] Step S206: Generate the global structure graph according to the second node matrix and the second adjacency matrix.

[0076] Specifically, in order to extract the global structure diagram of the target spectral image, the present embodiment uses each super pixel block as a node, wherein the node corresponding to the super pixel block is defined as the second node in the present embodiment, and the second nodes corresponding to all super pixel blocks can form a second node matrix. Then the adjacency relationship between all second nodes is obtained. For example, taking four neighborhoods as an example, each second node has an adjacency relationship with its four second nodes in the east, south, west, and north; taking eight neighborhoods as an example, each second node has an adjacency relationship with its eight second nodes in the east, south, west, north, southeast, northeast, southwest, and northwest. And a second adjacency matrix is ​​generated based on the adjacency relationship of each second node. Since the second node matrix and the second adjacency matrix can reflect the node features in the target spectral image and the adjacency information between nodes, the present embodiment uses the second node matrix and the second adjacency matrix as the global structure diagram of the target spectral image.

[0077] For example, according to the SLIC segmentation result map, each superpixel B s A global structure graph is constructed for the adjacency relationship between nodes and superpixels as edges. In another implementation, edge sets may also be constructed based on the Euclidean distance or Mahalanobis distance between the superpixels.

[0078] like Figure 1 As shown, the method further comprises the following steps:

[0079] Step S300: determining a target image category corresponding to the target spectral image according to the plurality of local structure graphs and the global structure graph.

[0080] Specifically, since the local structure map retains the extremely good homogeneous information within the superpixel block, and the global structure map can reflect the global information of the target spectral image, this embodiment uses a combination of the local structure map and the global structure map to classify the target spectral image, which can improve the representation ability and discriminability of the features, thereby improving the accuracy of image classification.

[0081] In one implementation, step S300 specifically includes the following steps:

[0082] Step S301, embedding a plurality of the local structure diagrams into the global structure diagram to obtain a target structure diagram;

[0083] Step S302: Determine the target image category according to the target structure diagram.

[0084] Specifically, in order to improve the representation ability and discriminability of features, this embodiment embeds each local structure graph into the global structure graph to obtain a target structure graph of a graph-in-graph structure. The target structure graph fully expresses the information of the ground objects within and between superpixels, so by classifying the target spectral image based on the target structure graph, a more accurate classification result, i.e., the target image category, can be obtained.

[0085] In one implementation, step S301 specifically includes the following steps:

[0086] Step S3011: determining the embedding position corresponding to each local structure graph in the global structure graph according to the superpixel block corresponding to each local structure graph;

[0087] Step S3012: embed each of the local structure graphs into the global structure graph according to the embedding position corresponding to each of the local structure graphs to obtain the target structure graph.

[0088] Specifically, since the global structure graph regards each superpixel block as a node, and each local structure graph corresponds to a superpixel block, in this embodiment, the node corresponding to each superpixel block is regarded as an embedding position. Then, according to the correspondence between the local structure graph and the superpixel block, each local structure graph is embedded into its corresponding embedding position, thereby obtaining the target structure graph of the graph structure in the figure.

[0089] For example, each node in the global structure graph is embedded with a corresponding local structure graph G i The graph structure is formed. Therefore, the node set of the global structure graph is G int = {G 1 , ..., G S}, and its edge set is ε ext ={(G i , G j )}, and its corresponding adjacency matrix is Therefore, G ext = {G int , ε ext}.

[0090] In one implementation, step S302 specifically includes the following steps:

[0091] Step S3021, inputting the target structure diagram into an image classification model to obtain the target image category, wherein the image classification model includes a first convolution module, a second convolution module, a fusion module and a classification module;

[0092] Step S3022, the first convolution module is used to perform feature extraction according to the plurality of local structure graphs, obtain first feature vectors corresponding to the plurality of super pixel blocks respectively, and determine a first feature matrix according to all the first feature vectors;

[0093] Step S3023, the second convolution module is used to perform feature extraction according to the first feature matrix and the global structure graph to obtain second feature vectors corresponding to the plurality of super pixel blocks respectively;

[0094] Step S3024, the fusion module is used to fuse the first feature vector and the second feature vector corresponding to each super pixel block to obtain a fused feature vector corresponding to each super pixel block;

[0095] Step S3025, the classification module is used to generate probability distribution information according to the fusion feature vector corresponding to each superpixel block, and determine the target image category according to the probability distribution information.

[0096] In order to classify the target spectrum image, this embodiment pre-trains an image classification model, and inputs the target structure diagram corresponding to the target spectrum image into the image classification model. The image classification model extracts features from the target structure diagram and classifies it based on the extracted features, thereby outputting the target image category corresponding to the target spectrum image. The image classification model mainly consists of four modules: a first convolution module, a second convolution module, a fusion module, and a classification module.

[0097] Specifically, the input of the first convolution module is the information of each local structure graph in the target structure graph. In the first convolution module, each local structure graph will be feature extracted to obtain the first feature vector corresponding to each local structure graph, wherein the first feature vector can reflect the local features of the ground object in the superpixel. Finally, based on each first feature vector, the first feature matrix used to characterize the features of each local structure graph is output.

[0098] For example, Figure 2 As shown, the first convolution module The node feature matrix of the local structure graph and the adjacency matrix As input, after graph convolution operation and mean pooling, the representation vector of the local structure graph is output Specifically,

[0099]

[0100] in, is the adjacency matrix with its own connections, is the degree matrix of the adjacency matrix, W is the trainable parameter matrix, and K controls the order of the node. Considering the local similarity of the spectrum, L is imposed on the parameter matrix W. 1 The regularization term, i.e. L 1 =∑ ij |W ij |. During the back propagation process, its derivative is L 1 The derivative of the regularization term forces the elements of the parameter matrix W to ij The parameter tends to zero, making the parameters sparse. The result of parameter sparseness can disconnect some unnecessary connections between neuron layers, allowing deep neurons to connect only to some shallow neurons, achieving the effect of local perception, so that the local structure graph convolution can adaptively focus on the local correlation of the spectrum. In another implementation, the convolution operation of the first convolution module can be replaced by graph convolution (GCN) or graph attention (GAT), etc.

[0101] Specifically, the input of the second convolution module is the information in the first feature matrix and the global structure graph, and it outputs the hidden representation of the nodes corresponding to each superpixel block based on the information in the first feature matrix and the global structure graph, that is, the second feature vector, wherein the second feature vector can reflect the global characteristics of the ground objects between superpixels.

[0102] For example, Figure 2 As shown in the figure, after the feature extraction of the first convolution module, all local structure graphs correspond to a corresponding representation vector This representation vector contains the homogeneous information within the superpixel. At the same time, the first eigenvectors of all local structure graphs constitute the node feature matrix of the global structure graph. Similarly, the second convolution module uses the node matrix X of the global structure graph ext and the adjacency matrix A ext As input, output the hidden representation of each node l represents the number of layers in the model. Different from the first convolution module, the second convolution module of each layer contains graph convolution operation, activation function relu (·) and batch normalization BN (·). Specifically, in, is the hidden representation of the lth layer, and Θ l is the trainable parameter of the lth layer. When l=0, After l layers of operations, the final representation of the global structure graph node is In another implementation, the convolution operation of the second convolution module can be replaced by graph attention (GAT) or the like.

[0103] Specifically, the input of the fusion module is the first feature vector and the second feature vector corresponding to each superpixel block, and the two are fused in the fusion module, and the fusion feature vector corresponding to each superpixel block is output. For example, the fusion module can integrate the local features of the ground object in the superpixel and the global features between superpixels to form a fusion feature H GiG =H int +H ext .

[0104] Specifically, the input of the classification module is the fusion feature vector of each superpixel block output by the fusion module. A multi-layer perceptron can be set in the classification module, and the multi-layer perceptron can automatically calculate the probability distribution information based on the input fusion feature vector. Since the probability distribution information can reflect the probability that the target spectral image belongs to each image category, the target image category corresponding to the target spectral image can be output based on the probability distribution information.

[0105] For example, the fusion feature H GiG The corresponding probability distribution vector O is obtained by the multi-layer perceptron MLP(·). In the training process, the cross entropy loss function is used to calculate the loss value between the prediction and the true label, that is:

[0106]

[0107] Among them, y i is pixel x i The true label of . Therefore, the objective function of the model is:

[0108] L=L entropy +λL 1

[0109] Among them, λ is the hyperparameter adjustment L 1 The weight of the regularization term.

[0110] In one implementation, the target structure graph includes a plurality of target structure graphs, each of which corresponds to a different hyperparameter; the image classification model includes a plurality of image classification models, each of which corresponds to a different hyperparameter, and the hyperparameter is used to reflect the number of pixels in a unit superpixel block; and the target structure graph is input into the image classification model to obtain the target image category, which specifically includes the following steps:

[0111] Step S30211: multiple target structures Figure 1 Inputting a plurality of the image classification models one by one to obtain a plurality of classification results, wherein a pair of the target structure graphs and the image classification models having a corresponding relationship correspond to the same hyperparameters;

[0112] Step S30212: Determine the target image category according to the classification results, wherein the target image category is the image category with the largest number of votes among the classification results.

[0113] Specifically, in order to adapt to the target spectral images of scenes of different scales, this embodiment introduces a hyperparameter (the number of pixels contained in each superpixel) to indirectly control the setting of the number of superpixels in the superpixel segmentation algorithm. At the same time, in order to avoid the influence of the setting of the number of superpixels in the superpixel segmentation algorithm on the performance of the image classification model, it is possible to set multiple image classification models, each of which corresponds to different hyperparameters to adapt to scenes of different scales. Among them, the target structure diagram corresponding to each hyperparameter passes through the corresponding image classification model, and each image classification model can output a classification result. These classification results can use the principle of relative majority voting to determine the final target image category of the target spectral image.

[0114] For example, Figure 3 As shown in the figure, in order to adapt to the target spectral image data of scenes of different scales, the hyperparameter n is introduced. p (the number of pixels contained in each superpixel) is used to indirectly control the setting of the number of superpixels in the superpixel segmentation algorithm, that is, At the same time, in order to avoid the influence of the number of superpixels in the superpixel segmentation algorithm on the classification performance of the model, four hyperparameters n are set in advance. p ∈[50, 100, 150, 200] corresponds to scenes of different scales. Each hyperparameter n p The corresponding images in the figure are passed through different GiGCNs to obtain the corresponding classification results. These four classification results use the principle of relative majority voting in the label space to decide the final prediction result of each pixel, which improves the robustness of the model.

[0115] In another implementation manner, an absolute majority voting method, a weighted voting method, or the like may be used to determine the target image category corresponding to the target spectral image.

[0116] In general, the technical solution of the present invention is a graph convolutional neural network (Graph-in-Graph Convolutional Network, GiGCN) based on graph-in-graph (GiG) for target spectral image classification. First, a simple linear iterative clustering method (SLIC) is used to over-segment the target spectral image to form a number of superpixel blocks. In the case of over-segmentation, each superpixel block is equivalent to a homogeneous region. These homogeneous regions are converted into a graph structure representation in the form of pixels in the region as nodes and the four neighborhoods of the pixels as adjacency relationships, which is called an internal graph. At the same time, an external graph structure representation is constructed with superpixels as nodes and the neighboring relationships between superpixels as edges. Then, each node of the external graph is embedded with a corresponding internal graph to form a graph structure in the graph, which fully expresses the information of the objects within the superpixel and between the superpixels, that is, local and global information. After that, a GiGCN network is constructed to hierarchically extract the hierarchical information of the graph structure in the graph. GiGCN consists of an internal graph convolution and an external graph convolution, and gradually extracts the features of the internal graph and the external graph in a hierarchical manner. In the internal graph convolution, the pixel features in the homogeneous region interact along the edge of the internal graph; while in the external graph, the homogeneous region features belonging to the same type of landform interact along the edge of the external graph. Afterwards, the features that combine the local and global information of the landform are sent to the classifier to obtain the label of the external graph node (that is, each internal graph), and the pixels in the homogeneous region share the label. Finally, the classification results of the superpixel segmentation map with different hyperparameter settings are determined by the relative majority voting method to determine the final category label of the pixel, so as to reduce the impact of the hyperparameter setting on the classification performance of the model.

[0117] The advantages of the present invention are:

[0118] 1. The local and global information of objects within and between superpixels is fully mined to improve the representation and discriminability of features. At the same time, an ensemble learning-based method is used to overcome the deficiency that a single superpixel is difficult to segment spectral images of targets in scenes of different scales.

[0119] 2. It can effectively extract the features of internal and external graphs in the graph-in-graph data structure in a hierarchical manner, and fuse them into a fused feature with high discriminability and representation ability, thereby improving the classification accuracy of the model.

[0120] Based on the above embodiments, the present invention also provides a hyperspectral image classification device based on a graph convolutional network, such as Figure 4 As shown, the device comprises:

[0121] An image segmentation module 01 is used to obtain a target spectral image and segment the target spectral image into a plurality of super-pixel blocks;

[0122] A structure determination module 02, used to determine local structure graphs corresponding to the plurality of superpixel blocks and a global structure graph corresponding to the target spectral image;

[0123] The image classification module 03 is used to determine the target image category corresponding to the target spectral image according to the local structure images and the global structure image.

[0124] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Figure 5 As shown. The terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a hyperspectral image classification method based on the graph convolution network in the figure is implemented. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0125] Those skilled in the art will understand that Figure 5 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0126] In one implementation, one or more programs are stored in the memory of the terminal, and are configured to be executed by one or more processors, wherein the one or more programs include instructions for performing a hyperspectral image classification method based on a graph convolutional network in a graph.

[0127] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0128] In summary, the present invention discloses a hyperspectral image classification based on graph convolutional network, the method comprising: obtaining a target spectral image, dividing the target spectral image into a number of super-pixel blocks; determining the local structure graphs corresponding to the several super-pixel blocks and the global structure graph corresponding to the target spectral image; determining the target image category corresponding to the target spectral image according to the several local structure graphs and the global structure graph. The present invention retains the excellent homogeneous information within the super-pixel by determining the local structure graph of each super-pixel block, and compares it with the global structure graph of the target spectral image. Figure 1 The image category of the target spectral image is comprehensively determined, which improves the classification accuracy. The existing target spectral image classification method based on graph convolution solves the problem that when representing superpixels, the mean or maximum statistical variables in the superpixel block are used, which seriously ignores the extremely excellent homogeneous information in the superpixel, resulting in inaccurate classification results.

[0129] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A hyperspectral image classification method based on graph convolutional network in graph, characterized in that: The method comprises: Acquire a target spectral image, and divide the target spectral image into a plurality of super-pixel blocks; Determine local structure graphs corresponding to the plurality of superpixel blocks and a global structure graph corresponding to the target spectral image; Determining a target image category corresponding to the target spectral image according to the plurality of local structure graphs and the global structure graph; Determining the target image category corresponding to the target spectral image according to the plurality of local structure graphs and the global structure graph includes: Embedding a plurality of the local structure graphs into the global structure graph to obtain a target structure graph; Determining the target image category according to the target structure diagram; Determining the target image category according to the target structure diagram includes: Inputting the target structure graph into an image classification model to obtain the target image category, wherein the image classification model includes a first convolution module, a second convolution module, a fusion module and a classification module; The first convolution module is used to perform feature extraction according to the local structure graphs to obtain first feature vectors corresponding to the super pixel blocks respectively, and determine a first feature matrix according to all the first feature vectors; The second convolution module is used to perform feature extraction according to the first feature matrix and the global structure graph to obtain second feature vectors corresponding to the plurality of super pixel blocks respectively; The fusion module is used to fuse the first feature vector and the second feature vector corresponding to each super pixel block to obtain a fused feature vector corresponding to each super pixel block; The classification module is used to generate probability distribution information according to the fusion feature vector corresponding to each super pixel block, and determine the target image category according to the probability distribution information.

2. The hyperspectral image classification method based on graph convolutional network in a graph according to claim 1 is characterized in that: The step of dividing the target spectral image into a plurality of super-pixel blocks comprises: The target spectral image is input into a superpixel segmentation algorithm to obtain a plurality of superpixel blocks, wherein each superpixel block corresponds to a homogeneous region.

3. The hyperspectral image classification method based on graph convolutional network in a graph according to claim 1 is characterized in that: Taking each of the super pixel blocks as a target super pixel block, the process of determining the local structure graph includes: Taking each pixel point in the target superpixel block as a first node, obtaining a first node matrix; Obtaining a first adjacency matrix according to the adjacency relationships between all the first nodes; The local structure graph corresponding to the target superpixel block is generated according to the first node matrix and the first adjacency matrix.

4. The hyperspectral image classification method based on graph convolutional network in a graph according to claim 1 is characterized in that: The process of determining the global structure diagram includes: Taking each of the super pixel blocks as a second node, obtaining a second node matrix; Obtaining a second adjacency matrix according to the adjacency relationships between all the second nodes; The global structure graph is generated according to the second node matrix and the second adjacency matrix.

5. The hyperspectral image classification method based on graph convolutional network in a graph according to claim 1 is characterized in that: The embedding of the plurality of local structure graphs into the global structure graph to obtain a target structure graph comprises: Determine, according to the superpixel block corresponding to each of the local structure graphs, an embedding position corresponding to each of the local structure graphs in the global structure graph; According to the embedding position corresponding to each of the local structure graphs, each of the local structure graphs is embedded in the global structure graph to obtain the target structure graph.

6. The hyperspectral image classification method based on graph convolutional network in a graph according to claim 1 is characterized in that: The target structure graph includes a plurality of target structure graphs, each of which corresponds to a different hyperparameter. The image classification model includes a plurality of image classification models, each of which corresponds to a different hyperparameter. The hyperparameter is used to reflect the number of pixels in a unit superpixel block. The target structure graph is input into the image classification model to obtain the target image category, including: Inputting the target structure graphs into the image classification models one by one to obtain a plurality of classification results, wherein a pair of the target structure graphs and the image classification models having a corresponding relationship correspond to the same hyperparameters; The target image category is determined according to the plurality of classification results, wherein the target image category is the image category with the largest number of votes among the plurality of classification results.

7. A hyperspectral image classification device based on graph convolutional network in the graph, characterized in that: The device comprises: An image segmentation module is used to obtain a target spectral image and segment the target spectral image into a plurality of super-pixel blocks; A structure determination module, used to determine local structure graphs corresponding to the plurality of superpixel blocks and a global structure graph corresponding to the target spectral image; An image classification module, used for determining a target image category corresponding to the target spectral image according to the local structure images and the global structure image; Determining the target image category corresponding to the target spectral image according to the plurality of local structure graphs and the global structure graph includes: Embedding a plurality of the local structure graphs into the global structure graph to obtain a target structure graph; Determining the target image category according to the target structure diagram; Determining the target image category according to the target structure diagram includes: Inputting the target structure graph into an image classification model to obtain the target image category, wherein the image classification model includes a first convolution module, a second convolution module, a fusion module and a classification module; The first convolution module is used to perform feature extraction according to the local structure graphs to obtain first feature vectors corresponding to the super pixel blocks respectively, and determine a first feature matrix according to all the first feature vectors; The second convolution module is used to perform feature extraction according to the first feature matrix and the global structure graph to obtain second feature vectors corresponding to the plurality of super pixel blocks respectively; The fusion module is used to fuse the first feature vector and the second feature vector corresponding to each super pixel block to obtain a fused feature vector corresponding to each super pixel block; The classification module is used to generate probability distribution information according to the fusion feature vector corresponding to each super pixel block, and determine the target image category according to the probability distribution information.

8. A computer-readable storage medium having a plurality of instructions stored thereon, characterized in that: The instructions are suitable for being loaded and executed by a processor to implement the steps of the hyperspectral image classification method based on graph convolutional network in a graph as described in any one of claims 1 to 6 above.

Images