Hyperspectral image open set classification method and system based on spectral unmixing

Abstract

The invention discloses a hyperspectral image open set classification method and system based on spectral unmixing, and belongs to the technical field of hyperspectral image processing. The method comprises the following steps: constructing a classification network model based on spectral unmixing; training the classification network model by using a first hyperspectral image of a known category to obtain a trained classification network model; and performing open set classification on a to-be-predicted second hyperspectral image by using the trained classification network model. According to the method, the classification network model based on spectral unmixing is constructed, the components of each pixel are analyzed in a spectral unmixing mode, and the influence of mixed pixels is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of hyperspectral image processing, and in particular to a hyperspectral image open set classification method and system based on spectral unmixing. Background Art

[0002] Hyperspectral image classification has a wide range of applications in surface monitoring, precision agriculture, and resource development. In traditional classification models, it is generally assumed that the target categories to be classified are all among the categories that the model has learned. However, in actual scenarios, only limited categories of training data can be collected, and there are many categories that are not included in the training data in the scenes that need to be classified. In order to enable the classification model to recognize untrained categories, an open set classification algorithm is proposed, which enables the model to determine whether the data belongs to an unknown category or to a category in the known categories.

[0003] At present, many types of open set classification algorithms have been proposed, such as one-class support vector machine (One-class SVM), Weibull-calibrated SVM (W-SVM), one-to-many support vector machine and other open set classification algorithms based on SVM; maximum classification method, nearest neighbor distance and other open set classification algorithms based on neighborhood; deep feature reconstruction method, adversarial sample generation method and other open set classification algorithms based on deep learning. However, these algorithms are designed for grayscale images or color images, and they are all classified at the image level, which is not suitable for pixel-level hyperspectral image classification tasks with multiple spectral bands.

[0004] Since deep learning has excellent performance in classification tasks, the open set classification of hyperspectral images in recent years is designed based on deep learning models. For example, the open set classification of hyperspectral images is performed using the "space-spectrum" convolutional neural network combined with the threshold judgment method. The method of extracting ground features by spectral-spatial feature reconstruction and then performing open set classification. The reciprocal point theory and neural network are combined to realize the open set classification of hyperspectral images. However, these methods are easily affected by mixed pixels, and the model complexity is high, making them difficult to apply in practice. Summary of the invention

[0005] The object of the present invention is to provide a hyperspectral image open set classification method and system based on spectral unmixing, so as to provide a hyperspectral image open set classification method which is not affected by mixed pixels.

[0006] To achieve the above object, the present invention provides the following solutions:

[0007] The present invention provides a hyperspectral image open set classification method based on spectral unmixing, the method comprising the following steps:

[0008] Construct a classification network model based on spectral unmixing.

[0009] The classification network model is trained using a first hyperspectral image of a known category to obtain a trained classification network model.

[0010] The trained classification network model is used to perform open set classification on the second hyperspectral image to be predicted.

[0011] Optionally, the classification network model includes: an encoding module, an abundance matrix generation module, an end member matrix generation module, a reconstruction module and a classification output module.

[0012] The output end of the encoding module is connected to the input end of the abundance matrix generation module and the input end of the end member matrix generation module respectively; the encoding module is used to preliminarily extract the image features of the input hyperspectral image block and pre-process the image features.

[0013] The output end of the endmember matrix generation module is connected to the input end of the reconstruction module, and the endmember matrix generation module is used to generate the endmember matrix of the input hyperspectral image block according to the preprocessed image features.

[0014] The output end of the abundance matrix generation module is connected to the input end of the reconstruction module and the input end of the classification output module respectively; the abundance matrix generation module is used to generate the abundance matrix of the input hyperspectral image block according to the preprocessed image features.

[0015] The reconstruction module is used to reconstruct according to the end member matrix and the abundance matrix to obtain a reconstructed hyperspectral image block.

[0016] The classification output module is used to output the probability of predicting the input hyperspectral image block as each category.

[0017] Optionally, the encoding module includes a convolutional layer, a batch normalization layer, a Relu activation layer and a first shape transformation layer connected in sequence.

[0018] The abundance matrix generation module includes a first fully connected layer, a nonlinear abundance generation layer and a first dimension splicing layer which are connected in sequence.

[0019] The end-member matrix generation module includes a first matrix transposition layer, a second fully connected layer, a second matrix transposition layer, a nonlinear end-member generation layer, and a second dimensional splicing layer which are connected in sequence.

[0020] The reconstruction module includes a matrix multiplication layer, a second shape transformation layer and a Sigmoid activation layer which are connected in sequence.

[0021] The classification output module includes a flat layer, a third fully connected layer and a Softmax output layer which are connected in sequence.

[0022] Optionally, the classification network model is trained using a first hyperspectral image of a known category to obtain a trained classification network model, specifically including:

[0023] The first hyperspectral image is diffused to surrounding areas by a number of pixels to obtain a diffused third hyperspectral image.

[0024] Taking each pixel in the first hyperspectral image as the center, an image block with a spatial size of A×A is selected in the third hyperspectral image as an input hyperspectral image block for training; wherein A=2a+1, a>1, and is an integer.

[0025] The classification network model is trained based on the input hyperspectral image blocks and their categories for training to obtain a trained classification network model.

[0026] Optionally, the loss function used to train the classification network model is:

[0027] L=L c +bL r .

[0028]

[0029] Among them, L represents the loss function, L c represents the classification loss, L r represents the reconstruction loss, b is the weighting coefficient, x is the input hyperspectral image block, To reconstruct the hyperspectral image block.

[0030] Optionally, the trained classification network model is used to perform open set classification on the second hyperspectral image to be predicted, specifically including:

[0031] The second hyperspectral image is diffused to surrounding areas by a number of pixels to obtain a diffused fourth hyperspectral image.

[0032] Taking each pixel in the second hyperspectral image as the center, an image block with a spatial size of A×A is selected in the fourth hyperspectral image as the input hyperspectral image block to be predicted; wherein A=2a+1, a>1, and is an integer.

[0033] The input hyperspectral image block to be predicted is input into the trained classification network model to obtain the probability of the input hyperspectral image block to be predicted being each category.

[0034] According to the probability of each category of the input hyperspectral image block to be predicted, it is determined whether the maximum probability is greater than a preset threshold to obtain a determination result.

[0035] If the judgment result indicates yes, then the category corresponding to the maximum probability is determined to be the category to which the input hyperspectral image block to be predicted belongs.

[0036] If the judgment result indicates no, it is determined that the category to which the input hyperspectral image block to be predicted belongs is an unknown category.

[0037] A hyperspectral image open set classification system based on spectral unmixing, the system is applied to the above method, the system comprises:

[0038] Model building module, used to build a classification network model based on spectral unmixing.

[0039] The model training module is used to train the classification network model using a first hyperspectral image of a known category to obtain a trained classification network model.

[0040] The classification module is used to perform open set classification on the second hyperspectral image to be predicted using the trained classification network model.

[0041] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0042] A computer-readable storage medium stores a computer program, which implements the above method when executed.

[0043] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0044] The embodiment of the present invention provides a method and system for open set classification of hyperspectral images based on spectral unmixing, the method comprising the following steps: constructing a classification network model based on spectral unmixing; training the classification network model using a first hyperspectral image of a known category to obtain a trained classification network model; and performing open set classification on a second hyperspectral image to be predicted using the trained classification network model. The present invention constructs a classification network model based on spectral unmixing, and uses spectral unmixing to analyze the components of each pixel, thereby avoiding the influence of mixed pixels. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 A flowchart of an open set classification method for hyperspectral images based on spectral unmixing provided by an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of an open set classification method for hyperspectral images based on spectral unmixing provided by an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of the structure of a classification network model based on spectral unmixing provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The object of the present invention is to provide a hyperspectral image open set classification method and system based on spectral unmixing, so as to provide a hyperspectral image open set classification method which is not affected by mixed pixels.

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Example 1

[0053] Embodiment 1 of the present invention provides an open set classification method for hyperspectral images based on spectral unmixing, such as Figure 1 and Figure 2 As shown, the method comprises the following steps:

[0054] Step 101, constructing a classification network model based on spectral unmixing.

[0055] For example, Figure 3 As shown, the spectral unmixing network includes a coding module, an abundance matrix generation module, an end member matrix generation module, a reconstruction module and a classification output module; the input end of the coding module is connected to the input image block, the output end of the coding module is connected to the input ends of the abundance matrix generation module and the end member matrix generation module, the output end of the end member matrix generation module is connected to the input end of the reconstruction module; the abundance matrix generation module is connected to the input ends of the reconstruction module and the classification output module at the same time.

[0056] The encoding module is composed of a convolutional layer, a batch normalization layer, a Relu activation layer and a first shape transformation layer in sequence; the convolution kernel size of the convolutional layer is 1×1, and the number of convolution kernels is 200.

[0057] The abundance matrix generation module consists of a first fully connected layer, a nonlinear abundance generation layer, and a first dimension concatenation layer. The number of output nodes of the first fully connected layer is 4, and its output matrix S is recorded as L For S L =[s1,s2,s3,s4], its shape size is N×4, where s1, s2, s3, s4 are S L The column vector in is N, where N is the total number of pixels in the input image block, and N = A × A, where A is the side length of the image block.

[0058] Among them, the output matrix S of the nonlinear abundance generation layer N , whose shape is N×6, and is composed of the output matrix S of the first fully connected layer L The Hadamard product of the column vectors in is obtained.

[0059] Right now, S N=[s1⊙s2, s1⊙s3, s1⊙s4, s2⊙s3, s2⊙s4, s3⊙s4].

[0060] where s i is the matrix S L The column vector in is the Hadamard product, which means the multiplication of corresponding elements of the vector.

[0061] The first dimension concatenation layer concatenates the outputs of the first fully connected layer and the nonlinear abundance generation layer in the feature dimension, that is, its output matrix S is: S = [S L ,S N ], whose shape and size is N×10.

[0062] The end-member matrix generation module is composed of a first matrix transposition layer, a second fully connected layer, a second matrix transposition layer, a nonlinear end-member generation layer, and a second dimension concatenation layer. The number of output nodes of the second fully connected layer is 4. The output matrix E of the second matrix transposition layer is recorded as L For E L =[e1,e2,e3,e4] T , whose shape size is 4×P, where e1, e2, e3, e4 are E L is the row vector of , and P is the number of bands of the hyperspectral image.

[0063] Among them, the output matrix E of the nonlinear end member generation layer is N , whose shape is 6×P, and is the output matrix E of the second matrix transpose layer L The Hadamard product of the two vectors in is obtained.

[0064] That is, E N =[e1⊙e2,e1⊙e3,e1⊙e4,e2⊙e3,e2⊙e4,e3⊙e4] T .

[0065] The second dimension concatenation layer concatenates the outputs of the second matrix transposition layer and the nonlinear end member generation layer in the feature dimension, that is, its output matrix E is: Its shape and size are 10×P.

[0066] The nonlinear abundance generation layer and the nonlinear endmember generation layer are used to establish a nonlinear spectral mixing pixel model in actual situations, so that the abundance matrix generation module and the endmember matrix generation module are closer to the actual physical model, and the ability of the model spectral unmixing is improved, thereby obtaining more accurate sub-pixel information.

[0067] The reconstruction module consists of a matrix multiplication layer, a second shape transformation layer, and a Sigmoid activation layer in sequence.

[0068] The classification output module is composed of a flat layer, a third fully connected layer, and a Softmax output layer in sequence; the number of output nodes in the Softmax output layer is consistent with the number of categories contained in the hyperspectral image X.

[0069] The encoding module is used for preliminary extraction of image features and data preprocessing of hyperspectral images; the abundance matrix generation module is used for extracting features of input hyperspectral image blocks and generating an abundance matrix of hyperspectral images; the endmember matrix generation module is used for generating an endmember matrix of hyperspectral images; the reconstruction module reconstructs hyperspectral image blocks according to the endmember matrix and the abundance matrix with image features; the classification output module outputs the probability of the network predicting the image blocks as various categories.

[0070] Step 102: training the classification network model using a first hyperspectral image of a known category to obtain a trained classification network model.

[0071] The three-dimensional hyperspectral image X is expanded by a pixels in all directions, and the expanded parts are copied and filled. Then, each pixel point in the original three-dimensional hyperspectral image X is taken as the center, and an image block with a spatial size of A×A is selected. The category of each block data corresponds to the category of its central pixel point; where a>1 and is an integer; A=2a+1.

[0072] As a specific implementation manner, in the embodiment, the first three-dimensional hyperspectral image X is expanded by 4 pixels in all directions, and the expanded parts are copied and filled. Then, with each pixel point in the first three-dimensional hyperspectral image X as the center, an image block with a spatial size of 9×9 is selected, and the category of the data of each image block corresponds to the category of its central pixel point.

[0073] The classification network is trained using image patches and their corresponding object labels, and the loss function L is composed of the reconstruction loss L r and classification loss L c Weighted.

[0074] The reconstruction loss L r for:

[0075]

[0076] Among them, x is the image block input to the network, is the output of the reconstruction module in the classification network; the classification loss L c The cross entropy loss function is used.

[0077] The total loss function L is:

[0078] L=L c +bL r .

[0079] Wherein, b is the weighting coefficient, b>0.

[0080] For example, b is 0.01, and the loss function is: L = L c +0.01L r .

[0081] Step 103: perform open set classification on the second hyperspectral image to be predicted using the trained classification network model.

[0082] Note that the first hyperspectral image X contains q types of data, and the second hyperspectral data Y to be predicted contains p types of data, where p>q. The hyperspectral data Y to be predicted is input into the dual loss function classification network, and q outputs {y1, y2, …y q}, let y i =max(y1,y2,…,y q ), and then judge according to the following formula to get category i:

[0083]

[0084] Among them, i is the maximum probability y i The corresponding category, M is the category of a certain hyperspectral image block in the second hyperspectral data Y, and T = 0.99995. In this way, the classification result of the hyperspectral image Y is obtained.

[0085] Example 2

[0086] Embodiment 2 of the present invention provides a hyperspectral image open set classification system based on spectral unmixing, the system is applied to the above method, and the system includes:

[0087] Model building module, used to build a classification network model based on spectral unmixing.

[0088] The model training module is used to train the classification network model using a first hyperspectral image of a known category to obtain a trained classification network model.

[0089] The classification module is used to perform open set classification on the second hyperspectral image to be predicted using the trained classification network model.

[0090] Example 3

[0091] Embodiment 3 of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0092] In addition, when the computer program in the above-mentioned memory is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0093] Example 4

[0094] Embodiment 4 of the present invention provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the above method when executed.

[0095] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0096] 1. The present invention uses a convolutional neural network to simulate a nonlinear spectral unmixing model, making the network model more consistent with the actual physical model, effectively improving the unmixing accuracy of hyperspectral images, and providing more accurate sub-pixel information. This method of exploring the components of each pixel can effectively reduce the negative impact of mixed pixels on image classification and make the classification results more robust.

[0097] 2. The present invention utilizes reconstruction constraint auxiliary classification network for training, so that the image reconstructed by the abundance matrix and the end member matrix is ​​as similar as possible to the input image, thereby ensuring the accuracy of the extracted abundance information.

[0098] 3. The network structure designed by the present invention is simple, the training parameters and the model occupy less memory, and it can be run on low-performance computers or chips, and has good practical application prospects.

[0099] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0100] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A hyperspectral image open set classification method based on spectral unmixing, characterized in that: The method comprises the following steps: Construct a classification network model based on spectral unmixing; Using a first hyperspectral image of a known category to train the classification network model to obtain a trained classification network model; The trained classification network model is used to perform open set classification on the second hyperspectral image to be predicted.

2. The open set classification method for hyperspectral images based on spectral unmixing according to claim 1 is characterized in that: The classification network model includes: an encoding module, an abundance matrix generation module, an end member matrix generation module, a reconstruction module and a classification output module; The output end of the encoding module is connected to the input end of the abundance matrix generation module and the input end of the end member matrix generation module respectively; the encoding module is used to preliminarily extract the image features of the input hyperspectral image block and pre-process the image features; The output end of the endmember matrix generation module is connected to the input end of the reconstruction module, and the endmember matrix generation module is used to generate the endmember matrix of the input hyperspectral image block according to the preprocessed image features; The output end of the abundance matrix generation module is connected to the input end of the reconstruction module and the input end of the classification output module respectively; the abundance matrix generation module is used to generate the abundance matrix of the input hyperspectral image block according to the preprocessed image features; The reconstruction module is used to reconstruct according to the end member matrix and the abundance matrix to obtain a reconstructed hyperspectral image block; The classification output module is used to output the probability of predicting the input hyperspectral image block as each category.

3. The open set classification method for hyperspectral images based on spectral unmixing according to claim 2 is characterized in that: The encoding module includes a convolutional layer, a batch normalization layer, a Relu activation layer and a first shape transformation layer connected in sequence; The abundance matrix generation module includes a first fully connected layer, a nonlinear abundance generation layer and a first dimension splicing layer connected in sequence; The end-member matrix generation module includes a first matrix transposition layer, a second fully connected layer, a second matrix transposition layer, a nonlinear end-member generation layer, and a second dimensional splicing layer connected in sequence; The reconstruction module includes a matrix multiplication layer, a second shape transformation layer and a Sigmoid activation layer connected in sequence; The classification output module includes a flat layer, a third fully connected layer and a Softmax output layer which are connected in sequence.

4. The open set classification method for hyperspectral images based on spectral unmixing according to claim 1 is characterized in that: The classification network model is trained using a first hyperspectral image of a known category to obtain a trained classification network model, specifically comprising: Diffuse the first hyperspectral image by a number of pixels in all directions to obtain a diffused third hyperspectral image; Taking each pixel in the first hyperspectral image as the center, select an image block with a spatial size of A×A in the third hyperspectral image as an input hyperspectral image block for training; wherein A=2a+1, a>1, and is an integer; The classification network model is trained based on the input hyperspectral image blocks and their categories for training to obtain a trained classification network model.

5. The open set classification method for hyperspectral images based on spectral unmixing according to claim 1 or 4, characterized in that: The loss function used to train the classification network model is: L=L c +bL r ; Among them, L represents the loss function, L represents the classification loss, L represents the reconstruction loss, and b is the weighted Coefficient, x is the input hyperspectral image patch, To reconstruct the hyperspectral image block.

6. The open set classification method for hyperspectral images based on spectral unmixing according to claim 1, characterized in that: The trained classification network model is used to perform open set classification on the second hyperspectral image to be predicted, including: The second hyperspectral image is diffused to surrounding areas by a number of pixels to obtain a diffused fourth hyperspectral image; Taking each pixel in the second hyperspectral image as the center, select an image block with a spatial size of A×A in the fourth hyperspectral image as the input hyperspectral image block to be predicted; wherein A=2a+1, a>1, and is an integer; Input the input hyperspectral image block to be predicted into the trained classification network model to obtain the probability of the input hyperspectral image block to be predicted being each category; According to the probability of each category of the input hyperspectral image block to be predicted, whether the maximum probability is greater than a preset threshold is determined to obtain a judgment result; If the judgment result indicates yes, then determining the category corresponding to the maximum probability is the category to which the input hyperspectral image block to be predicted belongs; If the judgment result indicates no, it is determined that the category to which the input hyperspectral image block to be predicted belongs is an unknown category.

7. A hyperspectral image open set classification system based on spectral unmixing, characterized in that: The system is applied to the method described in any one of claims 1 to 6, and the system comprises: Model building module, used to build a classification network model based on spectral unmixing; A model training module, used for training the classification network model using a first hyperspectral image of a known category to obtain a trained classification network model; The classification module is used to perform open set classification on the second hyperspectral image to be predicted using the trained classification network model.

8. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed, the method according to any one of claims 1 to 6 is implemented.

Images