Classification method, classification device and storage medium

Abstract

The present disclosure relates to a classification method, a classification device, and a storage medium for estimating object classification based on hyperspectral imagery. According to one embodiment of the present disclosure, the classification method includes: extracting point-level classification features for each of a plurality of points within a contour region of a single object of interest in a hyperspectral image using a first neural network-based point-level classifier; and estimating a lower-level classification of the single object of interest relative to the type of interest using a second neural network-based block-level classifier based on block-level classification features associated with counts of the point-level classification features of the plurality of points with respect to each cluster feature group in a set of cluster feature groups. Advantageous effects of the method, device, and storage medium of the present disclosure include at least improving classification accuracy.

Description

Technical Field

[0001] The present disclosure generally relates to image processing, and more particularly, to a classification method, a classification device, and a storage medium for estimating object classification based on hyperspectral images. Background Art

[0002] Determining the tree composition of a forest is crucial. Traditional manual on-site identification and labeling is time-consuming and labor-intensive. Remote sensing technology is a recently emerging and effective method for monitoring and studying forests. For example, hyperspectral imagery can be used to determine the tree species in various areas of a forest.

[0003] Hyperspectral images contain not only position information, but also spectral information. Each point in a hyperspectral image can have position coordinates. In the spectral dimension, each point in a hyperspectral image also has hyperspectral information. For example, the exemplary wavelength range of a hyperspectral image is selected to be 380nm to 780nm, and the wavelength range can be divided into 80 bands (wavelength segments); each point in a hyperspectral image has a hyperspectral value (ordinate) curve with respect to the band (abscissa). The hyperspectral value curve of each substance will have different characteristics, so object recognition or classification can be performed based on hyperspectral images.

[0004] Hyperspectral images acquired by remote sensing satellites, for example, can be used to analyze the distribution of objects on the Earth's surface and generate meaningful results. For example, by analyzing hyperspectral images of a forest area acquired by remote sensing satellites, information about the distribution of trees within the area can be obtained. For example, the number and distribution area of ​​various tree types within the forest area can be determined.

[0005] However, object classification using hyperspectral images is challenging. For example, high hyperspectral similarity between objects of different classes (different objects with the same spectrum) and hyperspectral variations at different points on the same object (same object with different spectra) increase the difficulty of correctly classifying objects based on hyperspectral images.

[0006] In recent years, with the development of neural networks, neural networks have also been used in hyperspectral image analysis. For example, a trained neural network-based classification model can be used to classify objects appearing in hyperspectral images. For example, a segmented hyperspectral image showing the outline of a single tree is input into the classification model. The classification model can extract classification features for each of the multiple points within the tree outline (also called hyperspectral points) based on the hyperspectral information of each point within the tree outline. Considering that the classification features of each of the multiple points within the tree outline may vary, using the classification features of a single point to determine the classification of the corresponding tree may result in large deviations. Therefore, it is possible to consider averaging these classification features and using the obtained average features to estimate the classification of the corresponding tree. However, the classification results of such a classification method may still be unsatisfactory or there is room for improvement. Summary of the Invention

[0007] A brief overview of the present disclosure will be provided below in order to provide a basic understanding of certain aspects of the present disclosure. It should be understood that this overview is not an exhaustive overview of the present disclosure. It is not intended to identify key or important portions of the present disclosure, nor is it intended to limit the scope of the present disclosure. Its purpose is simply to present certain concepts in a simplified form as a prelude to a more detailed description that will be discussed later.

[0008] According to one aspect of the present disclosure, a computer-implemented classification method for estimating object classification based on hyperspectral images is provided. The classification method includes: extracting point-level classification features of each of a plurality of points within a contour region of a single object of interest in a hyperspectral image using a point-level classifier based on a first neural network; and estimating a lower-level classification of the single object of interest relative to the type of interest using a block-level classifier based on a second neural network based on block-level classification features associated with counts of the point-level classification features of the plurality of points with respect to each cluster feature group in a set of cluster feature groups; wherein the point-level classifier extracts point-level classification features of the corresponding point based on a hyperspectral value of each point; the point-level classifier is configured to estimate the classification of the single point with respect to the lower-level classification set based on hyperspectral information of the single point within the contour region; the contour region of the single object of interest is an area indicated by a contour of the single object of interest or an outer bounding box of the contour of the single object of interest in the hyperspectral image; the set of cluster feature groups is determined by clustering the point-level classification features extracted by the point-level classifier from a predetermined set of hyperspectral images; and the counts of the point-level classification features of the plurality of points with respect to each cluster feature group in the set of cluster feature groups indicate counts of point-level classification features of the plurality of points that fall into the corresponding cluster feature group.

[0009] According to one aspect of the present disclosure, a classification device for estimating object classification based on a hyperspectral image is provided. The classification device includes: a memory having instructions stored thereon; and one or more processors, the one or more processors being capable of communicating with the memory to execute instructions retrieved from the memory, and the instructions causing the one or more processors to: extract point-level classification features of each of a plurality of points within a contour region of a single object of interest of a type in the hyperspectral image using a point-level classifier based on a first neural network; and estimate a lower-level classification of the single object of interest of a type of interest relative to the type of interest based on block-level classification features associated with counts of the point-level classification features of the plurality of points with respect to counts of each cluster feature group in a cluster feature group set using a block-level classifier based on a second neural network; wherein the point-level classifier Point-level classification features of corresponding points are extracted based on the hyperspectral values ​​of respective points; the point-level classifier is configured to estimate the classification of a single point with respect to a lower-level classification set based on the hyperspectral information of the single point within the contour area; the contour area of ​​a single object of interest type is the contour of a single object of interest type or the area indicated by the bounding box of the contour of a single object of interest type in the hyperspectral image; the cluster feature group set is determined by clustering the point-level classification features extracted by the point-level classifier from a predetermined hyperspectral image set; and the count of the point-level classification features of the plurality of points with respect to each cluster feature group in the cluster feature group set indicates the count of the point-level classification features among the point-level classification features of the plurality of points that fall into the corresponding cluster feature group.

[0010] According to another aspect of the present disclosure, there is provided a computer-readable storage medium having a program stored thereon. The program causes a computer to: extract point-level classification features for each of a plurality of points within a contour region of a single object of type of interest in a hyperspectral image using a point-level classifier based on a first neural network; and estimate a lower-level classification of the single object of type of interest relative to the type of interest using a block-level classifier based on a second neural network based on block-level classification features associated with counts of the point-level classification features of the plurality of points with respect to each cluster feature group in a set of cluster feature groups; wherein the point-level classifier extracts point-level classification features of the corresponding point based on hyperspectral values ​​of the corresponding point; the point-level classifier is configured to estimate the classification of the single point with respect to the lower-level classification set based on hyperspectral information of the single point within the contour region; the contour region of the single object of type of interest is an area indicated by a contour of the single object of type of interest or an outer bounding box of the contour of the single object of type of interest in the hyperspectral image; the set of cluster feature groups is determined by clustering the point-level classification features extracted by the point-level classifier from a predetermined set of hyperspectral images; and the counts of the point-level classification features of the plurality of points with respect to each cluster feature group in the set of cluster feature groups indicate counts of point-level classification features of the plurality of points that fall into the corresponding cluster feature group.

[0011] The beneficial effects of the classification method, classification device, and storage medium of the present disclosure include at least improving the accuracy of classification and improving the robustness of the classification scheme. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The following description of embodiments of the present disclosure will facilitate an easier understanding of the above and other objects, features, and advantages of the present disclosure with reference to the accompanying drawings. The accompanying drawings are intended only to illustrate the principles of the present disclosure. The sizes and relative positions of elements in the drawings are not necessarily drawn to scale. Identical reference numerals may denote identical features. In the accompanying drawings:

[0013] Figure 1 An exemplary flow chart of a classification method for estimating object classification according to one embodiment of the present disclosure is shown;

[0014] Figure 2 showing a hyperspectral curve for a plurality of points within a contour region of a single object of interest;

[0015] Figure 3 A schematic diagram of a first neural network according to one embodiment of the present disclosure is shown;

[0016] Figure 4 shows exemplary counts for each cluster feature group according to one embodiment of the present disclosure;

[0017] Figure 5 An exemplary flow chart of a classification method for estimating object classification according to one embodiment of the present disclosure is shown;

[0018] Figure 6 A classification device for estimating object classification based on hyperspectral images according to one embodiment of the present disclosure is shown;

[0019] Figure 7 A classification apparatus for estimating object classification based on hyperspectral images according to one embodiment of the present disclosure is shown; and

[0020] Figure 8 is an exemplary block diagram of an information processing device according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings. For the sake of clarity and conciseness, not all features of actual embodiments are described in this specification. However, it should be understood that in the process of developing any such actual embodiment, many implementation-specific decisions may be made to achieve the developer's specific goals, and these decisions may vary from one implementation to another.

[0022] It should also be noted here that, in order to avoid obscuring the content of the present disclosure due to unnecessary details, the accompanying drawings only show the device structure closely related to the solution according to the content of the present disclosure, while omitting other details that are not closely related to the content of the present disclosure.

[0023] It should be understood that the present disclosure is not limited to the described embodiments due to the following description with reference to the accompanying drawings. In this document, where feasible, the embodiments may be combined with each other, features between different embodiments may be replaced or borrowed, and one or more features may be omitted in one embodiment.

[0024] Computer program code for carrying out operations for various aspects of embodiments of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and the like, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages.

[0025] The classification method of the present disclosure can be implemented by a circuit or circuit system with corresponding functional configurations, wherein the circuit includes a circuit for a processor.

[0026] One aspect of the present disclosure provides a classification method for estimating object classification based on a hyperspectral image. The classification method can be implemented by a computer. The input of the classification method includes a hyperspectral image, in which an object of interest has its outline identified in the hyperspectral image through target detection or segmentation. The outline of a single object of interest is the outline of the single object of interest or the area indicated by the bounding box of the single object of interest in the hyperspectral image. Objects of interest include trees. For example, the input hyperspectral image contains a single tree, in which the outline of the tree is already shown, or in other words, the area occupied by the hyperspectral image tile of the tree in the image is predetermined. The method for determining the outline is not essential to the present disclosure and will not be described in detail. The output of the classification method of the present disclosure includes sub-classifications of the object of interest relative to the object of interest. Each sub-classification constitutes a sub-classification set. For example, if the object of interest is tree, the sub-classifications relative to the object of interest may include pine, sycamore, willow, etc. For example, if the object of interest is pine, the sub-classifications relative to the object of interest may include podocarpus, scotch pine, and red pine, etc. For example, to study the distribution of 63 tree species in a forest using hyperspectral imagery, these 63 tree types constitute a lower-level classification set. Generally speaking, the classification method of this disclosure is divided into two stages: in the first stage, point-level classification features are obtained; in the second stage, block-level classification features are obtained for classification.

[0027] Refer to the following Figure 1 The classification method of the present disclosure is exemplified.

[0028] Figure 1 An exemplary flow chart of a classification method 100 for estimating object classification based on hyperspectral images according to one embodiment of the present disclosure is shown.

[0029] In step S101, a point-level classifier Cpo based on the first neural network NN1 is used to extract the point-level classification features Fpo[i] of each point p[i] in the multiple points within the contour area ct of a single object of interest Ob in the hyperspectral image Imh. The multiple points here can be all the hyperspectral points within the contour area ct, or can be some randomly selected or framed hyperspectral points, preferably all the hyperspectral points. The contour area ct is preferably the area indicated by the circumscribed frame of the contour. The first neural network NN1 can be a deep neural network containing several fully connected layers to ensure feature integration capabilities. The point-level classifier Cpo can extract the point-level classification features of the corresponding point based on the hyperspectral value p[i].wl[k].value of each point, where k is the index of the band (denoted by wl) and has a value range of 1 to kmax. kmax is usually greater than 100, for example, a value of 200. For example, if the spectral range of a hyperspectral image is divided into 100 bands, k can be an integer between 1 and 100. For each point, there are 100 hyperspectral values, each corresponding to a band. The hyperspectral image contains information representing the hyperspectral values ​​of each point in the hyperspectral image for each of the multiple bands. The point-level classifier Cpo is designed to estimate the classification of a single point with respect to a set of sub-classifications based on the hyperspectral information of the single point within the contour region. In other words, the point-level classifier Cpo is designed to extract classification features for the hyperspectral point and, based on the extracted classification features, to determine the sub-classification corresponding to the hyperspectral point. Based on this functional requirement, the initial point-level classifier can be trained using a training sample set to determine the parameters of the first neural network NN1. The features output by the last fully connected layer of the first neural network NN1 can be selected as the point-level classification features. For example, the point-level classification features can be a vector with 2048 components. The last fully connected layer of the first neural network NN1 can be connected to the output layer. The output layer can use softmax logistic regression to determine the probabilities of each sub-classification. The input of softmax logistic regression is an N*1 vector, and the output is also an N*1 vector (N is the total number of elements in the lower-level classification set). The value of each element in this output vector represents the probability that the detected object belongs to each lower-level classification. The value of each element in the output vector ranges from 0 to 1 to represent the probability. Exemplarily, in order to introduce nonlinearity, nonlinear activation, such as Relu (Rectified linear unit) activation, can be performed on the output of the fully connected layer other than the last fully connected layer. For example, the current fully connected layer performs nonlinear activation such as Relu activation on the input features from the previous fully connected layer. In order to enhance the performance of the point-level classifier Cpo, various data enhancement schemes can be used in training the point-level classifier Cpo, such as the Mixup scheme. When the object of interest Ob is a tree, the contour area ct is the outline of the crown of the tree in the image or the area indicated by the bounding box of the crown outline.

[0030] In step S103, a block-level classifier Cpa based on a second neural network NN2 is used to estimate the sub-classification of a single object of type of interest Ob relative to the type of interest based on block-level classification features Fpa associated with counts c[j].count of point-level classification features of multiple points with respect to each cluster feature group c[j] in the cluster feature group set. The cluster feature group set is determined by clustering the point-level classification features extracted by the point-level classifier Cpo from the predetermined hyperspectral image set Shi. The predetermined hyperspectral image set Shi may include a training sample set and other valid data sets. The training sample set may be a training sample set for the point-level classifier Cpo and / or the block-level classifier Cpa. Other valid data sets may include a test set. Various clustering methods may be used to cluster the point-level classification features extracted by the point-level classifier Cpo from the predetermined hyperspectral image set Shi, such as a K-means clustering method. The number of cluster feature groups included in the cluster feature group set obtained after clustering is denoted as L. For such a set of cluster feature groups obtained, L>N, that is, the number of cluster feature groups is greater than the number of lower-level categories in the lower-level category set. The index j of the cluster feature group can take an integer from 1 to L. The count of the point-level classification features of multiple points with respect to each cluster feature group c[j] in the cluster feature group set indicates the count of the point-level classification features of the multiple points that fall into the corresponding cluster feature group. For example, if there are imax highlights in the contour area ct of a single object of interest Ob, imax point-level classification features Fpo[1], Fpo[2], ..., Fpo[imax] are extracted; the counts of the imax point-level classification features with respect to each cluster feature group c[1], c[2], ..., c[L] in the cluster feature group set are c[1].count, c[2].count, ..., c[L].count. When counting the count c[j].count, each time a point-level classification feature that falls into the cluster feature group c[j] is found among the imax point-level classification features, the current count is increased by 1. The minimum value of the count is zero. The block-level classification feature Fpa can be a vector with L components, each component being associated with the count of the point-level classification feature of the corresponding cluster feature group. The block-level classifier Cpa is designed to be able to estimate the lower-level classification of a single object of the type of interest relative to the type of interest based on the block-level classification feature Fpa. Those skilled in the art will understand that before using the block-level classifier Cpa for classification, it is necessary to train the block-level classifier Cpa using a sample training set to determine the appropriate neural network parameters and ensure the expected performance level. The cluster feature set here is equivalent to a feature dictionary or codebook.

[0031] Classification method 100 estimates the classification of objects within a contour region based on block-level classification features for the contour region of a single object of interest, rather than estimating the classification based on average classification features obtained by averaging point-level classification features within the contour region. Block-level classification features for objects of the same type exhibit certain regularities, which can be obscured by averaging point-level classification features within the contour region. Experiments have shown that classification method 100 achieves improved accuracy.

[0032] Refer to the following Figure 2 A preferred embodiment of the method 100 is exemplified. Experiments show that when using a point-level classifier Cpo to extract point-level classification features for each of multiple points within a contour region ct of a single object of interest Ob in a hyperspectral image Imh, it is preferable to normalize the hyperspectral values ​​of the multiple points within the contour region ct. Figure 2 shows the hyperspectral curves of multiple points within the contour area ct of a single object of interest Ob, where Figure 2 (a) shows the hyperspectral values ​​(ordinates) of multiple points in the contour area of ​​the hyperspectral image with respect to each band index (abscissa), Figure 2 (b) shows the normalized hyperspectral values ​​(ordinate) of a plurality of points within the contour region of the hyperspectral image with respect to each band index (abscissa). Figure 2 Each curve in (a) corresponds to a high spectral point in the contour area. Figure 2 As can be seen in (a), at least partly due to the scattering phenomenon, the hyperspectral curves of various points of the same object are not the same and there is a certain divergence. In order to reflect the commonality of each point belonging to the same object, the hyperspectral values ​​of multiple points in the same contour area can be normalized to suppress the divergence of the hyperspectral curve. In this embodiment, the min-max normalization method is exemplarily selected to normalize the multiple hyperspectral values ​​of multiple points in the contour area ct. Specifically, for Figure 2 For the multiple curves in (a), determine the minimum hyperspectral value Vmin (e.g. 1) and the maximum hyperspectral value Vmax (e.g. 5501) among all the hyperspectral values, and then use Vmin and Vmax to transform all the hyperspectral values ​​to the interval [0,1] in a positive correlation through linear transformation. For example, there are 100 points in the contour area, and each point has 80 hyperspectral values. Then determine the minimum hyperspectral value Vmin and the maximum hyperspectral value Vmax from a total of 8000 hyperspectral values. Figure 2 As shown in (b), after normalization, the hyperspectral curves of multiple points within the same contour area converge. The curves exhibit more obvious similarities. Normalization helps improve the classification accuracy of classification method 100. This normalization method can be viewed as a method for updating hyperspectral values.

[0033] In order to make full use of the data, some data enhancement operations can be used in the classification method 100. Exemplarily, the hyperspectral values ​​of the hyperspectral points are randomly spatially enhanced with a predetermined probability. Random spatial enhancement includes the following operations: updating the hyperspectral value of the corresponding point with the hyperspectral values ​​of the neighboring points of the corresponding point and the weighted average of the hyperspectral values ​​of the corresponding point. A random function is used to generate multiple weights for calculating the weighted average. Performing random spatial enhancement with a predetermined probability means that data enhancement operations are not performed on some points in the contour area. For example, the hyperspectral values ​​of the 8 neighboring points p[1], p[2], p[3], p[4], p[6], p[7], p[8], and p[9] around the corresponding point p[5] can be used to update the hyperspectral value of the corresponding point p[5]. These points form a 3*3 matrix, and the corresponding point p[5] is located at the center of the matrix. The calculation formula is as follows:

[0034]

[0035] A rule can be set to not perform data enhancement operations on edge points. w[i] is a plurality of weights used to calculate the weighted average (p[5].wl[k].value”); p[5].wl[k].value” is the updated hyperspectral value of point p[5], and p[5].wl[k].value' is the hyperspectral value of point p[5] before the update. The weights can be generated using a random function. For example, 9 random numbers are generated at a time as w[1] to w[9] using a random Gaussian distribution function. It can be understood that random spatial enhancement can be performed before the normalization operation or after the normalization operation. That is, extracting the point-level classification features using a point-level classifier includes: performing random spatial enhancement on the hyperspectral value of the corresponding point with a predetermined probability before or after the normalization operation to update the hyperspectral value of the corresponding point. The predetermined probability can be determined based on experience. Experiments show that it is preferred to perform random spatial enhancement on the hyperspectral value of the corresponding point with a predetermined probability before the normalization operation to update the hyperspectral value of the corresponding point.

[0036] Figure 3 FIG. 1 shows a schematic diagram of an exemplary first neural network NN1 according to an embodiment of the present disclosure. Figure 3As shown in , the first neural network NN1 includes an input layer L1, a fully connected layer L2, a fully connected layer L3, a fully connected layer L4, and an output layer L5. Exemplarily, the dimensions of the outputs of the three fully connected layers are 2048, 4096, and 2048, respectively. The input layer L1 receives a hyperspectral image. The features of the output of the fully connected layer L4 are provided to the output layer L5. After the output layer L5 transforms the input features, it performs softmax logistic regression processing and outputs an N*1 vector (N is the total number of elements in the lower-level classification set). The value of each element in this output vector represents the probability that the detected object belongs to each lower-level classification. In the classification method 100, the output of the fully connected layer L4 is used as a point-level classification feature. It is also possible to use the output features of the fully connected layer L3 as point-level classification features, but it is preferred to use the output features of the last level of the fully connected layer as point-level classification features for use by the block-level classifier.

[0037] Classification method 100 uses counts to build block-level classification features. Figure 4 Describe the count. Figure 4 , shows exemplary counts of each cluster feature group according to one embodiment of the present disclosure. Figure 4 As shown in , the cluster feature group set includes L cluster feature groups, where L is an integer. An exemplary value of L is 100. To distinguish these cluster feature groups, each feature group is assigned an index from 1 to L. Therefore, a cluster feature group can be represented by c[j], where j can be an integer from 1 to L. The extraction rules for the point-level classification features included in the cluster feature group are the same as the extraction rules for the point-level classification features of the hyperspectral points in the contour area extracted by the point-level classifier. Figure 4 The vertical axis in is the statistical count for the corresponding cluster feature group. When counting the counts of each cluster feature group, it is necessary to count the number of point-level classification features that fall into the cluster feature group. For each additional point-level classification feature that falls into the cluster feature group, the count is increased by 1. Whether a point-level classification feature falls into the cluster feature group can be judged based on distance, similarity, etc. Considering that the cluster feature group set is a set constructed for all lower-level classifications, when counting the counts of the cluster feature group set for a certain type of object of interest, the counts of some cluster feature groups may be 0. In addition, it is understandable that Figure 4 The counts in the hyperspectral image are for a particular object of interest. The sum of the counts for each clustered feature group is equal to the number of selected hyperspectral points within the contour region of the object of interest. The block-level classification features for the object of interest can be represented as an L-dimensional vector, denoted as: (c[1].count, c[2].count, ..., c[L].count). This block-level classification feature can also be called the hyperspectral image texture feature of the object of interest (abbreviated as "HSI feature").

[0038] Furthermore, in order to improve the accuracy of classification, the block-level classification features associated with the counts can be updated. In one embodiment, determining the block-level classification features includes: determining an intermediate vector composed of the point-level classification features of multiple points with respect to the counts of each cluster feature group in the cluster feature group set; and normalizing the intermediate vector as a block-level classification feature. The intermediate vector (also referred to as the block-level classification feature before updating) is, for example, (c[1].count, c[2].count, ..., c[L].count), and the normalized intermediate vector (i.e., the block-level classification feature, or the updated block-level classification feature) is (c[1].count / sum, c[2].count / sum, ..., c[L].count / sum), where sum is the sum of each count (sum = ∑c[j].count).

[0039] When the object of interest is a tree, the contour area can be the area indicated by the crown contour, and the block-level classification feature is the crown-level classification feature, which represents the HSI feature of the hyperspectral image of the crown. The distribution of hyperspectral points in the crown contour can be characterized as the frequency distribution of hyperspectral points belonging to each cluster feature group, that is, the frequency distribution of different feature points. Outliers and low-frequency points within the contour have little contribution to the block-level classification feature. The classification method of the present disclosure is highly robust to outliers and low-frequency points within the contour.

[0040] In one embodiment, the input of the second neural network NN2 is an L-dimensional block-level classification feature. The second neural network NN2 includes multiple fully connected layers (for example, two fully connected layers). The output of each fully connected layer is, for example, a vector with 2048 components. Nonlinear activation (for example, ReLU activation) is used after the previous fully connected layer (for example, the first fully connected layer) to introduce nonlinearity. The second neural network NN2 also includes an output layer, a softmax layer. The softmax layer is designed to estimate the probability of each of the N lower-level categories of the object of interest corresponding to the current contour area.

[0041] In one embodiment, the predetermined hyperspectral image set includes: detection set images, wherein the hyperspectral images to be detected included in the detection set are referred to as detection set images. For example, when the detection set and the training sample set differ greatly in terms of regionality, in order to improve regional adaptability, the following operation can be adopted: clustering the point-level classification features extracted by the point-level classifier from the predetermined hyperspectral image set including the training samples and the detection set images to determine the cluster feature group set. The predetermined hyperspectral image set may include the training samples and part of the detection set images in the detection set. For example, when the absolute value of the difference between the acquisition region of the current detection set and the image acquisition region of the training sample set is greater than a predetermined dimension threshold, at least part of the point-level classification features of the detection data set can be added to the point-level classification feature set to be clustered and clustered.

[0042] Reference below Figure 5 One embodiment of the present disclosure is described. Figure 5 An exemplary flow chart of a classification method 500 for estimating object classification according to one embodiment of the present disclosure is shown. In step S51011, the hyperspectral values ​​of multiple points extracted from the contour region of a single object of interest in a hyperspectral image are normalized. In step S51013, the normalized hyperspectral values ​​of selected points within the contour region are randomly spatially enhanced with a predetermined probability to update the hyperspectral values ​​of the points within the contour region. The selected points may be hyperspectral points with neighboring points required for a weighted average. Steps S51011 to S51013 may be performed by the input layer of a first neural network. In step S51015, point-level classification features are determined based on the updated hyperspectral values. The output features of the last fully connected layer in the fully connected layers of the first neural network may be used as the point-level classification features. In step S51031, the counts of the multiple points within the contour region with respect to each cluster feature group in a set of cluster feature groups are determined to generate an intermediate vector. In step S51033, the intermediate vector is normalized to serve as a block-level classification feature. In step S51035, the lower-level classification of the single object of the type of interest relative to the type of interest is estimated based on the block-level classification features. Steps S51031 to S51033 can be completed by the input layer of the second neural network.

[0043] The present disclosure also provides a classification device for estimating object classification based on hyperspectral images. Figure 6 An exemplary description is given. Figure 6A classification device 600 for estimating object classification based on a hyperspectral image, according to one embodiment of the present disclosure, is shown. The classification device includes a point-level classification feature extraction unit 601 and a classification unit 603. The point-level classification feature extraction unit 601 uses a point-level classifier based on a first neural network to extract point-level classification features for each of multiple points within a contour region of a single object of interest in a hyperspectral image. The classification unit 603 uses a block-level classifier based on a second neural network to estimate a sub-classification of the single object of interest relative to the type of interest based on block-level classification features associated with the counts of the point-level classification features of the multiple points with respect to each cluster feature group in a set of cluster feature groups. The point-level classifier extracts point-level classification features for each point based on the hyperspectral value of the corresponding point. The point-level classifier is configured to estimate the classification of a single point with respect to a sub-classification set based on the hyperspectral information of the single point within the contour region. The contour region of the single object of interest is the contour of the single object of interest or the area indicated by the bounding box of the contour of the single object of interest in the hyperspectral image. The cluster feature set is determined by clustering the point-level classification features extracted by the point-level classifier from a predetermined hyperspectral image set. The count of the point-level classification features of the plurality of points with respect to each cluster feature group in the cluster feature set indicates the count of the point-level classification features of the plurality of points that fall into the corresponding cluster feature group. This classification device corresponds to the classification method of the present disclosure. For further configuration details of the classification device, reference can be made to the description of the classification method of the present disclosure.

[0044] The present disclosure also provides a classification device for estimating object classification based on hyperspectral images. Figure 7 This image processing apparatus is described. Figure 7A classification device 700 for estimating object classification according to an embodiment of the present disclosure is shown. The classification device 700 includes: a memory 701 on which instructions are stored; and one or more processors 703. The one or more processors are capable of communicating with the memory to execute instructions retrieved from the memory, and the instructions cause the one or more processors to: extract point-level classification features of each of a plurality of points within a contour area of ​​a single object of the type of interest in a hyperspectral image using a point-level classifier based on a first neural network; and estimate a lower-level classification of the single object of the type of interest relative to the type of interest based on block-level classification features associated with counts of the point-level classification features of the plurality of points with respect to counts of each cluster feature group in a set of cluster feature groups using a block-level classifier based on a second neural network; wherein the point-level classifier extracts point-level features of the corresponding point based on the hyperspectral value of each point. Level classification features; the point-level classifier is configured to estimate the classification of a single point with respect to a lower-level classification set based on the hyperspectral information of a single point within the contour area; the contour area of ​​a single object of interest is the contour of a single object of interest or the area indicated by the outer bounding box of the contour of a single object of interest in the hyperspectral image; the cluster feature group set is determined by clustering the point-level classification features extracted by the point-level classifier from a predetermined hyperspectral image set; and the count of the point-level classification features of the plurality of points with respect to each cluster feature group in the cluster feature group set indicates the count of the point-level classification features of the plurality of points that fall into the corresponding cluster feature group. There is a corresponding relationship between the classification device and the classification method of the present disclosure. For further configuration of the classification device, reference can be made to the description of the classification method of the present disclosure.

[0045] One aspect of the present disclosure provides a computer-readable storage medium having a program stored thereon. The program causes a computer to: extract point-level classification features for each of a plurality of points within a contour region of a single object of type of interest in a hyperspectral image using a point-level classifier based on a first neural network; and estimate a lower-level classification of the single object of type of interest relative to the type of interest using a block-level classifier based on a second neural network based on block-level classification features associated with counts of the point-level classification features of the plurality of points with respect to each cluster feature group in a set of cluster feature groups; wherein the point-level classifier extracts point-level classification features of the corresponding point based on hyperspectral values ​​of the corresponding point; the point-level classifier is configured to estimate the classification of the single point with respect to the lower-level classification set based on hyperspectral information of the single point within the contour region; the contour region of the single object of type of interest is an area indicated by a contour of the single object of type of interest or an outer bounding box of the contour of the single object of type of interest in the hyperspectral image; the set of cluster feature groups is determined by clustering the point-level classification features extracted by the point-level classifier from a predetermined set of hyperspectral images; and the counts of the point-level classification features of the plurality of points with respect to each cluster feature group in the set of cluster feature groups indicate counts of point-level classification features of the plurality of points that fall into the corresponding cluster feature group. There is a corresponding relationship between the functions of the program and the classification method of the present disclosure. For further configuration of the program, please refer to the description of the classification method in the present disclosure.

[0046] According to one aspect of the present disclosure, an information processing device is also provided.

[0047] Figure 8 is an exemplary block diagram of an information processing device 800 according to one embodiment of the present disclosure. Figure 8 In the embodiment of the present invention, a central processing unit (CPU) 801 performs various processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage unit 808 to a random access memory (RAM) 803. The RAM 803 also stores data and the like required when the CPU 801 performs various processes as needed.

[0048] The CPU 801, the ROM 802, and the RAM 803 are connected to one another via a bus 804. An input / output interface 805 is also connected to the bus 804.

[0049] The following components are connected to the input / output interface 805: an input section 806 including a soft keyboard, etc.; an output section 807 including a display such as a liquid crystal display (LCD), a speaker, etc.; a storage section 808 such as a hard disk; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet, a local area network, a mobile network, or a combination thereof.

[0050] A drive 810 is also connected to the input / output interface 805 as needed. A removable medium 811 such as a semiconductor memory or the like is mounted on the drive 810 as needed, so that a program read therefrom is installed to the storage section 808 as needed.

[0051] The CPU 801 may run a program for implementing the method for estimating object classification based on hyperspectral images according to the present disclosure, and the program may implement the functions of the classification method 100 .

[0052] The solution of the present disclosure uses a two-stage classifier to achieve final classification when estimating object classification based on hyperspectral images, which helps to improve the accuracy of classification and the robustness of the classification solution.

[0053] As described above, according to the present disclosure, the principles of a classification method and a classification final device for estimating object classification are provided. It should be noted that the effects of the scheme of the present disclosure are not necessarily limited to the effects described above, and in addition to or instead of the effects described in the previous paragraphs, any of the effects shown in this specification or other effects that can be understood from this specification can be achieved.

[0054] Although the present invention has been disclosed above through the description of specific embodiments of the present invention, it should be understood that those skilled in the art can design various modifications (including, in the case of rows, combinations or replacements of features between embodiments), improvements or equivalents to the present invention within the spirit and scope of the appended claims. These modifications, improvements or equivalents should also be considered to be included in the scope of protection of the present invention. For example, when a hyperspectral image has been digitized, the position coordinate data of each hyperspectral point and the spectral value data of each band of each hyperspectral point are known, then the point-level classifier can be configured to extract point-level classification features of each of the multiple points within the contour area of ​​a single object of interest in the hyperspectral image based on these data, without having to analyze the hyperspectral image to obtain these data.

[0055] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0056] Furthermore, the methods of the various embodiments of the present invention are not limited to being executed in the chronological order described in the specification or shown in the accompanying drawings, and may also be executed in other chronological orders, in parallel, or independently. Therefore, the execution order of the methods described in this specification does not limit the technical scope of the present invention.

[0057] Notes

[0058] 1. A computer-implemented classification method for estimating object classification based on hyperspectral imagery, comprising:

[0059] extracting point-level classification features of each of a plurality of points within a contour region of a single object of interest in the hyperspectral image using a point-level classifier based on a first neural network; and

[0060] estimating, using a second neural network-based block-level classifier, a lower-level classification of the single object of the type of interest relative to the type of interest based on block-level classification features associated with counts of the point-level classification features of the plurality of points with respect to each cluster feature group in the set of cluster feature groups;

[0061] The point-level classifier extracts the point-level classification features of the corresponding points based on the hyperspectral values ​​of each point;

[0062] The point-level classifier is configured to estimate the classification of the single point with respect to the lower-level classification set based on the hyperspectral information of the single point within the contour area;

[0063] The outline area of ​​the single object of interest is the outline of the single object of interest or the area indicated by the outer bounding box of the outline of the single object of interest in the hyperspectral image;

[0064] The cluster feature set is determined by clustering the point-level classification features extracted by the point-level classifier from a predetermined hyperspectral image set; and

[0065] The counts of the point-level classification features of the plurality of points with respect to each cluster feature group in the cluster feature group set indicate the counts of the point-level classification features of the plurality of points that fall into the corresponding cluster feature group.

[0066] 2. The classification method according to Note 1, wherein the type of interest includes trees, and the lower-level classification set includes multiple tree types.

[0067] 3. The classification method according to Supplementary Note 1, wherein the hyperspectral image includes information representing a hyperspectral value of each of a plurality of bands for each point within the hyperspectral image.

[0068] 4. The classification method according to Note 1, wherein extracting the point-level classification features using the point-level classifier includes normalizing the hyperspectral values ​​of the points within the contour area.

[0069] 5. The classification method according to Supplementary Note 1, wherein extracting the point-level classification features using the point-level classifier comprises: performing random spatial enhancement on the hyperspectral values ​​of the points within the contour area with a predetermined probability;

[0070] The random spatial enhancement includes the following operations: updating the hyperspectral value of the corresponding point by using the hyperspectral values ​​of the neighboring points of the corresponding point and the weighted average of the hyperspectral value of the corresponding point;

[0071] A random function is used to generate a plurality of weights for calculating the weighted average.

[0072] 6. The classification method according to Note 5, wherein the random spatial enhancement includes the following operation: determining multiple weights for calculating the weighted average based on a random Gaussian distribution.

[0073] 7. The classification method according to Supplementary Note 6, wherein the hyperspectral value of the corresponding point is updated using a weighted average of the hyperspectral values ​​of eight neighboring points around the corresponding point and the hyperspectral value of the corresponding point.

[0074] 8. The classification method according to Note 1, wherein the first neural network includes multiple fully connected layers.

[0075] 9. The classification method according to Note 8, wherein the first neural network is configured to perform nonlinear activation on the output features of some of the multiple fully connected layers.

[0076] 10. The classification method according to Note 9, wherein the nonlinear activation includes Relu activation.

[0077] 11. The classification method according to Note 1, wherein the cluster feature group set includes L cluster feature groups, the lower-level classification set includes N lower-level classifications, and L>N.

[0078] 12. The classification method according to Note 1, wherein the block-level classification features are determined by the block-level classifier based on the counts of the point-level classification features of the plurality of points with respect to each cluster feature group in the cluster feature group set.

[0079] 13. The classification method according to Supplementary Note 1, wherein determining the block-level classification features comprises:

[0080] Determining an intermediate vector consisting of counts of the point-level classification features of the plurality of points with respect to each cluster feature group in the cluster feature group set; and

[0081] The intermediate vector is normalized as the block-level classification feature.

[0082] 14. The classification method according to Note 1, wherein the second neural network includes multiple fully connected layers.

[0083] 15. The classification method according to Note 14, wherein the second neural network is configured to perform nonlinear activation on the output features of the previous fully connected layer among the multiple fully connected layers.

[0084] 16. The classification method according to Note 15, wherein the nonlinear activation comprises ReLU activation.

[0085] 17. A classification device for estimating object classification based on hyperspectral imagery, comprising:

[0086] a memory having instructions stored thereon; and

[0087] one or more processors capable of communicating with the memory to execute the instructions retrieved from the memory, and the instructions causing the one or more processors to:

[0088] extracting point-level classification features of each of a plurality of points within a contour region of a single object of interest in the hyperspectral image using a point-level classifier based on a first neural network; and

[0089] estimating, using a second neural network-based block-level classifier, a lower-level classification of the single object of the type of interest relative to the type of interest based on block-level classification features associated with counts of the point-level classification features of the plurality of points with respect to each cluster feature group in the set of cluster feature groups;

[0090] The point-level classifier extracts the point-level classification features of the corresponding points based on the hyperspectral values ​​of each point;

[0091] The point-level classifier is configured to estimate the classification of the single point with respect to the lower-level classification set based on the hyperspectral information of the single point within the contour area;

[0092] The outline area of ​​the single object of interest is the outline of the single object of interest or the area indicated by the outer bounding box of the outline of the single object of interest in the hyperspectral image;

[0093] The cluster feature set is determined by clustering the point-level classification features extracted by the point-level classifier from a predetermined hyperspectral image set; and

[0094] The counts of the point-level classification features of the plurality of points with respect to each cluster feature group in the cluster feature group set indicate the counts of the point-level classification features of the plurality of points that fall into the corresponding cluster feature group.

[0095] 18. A computer-readable storage medium having a program stored thereon, wherein the program causes a computer to:

[0096] extracting point-level classification features of each of a plurality of points within a contour region of a single object of interest in the hyperspectral image using a point-level classifier based on a first neural network; and

[0097] estimating, using a second neural network-based block-level classifier, a lower-level classification of the single object of the type of interest relative to the type of interest based on block-level classification features associated with counts of the point-level classification features of the plurality of points with respect to each cluster feature group in the set of cluster feature groups;

[0098] The point-level classifier extracts the point-level classification features of the corresponding points based on the hyperspectral values ​​of each point;

[0099] The point-level classifier is configured to estimate the classification of the single point with respect to the lower-level classification set based on the hyperspectral information of the single point within the contour area;

[0100] The outline area of ​​the single object of interest is the outline of the single object of interest or the area indicated by the outer bounding box of the outline of the single object of interest in the hyperspectral image;

[0101] The cluster feature set is determined by clustering the point-level classification features extracted by the point-level classifier from a predetermined hyperspectral image set; and

[0102] The counts of the point-level classification features of the plurality of points with respect to each cluster feature group in the cluster feature group set indicate the counts of the point-level classification features of the plurality of points that fall into the corresponding cluster feature group.

Claims

1. A computer-implemented classification method for estimating object classification based on hyperspectral images, characterized in that: include: extracting point-level classification features of each of a plurality of points within a contour region of a single object of interest in the hyperspectral image using a point-level classifier based on a first neural network; as well as estimating, using a second neural network-based block-level classifier, a lower-level classification of the single object of the type of interest relative to the type of interest based on block-level classification features associated with counts of the point-level classification features of the plurality of points with respect to each cluster feature group in the set of cluster feature groups; The point-level classifier extracts the point-level classification features of the corresponding points based on the hyperspectral values ​​of each point; The point-level classifier is configured to estimate the classification of the single point with respect to the lower-level classification set based on the hyperspectral information of the single point within the contour area; The outline area of ​​the single object of interest is the outline of the single object of interest or the area indicated by the outer bounding box of the outline of the single object of interest in the hyperspectral image; The cluster feature set is determined by clustering the point-level classification features extracted by the point-level classifier from a predetermined hyperspectral image set; and The counts of the point-level classification features of the plurality of points with respect to each cluster feature group in the cluster feature group set indicate the counts of the point-level classification features of the plurality of points that fall into the corresponding cluster feature group.

2. The classification method according to claim 1, wherein: The types of interest include trees, and the lower-level classification set includes a plurality of tree types.

3. The classification method according to claim 1, wherein: Extracting the point-level classification features using the point-level classifier includes normalizing the hyperspectral values ​​of the points within the contour area.

4. The classification method according to claim 1, wherein: Extracting the point-level classification features using the point-level classifier includes: performing random spatial enhancement on the hyperspectral values ​​of the points within the contour area with a predetermined probability; The random spatial enhancement includes the following operations: updating the hyperspectral value of the corresponding point by using the hyperspectral values ​​of the neighboring points of the corresponding point and the weighted average of the hyperspectral value of the corresponding point; A random function is used to generate a plurality of weights for calculating the weighted average.

5. The classification method according to claim 1, wherein: The first neural network includes a plurality of fully connected layers.

6. The classification method according to claim 5, wherein: The first neural network is configured to perform nonlinear activation on output features of some of the multiple fully connected layers.

7. The classification method according to claim 1, wherein: Determining the block-level classification features includes: Determining an intermediate vector consisting of counts of the point-level classification features of the plurality of points with respect to each cluster feature group in the cluster feature group set; and The intermediate vector is normalized as the block-level classification feature.

8. The classification method according to claim 1, wherein: Estimating a lower-level classification of the single object of the type of interest relative to the type of interest using the block-level classifier includes updating the set of cluster features.

9. A classification device for estimating object classification based on hyperspectral images, characterized in that: include: a memory having instructions stored therein; as well as one or more processors capable of communicating with the memory to execute the instructions retrieved from the memory, and the instructions causing the one or more processors to: extracting point-level classification features of each of a plurality of points within a contour region of a single object of interest in the hyperspectral image using a point-level classifier based on a first neural network; as well as estimating, using a second neural network-based block-level classifier, a lower-level classification of the single object of the type of interest relative to the type of interest based on block-level classification features associated with counts of the point-level classification features of the plurality of points with respect to each cluster feature group in the set of cluster feature groups; The point-level classifier extracts the point-level classification features of the corresponding points based on the hyperspectral values ​​of each point; The point-level classifier is configured to estimate the classification of the single point with respect to the lower-level classification set based on the hyperspectral information of the single point within the contour area; The outline area of ​​the single object of interest is the outline of the single object of interest or the area indicated by the outer bounding box of the outline of the single object of interest in the hyperspectral image; The cluster feature set is determined by clustering the point-level classification features extracted by the point-level classifier from a predetermined hyperspectral image set; and The counts of the point-level classification features of the plurality of points with respect to each cluster feature group in the cluster feature group set indicate the counts of the point-level classification features of the plurality of points that fall into the corresponding cluster feature group.

10. A computer-readable storage medium having a program stored thereon, characterized in that The program causes the computer to: extracting point-level classification features of each of a plurality of points within a contour region of a single object of interest in the hyperspectral image using a point-level classifier based on a first neural network; as well as estimating, using a second neural network-based block-level classifier, a lower-level classification of the single object of the type of interest relative to the type of interest based on block-level classification features associated with counts of the point-level classification features of the plurality of points with respect to each cluster feature group in the set of cluster feature groups; The point-level classifier extracts the point-level classification features of the corresponding points based on the hyperspectral values ​​of each point; The point-level classifier is configured to estimate the classification of the single point with respect to the lower-level classification set based on the hyperspectral information of the single point within the contour area; The outline area of ​​the single object of interest is the outline of the single object of interest or the area indicated by the outer bounding box of the outline of the single object of interest in the hyperspectral image; The cluster feature set is determined by clustering the point-level classification features extracted by the point-level classifier from a predetermined hyperspectral image set; and The counts of the point-level classification features of the plurality of points with respect to each cluster feature group in the cluster feature group set indicate the counts of the point-level classification features of the plurality of points that fall into the corresponding cluster feature group.

Images