Spectral image classification method and device, storage medium, and electronic device

Abstract

The present application discloses a method and device for classifying spectral images, a storage medium, and an electronic device. The method includes: obtaining a first spectral image, wherein the first spectral image is an image collected by a spectral sensor; using a dual-branch coupling network to perform dimensionality reduction processing on the first spectral image to obtain a second spectral image, wherein the feature dimension of the second spectral image is lower than the feature dimension of the first spectral image; using a convolutional neural network to extract the spatial features of the second spectral image, and obtaining the classification result of the first spectral image through the spatial features. The present application solves the technical problem of low classification accuracy in related technologies.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular, to a spectral image classification method and device, a storage medium, and an electronic device. Background Art

[0002] Hyperspectral images are remote sensing images obtained using technologies such as hyperspectral remote sensing and hyperspectral imaging spectrometers. Figure 1 The basic process of hyperspectral remote sensing imaging is demonstrated. The satellite-borne spectral imager will emit spectral signals of different bands to the surface while flying along the orbit. Different types of ground objects have different absorption and reflectivity of the spectral signals of each band. The satellite-borne spectral imager can draw a three-dimensional hyperspectral image based on the feedback of the ground objects to the spectral signals. Each pixel in the hyperspectral image can be regarded as the performance of the same type of ground objects under different spectral band signals. Hyperspectral images often contain hundreds of spectral bands, so a continuous spectral curve can be drawn for each pixel, such as Figure 1 Soil, water, vegetation, etc.

[0003] Hyperspectral image classification is an important application in the field of hyperspectral. Each pixel in a hyperspectral image corresponds to a ground area of ​​a certain size. For example, the spatial resolution of the WorldView-2 satellite panchromatic image is 0.5m, and the actual ground size corresponding to each pixel in the obtained image is 0.5m*0.5m. Hyperspectral image classification refers to the analysis of the spectral and spatial characteristics of each pixel in the image to obtain the ground feature category to which it belongs. Hyperspectral image classification technology has been widely used in daily life. In agricultural production, hyperspectral image classification technology can be used to realize the growth analysis and quality control of agricultural products; research on urban hyperspectral image classification can realize urban environmental monitoring; in environmental monitoring and other fields, hyperspectral image classification can be used to detect atmospheric changes; in addition, it has a wide range of applications in many important fields such as national defense construction, surveying and mapping, and military, so the research on hyperspectral image classification is of great significance.

[0004] In recent years, the application of deep learning to hyperspectral image classification has become a major research trend. In the process of using deep learning to solve problems, in order to obtain ideal classification results, it is usually necessary to comprehensively utilize the spatial information and spectral information of hyperspectral data. However, too high-dimensional input data will cause the neural network to have a large number of parameters to learn, which increases the complexity of learning. At the same time, the limited number of training samples will also bring about the problem of dimensionality disaster and overfitting. Moreover, a large amount of band information in hyperspectral data is not always conducive to classification. There will always be some redundant bands. These redundant band information will bring about the problem of "same object, different spectrum, different objects, same spectrum", resulting in unsatisfactory classification results. Therefore, it is very critical to effectively extract features and reduce the dimensionality of the original hyperspectral data.

[0005] There are many traditional dimensionality reduction algorithms, such as unsupervised dimensionality reduction algorithms such as principal component analysis (PCA), local linear embedding (LLE), etc., and supervised dimensionality reduction algorithms such as linear discriminant analysis (LDA). The PCA method uses a transformation matrix composed of eigenvectors corresponding to larger eigenvalues ​​to achieve the transformation of high-dimensional data to low-dimensional data. It is a linear dimensionality reduction method. This unsupervised linear dimensionality reduction method is difficult to effectively solve the problem of "same object, different spectrum, different objects, same spectrum". The LLE method is an unsupervised nonlinear dimensionality reduction method. The manifold learned by this method can only be non-closed, and the sample set must be dense and uniform. It is sensitive to the selection of the number of nearest neighbor samples. Different nearest neighbor numbers have a great influence on the final dimensionality reduction result, which brings certain limitations to the application of the LLE method to hyperspectral data dimensionality reduction; in addition, in the process of feature extraction, this method ignores the category information, and it is difficult to obtain a discriminative dimensionality reduction output. Compared with PCA and LLE methods, LDA is a supervised dimensionality reduction method. It finds a hyperplane and makes samples of the same type projected onto this hyperplane close to each other, and samples of different types move away from each other, thereby increasing the difference between classes and reducing the difference within classes. LDA uses the class label information of the training data to obtain the discriminative features of the samples, so it will have a better dimensionality reduction effect. However, the LDA method requires that the number of dimensions reduced cannot exceed the total number of categories, which brings some limitations to its application. In addition to the above traditional dimensionality reduction algorithms, there are some improved dimensionality reduction algorithms that use the improved local discriminant embedding method to extract low-dimensional spectral features from the original hyperspectral data. By introducing the transformation of the covariance matrix, the spectral features of the same type of objects are close to each other in the feature space, and the spectral features of different types of objects are separated from each other, so that the discriminative features that can be used for classification can be obtained. However, the selection of the number of training samples in the algorithm depends on empirical parameters and there is no certain criterion, which will affect the obtained projection matrix.

[0006] In the process of hyperspectral image classification, the problem of "same object with different spectra, different objects with same spectrum" caused by high spectral resolution will lead to the misclassification of pixels in the image, which is not conducive to the subsequent analysis of the hyperspectral image.

[0007] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0008] The embodiments of the present application provide a spectral image classification method and device, a storage medium, and an electronic device to at least solve the technical problem of low classification accuracy in the related art.

[0009] According to one aspect of an embodiment of the present application, a method for classifying spectral images is provided, including: acquiring a first spectral image, wherein the first spectral image is an image collected by a spectral sensor; performing dimensionality reduction processing on the first spectral image using a dual-branch coupling network to obtain a second spectral image, wherein a feature dimension of the second spectral image is lower than a feature dimension of the first spectral image; extracting spatial features of the second spectral image using a convolutional neural network, and obtaining a classification result of the first spectral image through the spatial features.

[0010] According to another aspect of an embodiment of the present application, a spectral image classification device is also provided, including: an acquisition unit, used to acquire a first spectral image, wherein the first spectral image is an image collected by a spectral sensor; a processing unit, used to use a double-branch coupling network to perform dimensionality reduction processing on the first spectral image to obtain a second spectral image, wherein the characteristic dimension of the second spectral image is lower than the characteristic dimension of the first spectral image; a classification unit, used to use a convolutional neural network to extract spatial features of the second spectral image, and obtain a classification result of the first spectral image through the spatial features.

[0011] According to another aspect of an embodiment of the present application, a storage medium is further provided, which includes a stored program, and the above method is executed when the program is run.

[0012] According to another aspect of an embodiment of the present application, an electronic device is further provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above method through the computer program.

[0013] In an embodiment of the present application, a first spectral image is obtained, and the first spectral image is an image collected by a spectral sensor; a dual-branch coupling network is used to perform dimensionality reduction processing on the first spectral image to obtain a second spectral image, and the feature dimension of the second spectral image is lower than the feature dimension of the first spectral image; a convolutional neural network is used to extract spatial features of the second spectral image, and a classification result of the first spectral image is obtained through the spatial features. This solution maps the class label information from a sparse vector space to a dimensionality reduction space to obtain a trainable cluster center word vector, and uses the word vector to achieve the purpose of increasing inter-class discrimination and reducing intra-class differences, thereby solving the technical problem of low classification accuracy in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0015] Figure 1 is a schematic diagram of a hyperspectral imaging process according to an embodiment of the present application;

[0016] Figure 2 is a flow chart of an optional spectral image classification method according to an embodiment of the present application;

[0017] Figure 3 is a schematic diagram of an optional model training according to an embodiment of the present application;

[0018] Figure 4 is a schematic diagram of an optional image classification according to an embodiment of the present application;

[0019] Figure 5 is a schematic diagram of an optional image classification model according to an embodiment of the present application;

[0020] Figure 6 is a schematic diagram of an optional word embedding network according to an embodiment of the present application;

[0021] Figure 7 is a schematic diagram of an optional spectral image classification device according to an embodiment of the present application; and

[0022] Figure 8 It is a structural block diagram of a terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] In the process of hyperspectral image classification, the problem of "same object, different spectrum, different objects, same spectrum" caused by high spectral resolution will lead to the misclassification of pixels in the image, which is not conducive to the subsequent analysis of the hyperspectral image. In order to solve this problem, the present invention proposes a hyperspectral image dimensionality reduction method based on a dual-branch coupling network, which uses the proposed dual-branch coupling network structure to reduce the dimension of the original hyperspectral data, introduces clustering characteristics as the dimensionality reduction output by introducing cluster center word vectors, increases the difference between classes while reducing the difference within classes, and thus solves the problem of "same object, different spectrum, different objects, same spectrum", which is conducive to obtaining more ideal classification results in the future.

[0026] According to one aspect of the embodiments of the present application, a method embodiment of a spectral image classification method is provided. Figure 2 is a flow chart of an optional spectral image classification method according to an embodiment of the present application, such as Figure 2 As shown, the method may include the following steps:

[0027] Step S202: Acquire a first spectral image, which is an image collected by a spectral sensor and is an original hyperspectral image.

[0028] Step S204: Use a dual-branch coupling network to perform dimensionality reduction processing on the first spectral image to obtain a second spectral image, wherein the characteristic dimension of the second spectral image is lower than the characteristic dimension of the first spectral image.

[0029] The above-mentioned dual-branch coupling network includes a first coupling network and a second coupling network. When the dual-branch coupling network is used to perform dimensionality reduction processing on the first spectral image, the category features of the first spectral image are mapped from the first word vector space to the second word vector space through the first coupling network, the spatial dimension of the second word vector space is lower than the spatial dimension of the first word vector, and the category features are used to determine the classification result; and, the first spectral image is convolved through the second coupling network.

[0030] Step S206, using a convolutional neural network to extract the spatial features of the second spectral image, and obtaining the classification results of the first spectral image through the spatial features, that is, the classification results of different types of land objects, such as plants, mountains, rivers, etc.

[0031] Through the above steps S202 to S206, a first spectral image is obtained, and the first spectral image is an image collected by a spectral sensor; a dual-branch coupling network is used to perform dimensionality reduction processing on the first spectral image to obtain a second spectral image, and the feature dimension of the second spectral image is lower than the feature dimension of the first spectral image; a convolutional neural network is used to extract the spatial features of the second spectral image, and the classification result of the first spectral image is obtained through the spatial features. This scheme maps the class label information from a sparse vector space to a dimensionality reduction space to obtain a trainable cluster center word vector, and uses the word vector to achieve the purpose of increasing inter-class discrimination and reducing intra-class differences, thereby solving the technical problem of low classification accuracy in related technologies.

[0032] In this scheme, a double branch coupling network (DBCN) is designed to achieve dimensionality reduction of hyperspectral images. The DBCN network consists of a one-dimensional convolutional coupling network branch and a Word2Vec coupling network branch. In order to obtain discriminative dimensionality reduction output, a parallel word embedding network is introduced into the one-dimensional convolutional neural network to map the class label information of the training samples from the sparse vector space to the dimensionality reduction space, obtain a trainable cluster center word vector, and use the word vector to achieve the purpose of increasing inter-class discriminability and reducing intra-class differences, so as to solve the problem of "same object, different spectrum, different object, same spectrum".

[0033] Optionally, the first coupled network includes a first word embedding network and a second word embedding network, wherein, before using the dual-branch coupled network to perform dimensionality reduction processing on the first spectral image to obtain the second spectral image, the method further includes training the first coupled network in the following manner:

[0034] Determine a vector matrix between a first sample type and a second sample type of a training sample through a first word embedding network, specifically determine a difference vector between a cluster center vector of the first sample type in a first vector space and a cluster center vector of the second sample type in the first vector space through the first word embedding network, use the difference vectors between the first sample type and each second sample type as vector elements in a vector matrix, the first sample type is a currently used sample type, the second sample type is a sample type used before the first sample type, and the vector elements in the vector matrix are difference vectors between the cluster center vector of the first sample type in the first vector space and the cluster center vector of the second sample type in the first vector space;

[0035] The cluster center vector of the first sample type in the second vector space is determined through the second word embedding network, and specifically, the one-hot encoding of the training sample belonging to the first sample type can be obtained; the one-hot encoding of the training sample is input into the second word embedding network, and the cluster center vector of the first sample type in the second vector space is determined through the second word embedding network, wherein the spatial dimension of the second vector space is lower than the spatial dimension of the first vector space, and the parameters in the second word embedding network are the same as the parameters in the first word embedding network.

[0036] For the designed DBCN network, an iteratively alternative training method based on greedy strategy (IAGS) is proposed. For the dual-branch network structure of DBCN, this method is used to alternately train the network branches in DBCN, and finally achieve the convergence of the network parameters of the entire DBCN to its global optimal value by iteratively obtaining the local optimal solution of the network parameters.

[0037] The above-mentioned second coupled network includes a first convolutional network and a second convolutional network, wherein, before using the dual-branch coupled network to perform dimensionality reduction processing on the first spectral image to obtain the second spectral image, the method also includes training the second coupled network in the following manner: extracting spectral features in training samples through the first convolutional network; performing supervised classification using cross entropy loss through the second convolutional network, and outputting the reduced-dimensional spectral vector.

[0038] Based on the proposed DBCN network and the proposed iterative alternating network training method, a corresponding loss function is designed. The loss function contains intra-class loss and inter-class loss, which can be used to reduce the intra-class difference of the dimensionality reduction output and increase the inter-class discrimination of the dimensionality reduction output, respectively.

[0039] In the process of training the first coupled network and the second coupled network, the first coupled network and the second coupled network are trained in an alternating training manner, wherein the loss functions used in the training include an intra-class loss function and an inter-class loss function, the intra-class loss function is used for the intra-class difference of the dimensionality reduction output, and the inter-class loss function is used for increasing the inter-class difference of the dimensionality reduction output.

[0040] As an optional embodiment, the technical solution of the present application is further described in detail below in conjunction with specific implementation methods.

[0041] The process of the present invention is as follows Figure 3 and Figure 4 As shown in the figure, the hyperspectral image data is first divided into training samples and test samples, and then the DBCN network is trained with the training samples. After the training, the original spectral data is input into the trained DBCN network for dimensionality reduction to obtain the hyperspectral image after dimensionality reduction.

[0042] The network structure of DBCN, the hyperspectral image classification network based on DBCN dimension reduction is as follows Figure 5 As shown in Figure 1, DBCN includes two coupled network branches, namely the Word2Vec coupled network branch and the 1D-CNN coupled network branch. Figure 5 The output of the network in the middle dashed part is the dimension reduction output.

[0043] The Word2Vec coupled network branch contains two word embedding networks, Word2Vec1 and Word2Vec2, whose weight matrix can be regarded as a "query dictionary" composed of cluster center vectors of all classes. The cluster center vector of each class is the cluster center of the class sample in the reduced dimensionality space. The input of the network is the class label C of the training sample. Word2Vec1 outputs a matrix composed of the difference vectors of the cluster center vector of the Cth class and the cluster center vectors of other classes in the feature space; Word2Vec2 outputs the cluster center vector corresponding to the Cth class sample in the reduced dimensionality space. The two networks always maintain the same network parameters during the training process, which is used to map the class label information of the training samples from the sparse vector space to the reduced dimensionality space.

[0044] The 1D-CNN coupled network branch is used to extract one-dimensional spectral features. Its input is the spectral vector sample of the Cth category. The two network branches have different functions. Among them, 1D-CNN2 uses cross entropy loss for supervised classification. The output of 1D-CNN2 is the spectral vector after dimension reduction. The subsequent training process will introduce clustering characteristics for the dimension reduction output.

[0045] The specific parameters of the DBCN network structure are shown in Table 1 below, where B represents the number of spectral bands contained in the input data, and d represents the number of dimensionality reduction.

[0046] Table 1 Network structure parameters of DBCN

[0047]

[0048]

[0049] The mapping from the class label space to the reduced dimension space, the characteristic of the DBCN network is that it can map the input class label information to the reduced dimension space during dimensionality reduction. Assuming that the input DBCN is a training sample of the Cth class, the Word2Vec1 network branch obtains the cluster center vector of the Cth class and the cluster center vectors of all remaining classes Difference vector in feature space The output matrix o1 is formed. To this end, the class label C is transformed to obtain the input X. For a picture containing N C Hyperspectral images of ground object categories, X is a size (N C -1)*N C The transformation operation is: for the input training sample class label C, the Cth column of X is all -1, and the value at position (i, j) is 1 (the value of matrix row index number i is 1, 2, ... N C -1, the corresponding values ​​of column index number j are 1, 2…C-1, C+1, N C ), and the other positions are 0. Each row of X input into Word2Vec1 will calculate the difference vector of the cluster center vectors belonging to the two categories in the dimensionality reduction space.

[0050]

[0051] The calculation process of Word2Vec1 output is as follows Figure 6 , the input is the training sample of the second category, that is, the value of C is 2. The size of the input vector is 1*N C , N C -1 input vectors correspond to the row vectors of matrix X. The Cth element of the vector is -1, the elements at other positions are 1, and the values ​​of the remaining elements are 0. The output difference vector is recorded as The k-th column vector corresponding to o1. The calculation of the tth element in is shown below.

[0052] o1=ReLU(X*W wv )

[0053]

[0054] Among them, W wv is the weight matrix of the word embedding network, ReLU is the activation function acting on each element of o1, is a vector The tth element in .

[0055] The function of the network Word2Vec2 is to output the cluster center vector of the C-th class sample. Its input is the one-hot code x of the class label C, and the size of the vector x is N C , the Cth element is 1, and the others are 0. According to the characteristics of word embedding, as shown in the following formula, the cluster center vector of the Cth sample can be "queried"

[0056]

[0057] The loss function of DBCN,the loss function L of DBCN is as follows,L c is the cross entropy loss, L wv represents the clustering loss, λ is L c With L wv Ratio:

[0058] L=L c +λL wv .

[0059] The above article introduces how the DBCN network realizes the mapping from the class label space to the reduced dimension space. Its outputs are: the output of Word2Vec1 is the output matrix o1 composed of difference vectors, and Word2Vec2 outputs the cluster center vector of the C-th class sample in the reduced dimension space. The output of 1D-CNN1 corresponds to the original spectral vector x S The mapping to the reduced dimensional space is denoted by x DR , the output of 1D-CNN2 is the predicted class probability vector

[0060] L c The calculation is as follows, where N C is the number of neurons in the classification layer after the fc2 layer, and is also the total number of ground object categories contained in a hyperspectral image. y is the one-hot code corresponding to the true class label of the training sample. Represents the output of the classification layer neuron after the softmax activation function.

[0061]

[0062] Where z is the number of training samples in a batch, and Corresponding to vector y i and The kth element of .

[0063] Dimensionality reduction based on DBCN can introduce clustering characteristics to the dimension reduction output. In order to achieve the purpose of reducing intra-class differences, the loss function of DBCN needs to include intra-class loss, which can be expressed as the Euclidean distance between the dimension reduction output and its corresponding cluster center vector in the feature space. Figure 4 As shown, o2 is the cluster center And the dimension reduction output x DR The difference vector between them, whose Euclidean norm is the required Euclidean distance, is recorded as the intra-class loss L w :

[0064]

[0065] in, is the dimension reduction output of the i-th spectral vector sample, C i Corresponding to the class label of the i-th training sample.

[0066] In order to increase the discriminability between classes, the inter-class loss also needs to be calculated in the loss function of DBCN. The matrix o1 is composed of the difference vectors between the cluster centers, and its Frobenius norm (F norm) can be used to represent the intra-class loss, as shown in the following formula:

[0067]

[0068] In order to prevent the distance between classes from increasing indefinitely during the training process, a constraint of size C is added, as shown in the following formula, where C is a preset hyperparameter and L b is the inter-class loss.

[0069]

[0070] L wv The loss is as follows: Minimize L during network training wv Loss can be minimized within the class and maximized between-class losses, thereby introducing clustering characteristics for dimensionality reduction output.

[0071] L wv =L w -L b .

[0072] IAGS network training method, for DBCN, a dual-branch network structure with simultaneous input and output, an iterative alternating network training method IAGS based on a greedy strategy is proposed. The IAGS network training method is as follows:

[0073] Input: L c and L wv Loss ratio λ; L b The hyperparameter C in the loss; the total number of iterations of training E;

[0074] Output: A network that can be used for dimensionality reduction DBCN;

[0075] Step 1, initialization of network parameters, n = 1, 2, ... E, m = 1, 2;

[0076] Step 2, if m = 1, then (L n ) m =(L c ) n +λmax(0,(L wv ) n -(L wv ) n-1 );

[0077] Step 3, if m=2, (L n ) m =λ(L wv ) n +max(0,(L c ) n -(L c ) n-1 );

[0078] Step 4: According to the loss function (L n ) m Back-propagation updates the network parameters.

[0079] The classification loss L is introduced above. c and the inter-class loss L b With the intra-class loss L w The loss function L composed of wv , L c Related to the 1D-CNN coupled network branch, L wv Related to the Word2Vec coupled network branch, the IAGS method alternately trains the two coupled network branches, that is, alternately minimizes L c Loss and L wv Loss, loss function L during the nth iteration training process n The update formula is as follows:

[0080] L n =(L c ) n +λmax(0,(L wv ) n -(L wv ) n-1 ),

[0081] L n =λ(L wv ) n +max(0,(L c )n -(L c ) n-1 ),

[0082] In the above formula, n represents the nth iteration training of the network, L n Represents the loss function corresponding to the nth training, (L i ) n Represents the nth training L c or L wv Calculation of loss, (L i ) n-1 Represents L output after the n-1th training. i loss.

[0083] When solving a problem, the greedy algorithm always makes the current optimal choice, that is, it does not consider the overall optimal solution, and seeks a local optimal solution in a certain sense. After multiple iterations, the global optimal solution can be obtained. The IAGS method proposed in this paper is based on the greedy algorithm. Each training alternately updates the parameters of the single-branch network, that is, to obtain the local optimal solution of the network parameters. After multiple iterations of training, the global optimal solution of the entire DBCN network can be obtained.

[0084] Use max(0,(L i ) n -(L i ) n-1 ) is to make the two network branches always train in the direction of minimizing the loss, and prevent the performance degradation of the network during the alternating training process. The network performance degradation means that when training one network branch, due to the mutual correlation between the two network branches, the parameters adjusted by the back propagation make the loss of the other network larger. When (L i ) n >(L i ) n-1 , the network loss function will be as shown below. At this time, the two network branches will be trained at the same time. If (L i ) n <(L i ) n-1 , then the loss function L will be simply L c or L wv This is also in line with the idea of ​​the greedy strategy, and each iteration process requires a local optimal solution.

[0085]

[0086] According to the traditional method, L c and L wvThe sum of the two coupled network branches in DBCN is used as an overall loss to train the two coupled network branches in DBCN at the same time. During the training process, the parameters of the entire network may be trapped in the local optimal solution rather than the global optimal solution due to factors such as the different convergence speeds of each network. Based on the IAGS method to train the two network branches in DBCN, each training process can be regarded as alternately fixing the parameters of one network and training the other network branch, and the global optimal solution of the entire network is obtained by iteratively obtaining the local optimal solution.

[0087] The present invention proposes a dual-branch coupled network that can be used for dimensionality reduction of hyperspectral images, and proposes an iterative alternating network training method based on a greedy strategy and a corresponding loss function to train the network. Based on the proposed DBCN network, dimensionality reduction processing of hyperspectral images can introduce clustering characteristics for dimensionality reduction output, thereby achieving the purpose of increasing the discriminability between classes and reducing the difference within classes, thereby solving the problem of "same object, different spectrum, different objects, same spectrum". Training the dual-branch coupled network based on the proposed network training method can solve the local extreme value problem of the DBCN network to a certain extent, thereby obtaining a dimensionality reduction representation that is more conducive to classification.

[0088] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0089] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0090] According to another aspect of the embodiments of the present application, a spectral image classification device for implementing the above-mentioned spectral image classification method is also provided. Figure 7 is a schematic diagram of an optional spectral image classification device according to an embodiment of the present application, such as Figure 7 As shown, the device may include:

[0091] An acquisition unit 71 is used to acquire a first spectral image, wherein the first spectral image is an image collected by a spectral sensor;

[0092] A processing unit 73 is used to perform dimensionality reduction processing on the first spectral image using a dual-branch coupling network to obtain a second spectral image, wherein the characteristic dimension of the second spectral image is lower than the characteristic dimension of the first spectral image;

[0093] The classification unit 75 is used to extract the spatial features of the second spectral image by using a convolutional neural network, and obtain the classification result of the first spectral image through the spatial features.

[0094] It should be noted that the acquisition unit 71 in this embodiment can be used to execute step S202 in the embodiment of the present application, the processing unit 73 in this embodiment can be used to execute step S204 in the embodiment of the present application, and the classification unit 75 in this embodiment can be used to execute step S206 in the embodiment of the present application.

[0095] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules as part of the device can be run in Figure 1 In the hardware environment shown, it can be implemented by software or by hardware.

[0096] Through the above modules, a first spectral image is obtained, which is an image collected by a spectral sensor; a dual-branch coupling network is used to perform dimensionality reduction processing on the first spectral image to obtain a second spectral image, and the feature dimension of the second spectral image is lower than that of the first spectral image; a convolutional neural network is used to extract the spatial features of the second spectral image, and the classification result of the first spectral image is obtained through the spatial features. This scheme maps the class label information from a sparse vector space to a dimensionality reduction space to obtain a trainable cluster center word vector, and uses the word vector to achieve the purpose of increasing inter-class discrimination and reducing intra-class differences, thereby solving the technical problem of low classification accuracy in related technologies.

[0097] Optionally, the dual-branch coupling network includes a first coupling network and a second coupling network, wherein the processing unit, when performing dimensionality reduction processing on the first spectral image using the dual-branch coupling network, is further used to: map the category features of the first spectral image from the first word vector space to the second word vector space through the first coupling network, wherein the spatial dimension of the second word vector space is lower than the spatial dimension of the first word vector, and the category features are used to determine the classification result; and,

[0098] The first spectral image is convolved through the second coupling network.

[0099] Optionally, the first coupling network includes a first word embedding network and a second word embedding network, wherein the device of the present application may further include: a training unit, used to train the first coupling network in the following manner before using the two-branch coupling network to perform dimensionality reduction processing on the first spectral image to obtain the second spectral image: determine the vector matrix between the first sample type and the second sample type of the training sample through the first word embedding network, wherein the first sample type is the currently used sample type, the second sample type is the sample type used before the first sample type, and the vector elements in the vector matrix are the difference vectors between the cluster center vector of the first sample type in the first vector space and the cluster center vector of the second sample type in the first vector space; determine the cluster center vector of the first sample type in the second vector space through the second word embedding network, wherein the spatial dimension of the second vector space is lower than the spatial dimension of the first vector space, and the parameters in the second word embedding network are the same as the parameters in the first word embedding network.

[0100] Optionally, there are multiple second sample types, wherein the training unit is further used to determine the vector matrix between the first sample type and the second sample type of the training sample through the first word embedding network, and determine the difference vector between the cluster center vector of the first sample type in the first vector space and the cluster center vector of the second sample type in the first vector space through the first word embedding network; and use the difference vector between the first sample type and each second sample type as the vector element in the vector matrix.

[0101] Optionally, the training unit is also used to obtain the one-hot encoding of the training samples belonging to the first sample type when determining the cluster center vector of the first sample type in the second vector space through the second word embedding network; input the one-hot encoding of the training sample into the second word embedding network, and determine the cluster center vector of the first sample type in the second vector space through the second word embedding network.

[0102] Optionally, the second coupled network includes a first convolutional network and a second convolutional network, wherein the training unit is further used to: before using the dual-branch coupled network to perform dimensionality reduction processing on the first spectral image to obtain the second spectral image, train the second coupled network in the following manner: extract spectral features in the training samples through the first convolutional network; perform supervised classification using cross entropy loss through the second convolutional network, and output the reduced-dimensional spectral vector.

[0103] Optionally, the training unit is also used to: in the process of training the first coupled network and the second coupled network, train the first coupled network and the second coupled network in an alternating training manner, wherein the loss function used in training includes an intra-class loss function and an inter-class loss function, the intra-class loss function is used for the intra-class difference of the dimensionality reduction output, and the inter-class loss function is used to increase the inter-class difference of the dimensionality reduction output.

[0104] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules as part of the device can be run in Figure 1 In the hardware environment shown, it can be implemented by software or by hardware, wherein the hardware environment includes a network environment.

[0105] According to another aspect of the embodiments of the present application, a server or terminal for implementing the above-mentioned spectral image classification method is also provided.

[0106] Figure 8 is a structural block diagram of a terminal according to an embodiment of the present application, such as Figure 8 As shown, the terminal may include: one or more ( Figure 8 (only one is shown) processor 801, memory 803, and transmission device 805, as shown in Figure 8 As shown, the terminal may also include an input and output device 807.

[0107] Among them, the memory 803 can be used to store software programs and modules, such as the program instructions / modules corresponding to the spectral image classification method and device in the embodiment of the present application. The processor 801 executes various functional applications and data processing by running the software programs and modules stored in the memory 803, that is, realizing the above-mentioned spectral image classification method. The memory 803 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 803 may further include a memory remotely arranged relative to the processor 801, and these remote memories can be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0108] The above-mentioned transmission device 805 is used to receive or send data via a network, and can also be used for data transmission between a processor and a memory. Specific examples of the above-mentioned network may include a wired network and a wireless network. In one example, the transmission device 805 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 805 is a radio frequency (Radio Frequency, RF) module, which is used to communicate with the Internet wirelessly.

[0109] Specifically, the memory 803 is used to store application programs.

[0110] The processor 801 may call the application stored in the memory 803 through the transmission device 805 to perform the following steps:

[0111] Acquire a first spectral image, wherein the first spectral image is an image collected by a spectral sensor;

[0112] Performing dimensionality reduction processing on the first spectral image using a double-branch coupling network to obtain a second spectral image, wherein the characteristic dimension of the second spectral image is lower than the characteristic dimension of the first spectral image;

[0113] A convolutional neural network is used to extract spatial features of the second spectral image, and a classification result of the first spectral image is obtained through the spatial features.

[0114] The processor 801 is further configured to execute the following steps:

[0115] Determine a vector matrix between a first sample type and a second sample type of training samples through the first word embedding network, wherein the first sample type is a currently used sample type, the second sample type is a sample type used before the first sample type, and the vector elements in the vector matrix are difference vectors between a cluster center vector of the first sample type in a first vector space and a cluster center vector of the second sample type in the first vector space;

[0116] Determine a cluster center vector of the first sample type in a second vector space through the second word embedding network, wherein the spatial dimension of the second vector space is lower than the spatial dimension of the first vector space, and the parameters in the second word embedding network are the same as the parameters in the first word embedding network.

[0117] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0118] It can be understood by those skilled in the art that Figure 8 The structure shown is for illustration only, and the terminal may be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 8 It does not limit the structure of the above electronic device. For example, the terminal may also include Figure 8 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 8 Different configurations are shown.

[0119] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0120] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to execute the program code of the spectral image classification method.

[0121] Optionally, in this embodiment, the storage medium may be located on at least one network device among a plurality of network devices in the network shown in the above embodiment.

[0122] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps:

[0123] Acquire a first spectral image, wherein the first spectral image is an image collected by a spectral sensor;

[0124] Performing dimensionality reduction processing on the first spectral image using a double-branch coupling network to obtain a second spectral image, wherein the characteristic dimension of the second spectral image is lower than the characteristic dimension of the first spectral image;

[0125] A convolutional neural network is used to extract spatial features of the second spectral image, and a classification result of the first spectral image is obtained through the spatial features.

[0126] Optionally, the storage medium is further configured to store program codes for executing the following steps:

[0127] Determine a vector matrix between a first sample type and a second sample type of training samples through the first word embedding network, wherein the first sample type is a currently used sample type, the second sample type is a sample type used before the first sample type, and the vector elements in the vector matrix are difference vectors between a cluster center vector of the first sample type in a first vector space and a cluster center vector of the second sample type in the first vector space;

[0128] Determine a cluster center vector of the first sample type in a second vector space through the second word embedding network, wherein the spatial dimension of the second vector space is lower than the spatial dimension of the first vector space, and the parameters in the second word embedding network are the same as the parameters in the first word embedding network.

[0129] In an optional embodiment, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps of any embodiment of the above method.

[0130] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0131] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0132] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0133] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0134] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0135] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0137] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0138] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A spectral image classification method, characterized in that: include: Acquire a first spectral image, wherein the first spectral image is an image collected by a spectral sensor; Using a dual-branch coupling network to perform dimensionality reduction processing on the first spectral image to obtain a second spectral image: mapping the category features of the first spectral image from a first word vector space to a second word vector space through a first coupling network, the spatial dimension of the second word vector space is lower than the spatial dimension of the first word vector, and the category features are used to determine the classification result; performing convolution processing on the first spectral image through a second coupling network, wherein the dual-branch coupling network includes the first coupling network and the second coupling network, and the feature dimension of the second spectral image is lower than the feature dimension of the first spectral image; A convolutional neural network is used to extract spatial features of the second spectral image, and a classification result of the first spectral image is obtained through the spatial features.

2. The method according to claim 1, characterized in that: The first coupled network includes a first word embedding network and a second word embedding network, wherein before using the dual-branch coupled network to perform dimensionality reduction processing on the first spectral image to obtain the second spectral image, the method further includes training the first coupled network in the following manner: Determine a vector matrix between a first sample type and a second sample type of training samples through the first word embedding network, wherein the first sample type is a currently used sample type, the second sample type is a sample type used before the first sample type, and the vector elements in the vector matrix are difference vectors between a cluster center vector of the first sample type in a first vector space and a cluster center vector of the second sample type in the first vector space; Determine a cluster center vector of the first sample type in a second vector space through the second word embedding network, wherein the spatial dimension of the second vector space is lower than the spatial dimension of the first vector space, and the parameters in the second word embedding network are the same as the parameters in the first word embedding network.

3. The method according to claim 2, characterized in that The second sample type is multiple, wherein determining the vector matrix between the first sample type and the second sample type of the training sample through the first word embedding network includes: Determine, by means of the first word embedding network, a difference vector between a cluster center vector of the first sample type in the first vector space and a cluster center vector of the second sample type in the first vector space; The difference vectors between the first sample type and each of the second sample types are used as vector elements in the vector matrix.

4. The method according to claim 2, characterized in that: Determining the cluster center vector of the first sample type in the second vector space through the second word embedding network includes: Obtaining a one-hot encoding of a training sample belonging to the first sample type; The one-hot encoding of the training sample is input into the second word embedding network, and the cluster center vector of the first sample type in the second vector space is determined by the second word embedding network.

5. The method according to claim 1, characterized in that The second coupled network includes a first convolutional network and a second convolutional network, wherein before using the dual-branch coupled network to perform dimensionality reduction processing on the first spectral image to obtain the second spectral image, the method further includes training the second coupled network in the following manner: Extracting spectral features in training samples through the first convolutional network; The second convolutional network performs supervised classification using cross entropy loss and outputs a spectral vector after dimension reduction.

6. The method according to any one of claims 1 to 5, characterized in that In the process of training the first coupling network and the second coupling network, the method further includes: The first coupled network and the second coupled network are trained in an alternating training manner, wherein the loss function used in the training includes an intra-class loss function and an inter-class loss function, the intra-class loss function is used to reduce the intra-class difference of the output, and the inter-class loss function is used to increase the inter-class difference of the dimensionality reduction output.

7. A spectral image classification device, characterized in that: include: An acquisition unit, configured to acquire a first spectral image, wherein the first spectral image is an image acquired by a spectral sensor; A processing unit, configured to perform dimensionality reduction processing on the first spectral image using a dual-branch coupling network to obtain a second spectral image: mapping the category feature of the first spectral image from a first word vector space to a second word vector space through a first coupling network, wherein the spatial dimension of the second word vector space is lower than the spatial dimension of the first word vector, and the category feature is used to determine a classification result; performing convolution processing on the first spectral image through a second coupling network, wherein the dual-branch coupling network includes a first coupling network and a second coupling network, and the feature dimension of the second spectral image is lower than the feature dimension of the first spectral image; A classification unit is used to extract the spatial features of the second spectral image by using a convolutional neural network, and obtain a classification result of the first spectral image by using the spatial features.

8. A storage medium, characterized in that: The storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 6 when executed.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor executes the method described in any one of claims 1 to 6 through the computer program.

Images