Discriminative distribution adaptive multi-modal remote sensing image collaborative classification method based on structure preserving
By introducing discriminative least squares and spatial constraints into the learning of multispectral and hyperspectral mapping matrices, the problem of insufficient discriminative power of mapped features in multimodal remote sensing image collaborative classification is solved, and high-precision classification of large-scene remote sensing images is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-25
- Publication Date
- 2026-03-27
AI Technical Summary
Existing multimodal remote sensing image collaborative classification methods are insufficient in improving the discriminative power of mapped features, resulting in low classification accuracy for large-scene remote sensing images.
By introducing discriminative least squares constraints and spatial constraints during the learning of multispectral and hyperspectral mapping matrices, classification error and spatial mapping constraint terms are constructed, increasing the separability of the mapped features and improving the class separability of multispectral images.
It significantly improves the classification accuracy of large-scene multispectral images, achieving high-precision remote sensing image classification.
Smart Images
Figure CN116935224B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to a multi-modal remote sensing image collaborative classification method. BACKGROUND
[0002] Remote sensing image classification is widely used in life, and it is of great significance to realize fine classification of large scene remote sensing images. With the successive launch of satellites by various countries, it becomes easier to obtain large scene multispectral images. Multispectral remote sensing image data usually has wide band and fewer bands, resulting in low spectral discrimination. High spectral remote sensing technology uses spectral imagers to collect tens or even hundreds of spectral bands, which can better distinguish ground objects. However, due to the limitations of high spectral imaging devices, the width is much narrower than that of multispectral images. Therefore, combining the advantages of multi-modal remote sensing images, realizing fine classification of large scene remote sensing images has important significance for the research of multi-modal remote sensing image collaborative classification method.
[0003] At present, there are three main types of multi-modal remote sensing image collaborative classification methods, which are image fusion based method, image reconstruction based method and transfer learning based method. The image fusion based method usually fuses multi-modal images obtained at similar times to obtain high spectral images with high spatial resolution and high spectral resolution. However, it is difficult to obtain multi-modal data at the same time because the revisit period of multispectral data is shorter than that of hyperspectral data; the image reconstruction based method reconstructs the simulated hyperspectral image corresponding to the large scene multispectral image by learning the relationship between multi-modal remote sensing images. However, there is high uncertainty in estimating the regression matrix from limited data, and there is a certain difference between the reconstructed simulated hyperspectral image and the real hyperspectral image, so the classification performance of the reconstructed image may not be enhanced. The transfer learning based method transfers the spectral resolution capability of hyperspectral to multispectral image to realize multi-modal remote sensing image collaborative classification, but most of the methods are only adapted to the distribution of different modal images, without considering how to improve the discriminability of the mapped features. SUMMARY
[0004] The purpose of the present application is to improve the classification accuracy of existing large scene remote sensing images, and a multi-modal remote sensing image collaborative classification method based on discriminative distribution adaptive structure preservation is proposed.
[0005] The specific process of the multi-modal remote sensing image collaborative classification method based on discriminative distribution adaptive structure preservation is as follows:
[0006] Step one, obtain multispectral remote sensing images and hyperspectral remote sensing images covering the same geographical area, and corresponding ground object label images, and construct a classification error constraint term;
[0007] Step two, multiply the hyperspectral remote sensing image samples in the labeled sample set in the hyperspectral remote sensing image in step one by the hyperspectral mapping matrix , obtain mapping hyperspectral data structure information;
[0008] Step one, multiply the multispectral remote sensing image samples in the labeled sample set in the multispectral remote sensing image in step one by the multispectral mapping matrix , obtain mapping multispectral data structure information;
[0009] Based on the mapping hyperspectral data structure information and the mapping multispectral data structure information, a spatial mapping constraint term is constructed;
[0010] The spatial mapping constraint term includes a hyperspectral spatial mapping constraint term, a multispectral spatial mapping constraint term and a spatial mapping constraint term between the multispectral-hyperspectral;
[0011] Step three, based on the hyperspectral remote sensing image samples in the labeled sample set in the hyperspectral remote sensing image in step one, the multispectral remote sensing image samples in the labeled sample set in the multispectral remote sensing image, the hyperspectral mapping matrix , the multispectral mapping matrix , a probability adaptation constraint term is constructed;
[0012] The probability adaptation constraint term includes an edge probability and a conditional probability;
[0013] Step four, the constraint terms obtained in steps one, two and three are used to form an objective function, and the objective function is solved by adopting an alternating iteration, so that a mapping matrix of the multi-mode remote sensing image is obtained;
[0014] The multi-mode remote sensing image includes a multispectral remote sensing image and a hyperspectral remote sensing image;
[0015] The mapping matrix of the multi-mode remote sensing image is a hyperspectral mapping matrix and a multispectral mapping matrix;
[0016] Step five, the labeled samples in the hyperspectral and multispectral remote sensing images covering the same geographic area obtained in step one are multiplied by the hyperspectral mapping matrix and the multispectral mapping matrix obtained in step four respectively, so that the mapping hyperspectral data and the mapping multispectral data are obtained as training data;
[0017] The training data is used to train a classifier, so that a trained classifier is obtained;
[0018] Step six, the multispectral remote sensing image to be measured is multiplied by the multispectral mapping matrix obtained in step four, so that the mapping multispectral remote sensing image to be measured is obtained, and the trained classifier is used to classify the mapping multispectral remote sensing image to be measured, so that a classification result of the multispectral remote sensing image is obtained.
[0019] The beneficial effects of the present application are:
[0020] The present application can solve the above problems by constructing classification error constraints and spatial constraints to extract discriminative features (at present, there are three types of multi-modal remote sensing image collaborative classification methods, which are image fusion-based method, image reconstruction-based method and transfer learning-based method. The image fusion-based method usually fuses multi-modal images obtained at similar times to obtain hyperspectral images with high spatial resolution and high spectral resolution, but it is difficult to obtain multi-modal data at the same time because the revisit period of multispectral data is shorter than that of hyperspectral data; the image reconstruction-based method reconstructs the simulated hyperspectral image corresponding to the multispectral image of a large scene by learning the relationship between multi-modal remote sensing images. However, there is a certain difference between the reconstructed simulated hyperspectral image and the real hyperspectral image, and the classification performance of the reconstructed image is not necessarily enhanced. The transfer learning-based method transfers the spectral resolution capability of hyperspectral to multispectral image to realize multi-modal remote sensing image collaborative classification, but most of the methods are only adapted to the distribution of different modal images, without considering how to improve the discriminability of the mapped features.); Considering that transferring the spectral features of hyperspectral to multispectral image not only solves the spectral mismatch between multispectral and hyperspectral images, but also improves the separability of the mapped features, so as to realize high-precision large scene classification. The design idea of the present application is to impose spatial constraints and classification error constraints on the mapping during learning of the multispectral mapping matrix and the hyperspectral mapping matrix, and to increase the separability of the mapped features. Specifically, in the learning of the multispectral mapping matrix and the hyperspectral mapping matrix, discriminative least squares are introduced as constraint terms, and spatial constraints are added to the probability adaptation of the hyperspectral data and the multispectral data, so that the data of the same class are closer after mapping, and the data of different classes are farther away after mapping, thereby improving the classification ability of the mapped multispectral data; that is, the discriminative distribution self-adaptive multi-modal remote sensing image collaborative classification method based on structure preservation can significantly improve the class separability of multispectral image in theory; therefore, the features obtained by the mapping matrix of the large scene multispectral image have better discriminability, thereby significantly improving the classification precision of large scene remote sensing image.
[0021] The method of the present application learns the multi-modal mapping matrix (hyperspectral mapping matrix and multispectral mapping matrix) by adding discriminative least squares constraints and spatial constraints of hyperspectral data and multispectral image, so that the spectral mismatch between the large scene multispectral image and the hyperspectral image is eliminated after mapping, and the discriminability of the features of different classes is increased, the excellent spectral resolution of the hyperspectral image is transferred to the multispectral image, and finally the high-precision classification of large scene remote sensing image is realized.
[0022] In order to verify the performance of the present application, a group of real hyperspectral-multispectral image pairs are verified, and the experimental results show that, compared with the current representative method, the classification accuracy of the multispectral image is higher. The experimental results verify the effectiveness of the discriminative distribution adaptive multi-modal remote sensing image collaborative classification method based on structure preservation. BRIEF DESCRIPTION OF DRAWINGS
[0023] Figure 1 is an implementation flowchart of the present application;
[0024] Figure 2a is a false color composite image of the hyperspectral image;
[0025] Figure 2b is a false color composite image of the multispectral image, wherein the box represents the overlapping area with the hyperspectral data;
[0026] Figure 3a the ground object label map of the overlapping area, used for learning the mapping matrix;
[0027] Figure 3b is the ground object label map of the test area, used for evaluating the classification effect;
[0028] Figure 4 is a comparison chart of the classification effects of different methods. DETAILED DESCRIPTION
[0029] Specific implementation one: combined Figure 1 In this embodiment, the specific process of the discriminative distribution adaptive multi-modal remote sensing image collaborative classification method based on structure preservation is as follows:
[0030] Step one, obtain the multispectral remote sensing image and the hyperspectral remote sensing image covering the same geographical area, and the corresponding ground object label map, and construct a classification error constraint term for classification accuracy;
[0031] Step two, multiply the hyperspectral remote sensing image samples in the labeled sample set in the hyperspectral remote sensing image in step one by the hyperspectral mapping matrix to obtain the mapped hyperspectral data structure information;
[0032] multiply the multispectral remote sensing image samples in the labeled sample set in the multispectral remote sensing image in step one by the multispectral mapping matrix to obtain the mapped multispectral data structure information;
[0033] Based on the mapped hyperspectral data structure information and the mapped multispectral data structure information, a spatial mapping constraint term is constructed;
[0034] The spatial mapping constraint term comprises a hyperspectral spatial mapping constraint term, a multispectral spatial mapping constraint term and a spatial mapping constraint term between multispectral and hyperspectral;
[0035] Step three, based on the hyperspectral remote sensing image sample in the labeled sample set in step one, the multispectral remote sensing image sample in the labeled sample set in the multispectral remote sensing image, the hyperspectral mapping matrix , the multispectral mapping matrix , a probability adaptation constraint term is constructed;
[0036] The probability adaptation constraint term comprises an edge probability and a conditional probability;
[0037] Step four, the constraint terms obtained in steps one, two and three are used to form a target function, and the target function is solved by using an alternating iteration, so as to obtain a mapping matrix of the multi-mode remote sensing image;
[0038] The multi-mode remote sensing image comprises a multispectral remote sensing image and a hyperspectral remote sensing image;
[0039] The mapping matrix of the multi-mode remote sensing image is a hyperspectral mapping matrix and a multispectral mapping matrix;
[0040] Step five, the labeled samples in the hyperspectral and multispectral remote sensing images covering the same geographical area obtained in step one are multiplied by the hyperspectral mapping matrix and the multispectral mapping matrix obtained in step four, so as to obtain mapped hyperspectral data and mapped multispectral data as training data;
[0041] The training data is used to train a classifier, so as to obtain a trained classifier (the classifier uses a nearest neighbor 1NN);
[0042] Step six, the multispectral remote sensing image to be tested is multiplied by the multispectral mapping matrix obtained in step four, so as to obtain a mapped multispectral remote sensing image to be tested, and the trained classifier is used to classify the mapped multispectral remote sensing image to be tested, so as to obtain a fine classification result of the multispectral remote sensing image.
[0043] Specific implementation method two: the difference between this implementation method and the specific implementation method one is that: in step one, the multispectral remote sensing image and the hyperspectral remote sensing image covering the same geographical area are obtained, and a corresponding ground object label map is obtained, and a classification error constraint term is constructed for classification accuracy; the specific process is as follows:
[0044] The pixel points of the common coverage area of the hyperspectral remote sensing image and the multispectral remote sensing image are obtained, and each pixel point is a sample;
[0045] Let and respectively represent the set of labeled samples in the hyperspectral remote sensing image and the multispectral remote sensing image respectively covering the same geographical area;
[0046] represents the label of the i-th sample in the set of labeled samples in the hyperspectral remote sensing image;
[0047] represents the label of the i-th sample in the set of labeled samples in the multispectral remote sensing image;
[0048] and respectively represent the matrix composed of the label one-hot vector encoding of the labeled samples in the input hyperspectral and multispectral remote sensing images respectively covering the same geographical area;
[0049] wherein, represents the 1st hyperspectral remote sensing image sample in the set of labeled samples in the hyperspectral remote sensing image, represents the i-th hyperspectral remote sensing image sample in the set of labeled samples in the hyperspectral remote sensing image, represents the spectral dimension of the sample in the hyperspectral remote sensing image, represents the number of hyperspectral remote sensing image samples (pixels);
[0050] represents the 1st multispectral remote sensing image sample in the set of labeled samples in the multispectral remote sensing image, represents the i-th multispectral remote sensing image sample in the set of labeled samples in the multispectral remote sensing image, represents the spectral dimension of the sample in the multispectral remote sensing image, represents the number of multispectral remote sensing image samples (pixels); represents the set of real numbers;
[0051] represents the one-hot vector encoding of the label of the i-th hyperspectral remote sensing image sample in the set of labeled samples in the hyperspectral remote sensing image;
[0052] represents the one-hot vector encoding of the label of the i-th multispectral remote sensing image sample in the set of labeled samples in the multispectral remote sensing image; represents the one-hot vector encoding of the label of the 1st hyperspectral remote sensing image sample in the set of labeled samples in the hyperspectral remote sensing image, represents the one-hot vector encoding of the label of the i-th hyperspectral remote sensing image sample in the set of labeled samples in the hyperspectral remote sensing image,
[0053] represents the one-hot vector encoding of the label of the 1st hyperspectral remote sensing image sample in the set of labeled samples in the hyperspectral remote sensing image, represents the one-hot vector encoding of the label of the i-th hyperspectral remote sensing image sample in the set of labeled samples in the hyperspectral remote sensing image, A one-hot vector encoding of a hyperspectral remote sensing image sample label;
[0054] A one-hot vector encoding of a first multispectral remote sensing image sample label in the set of labeled samples in the multispectral remote sensing image, A one-hot vector encoding of a second multispectral remote sensing image sample label in the set of labeled samples in the multispectral remote sensing image, A one-hot vector encoding of a third multispectral remote sensing image sample label in the set of labeled samples in the multispectral remote sensing image;
[0055] T represents transposition, and c represents the total number of ground object categories;
[0056] The classification error constraint term for the multi-modal remote sensing image can be represented as:
[0057]
[0058] The multi-modal remote sensing image includes a multispectral remote sensing image and a hyperspectral remote sensing image;
[0059] wherein, represents a discriminant target term, represents a hyperspectral mapping matrix that needs to be learned and solved (step four, alternating iteration is solved to obtain the mapping matrix of the multi-modal remote sensing image, and the initial value is randomly assigned;), represents a multispectral mapping matrix that needs to be learned and solved (step four, alternating iteration is solved to obtain the mapping matrix of the multi-modal remote sensing image, and the initial value is randomly assigned;) represents the dimension of the data after mapping to the subspace, represents the square of the Frobenius norm (F norm for short) of the matrix, represents a multispectral traction matrix, represents a hyperspectral traction matrix; and is a relaxation matrix introduced, is a non-negative matrix that needs to be learned, and can magnify the distance between two data points of different categories, which helps to improve the distinguishability of the data; is a Hadamard product operator of the matrix.
[0060] The other steps are the same as in the first embodiment.
[0061] The third embodiment is different from the first or second embodiment in that the multispectral traction matrix ;
[0062] The multispectral traction matrix is defined as
[0063] wherein, the multispectral traction matrix is the first whether the i-th sample belongs to the j-th class, belongs to , does not belong to ; ; represents whether the i-th sample belongs to the j-th class, belongs to , does not belong to ; ; ; , ;
[0064] the hyperspectral traction matrix ;
[0065] the hyperspectral traction matrix ,
[0066] wherein, represents the hyperspectral traction matrix whether the i-th sample belongs to the j-th class, belongs to , does not belong to ; ; ; represents whether the i-th sample belongs to the j-th class, belongs to , does not belong to ; ; ; , .
[0067] The other steps are the same as those in embodiment one or two.
[0068] Embodiment four: different from one of embodiments one to three, in the classification error constraint term, the first term is a multispectral data classification error;
[0069] the second term in the classification error constraint term is a hyperspectral data classification error;
[0070] The first term and the second term are used to ensure that the classification accuracy of each class after mapping is as high as possible;
[0071] the third term in the classification error constraint term and the fourth term are overfitting prevention terms.
[0072] The other steps are the same as those in one of embodiments one to three.
[0073] Embodiment five: different from one of embodiments one to four, in the step two
[0074] The hyperspectral remote sensing image sample in the labeled sample set in step one is multiplied by the hyperspectral mapping matrix , to obtain the mapped hyperspectral data structure information;
[0075] The multispectral remote sensing image sample in the labeled sample set in step one is multiplied by the multispectral mapping matrix , to obtain the mapped multispectral data structure information;
[0076] Based on the mapped hyperspectral data structure information and the mapped multispectral data structure information, a spatial mapping constraint term is constructed;
[0077] Before mapping, it is the original hyperspectral image, and after mapping, it is the data obtained by multiplying the original hyperspectral image by the mapping matrix. Here, the constraint is made on the mapped data.
[0078] The spatial mapping constraint term includes a hyperspectral spatial mapping constraint term, a multispectral spatial mapping constraint term, and a spatial mapping constraint term between the multispectral and the hyperspectral;
[0079] The specific process is as follows:
[0080]
[0081] Wherein, represents the spatial mapping constraint term;
[0082] represents the i-th hyperspectral remote sensing image sample in the labeled sample set in the hyperspectral remote sensing image, ;
[0083] represents the j-th hyperspectral remote sensing image sample in the labeled sample set in the hyperspectral remote sensing image, ;
[0084] represents the i-th multispectral remote sensing image sample in the labeled sample set in the multispectral remote sensing image, ;
[0085] represents the j-th multispectral remote sensing image sample in the labeled sample set in the multispectral remote sensing image, ; represents the square of the 2-norm;
[0086] represents the transpose of
[0087] represents , and represents transpose;
[0088] This is a similarity matrix between hyperspectral remote sensing images. This represents a similarity matrix between hyperspectral remote sensing images, used to measure the similarity between hyperspectral samples; if ,but ,otherwise, , Indicates the hyperspectral first The labels of each sample;
[0089] It is a similarity matrix between multispectral remote sensing images. A similarity matrix representing multispectral remote sensing images is used to measure the similarity between multispectral samples; if ,but ,otherwise , Indicates multispectral first The labels of each sample;
[0090] It is a similarity matrix between hyperspectral and multispectral remote sensing images. This represents a similarity matrix between hyperspectral and multispectral remote sensing images, used to measure the similarity between hyperspectral and multispectral samples; if ,but ,otherwise .
[0091] The first constraint term in the above formula It involves mapping constraints onto the hyperspectral feature space, if in the original space... and Adjacent data should remain adjacent in the mapping space. Otherwise, they should be far apart. Minimizing this term ensures that data that are not neighbors in the source domain are separated as much as possible in the mapping space. The second term in the above equation... This involves spatial constraints on feature data from different domains (hyperspectral and multispectral). Minimizing the second term of the formula aligns data from different domains within the mapping space. The third term... Spatial constraints on multispectral feature domains. Similarly, if they are nearest neighbors in the original space, then they should remain nearest neighbors in the mapped space. Therefore, minimizing the above equation can constrain similar data in hyperspectral and multispectral regions to be mapped to similar locations, while dissimilar data are separated in the mapped space.
[0092] The other steps and parameters are the same as those in one of the specific implementation methods one to four.
[0093] Specific implementation six: different from one of the specific implementations one to five is that: in the step three, based on the hyperspectral remote sensing image sample in the labeled sample set in the step one, the labeled sample set in the multispectral remote sensing image, the hyperspectral mapping matrix , the multispectral mapping matrix , a probability adaptation constraint term is constructed (that is, the distribution adaptation is performed on ); the specific process is as follows:
[0094]
[0095] Among them, is the feature of the multispectral data and the hyperspectral data in the subspace; indicates the feature of the hyperspectral data, indicates the feature of the multispectral data, indicates the hyperspectral pixel belonging to the k-th class, indicates the multispectral pixel belonging to the k-th class, indicates the number of the hyperspectral pixels belonging to the k-th class, indicates the number of the multispectral pixels belonging to the k-th class. 、 indicates the matrix. The first term in the above formula indicates the edge probability adaptation of the hyperspectral mapped feature and the multispectral mapped feature, and the second term
[0096] indicates the conditional probability adaptation of the hyperspectral mapped feature and the multispectral mapped feature. The probability adaptation constraint term includes the first term edge probability and the second term conditional probability
[0097] .
[0098] The other steps and parameters are the same as one of the specific implementations one to five.
[0099] Specific implementation seven: different from one of the specific implementations one to six is that: the matrix and are as follows:
[0100]
[0101]
[0102] Among them, is the label (the label after data mapping) of . For the label after data mapping, and respectively represent the first and the second sample, , ; represents the intersection, represents the kth category, represents the element in the matrix , represents the element in the matrix .
[0103] The other steps and parameters are the same as one of the first to sixth embodiments.
[0104] Embodiment eight: different from one of the first to seventh embodiments, the step four utilizes the constraint terms obtained in steps one, two and three to form a target function, and the target function is solved by alternating iteration to obtain the mapping matrix of the multi-mode remote sensing image.
[0105] The multi-mode remote sensing image includes a multi-spectral remote sensing image and a hyperspectral remote sensing image.
[0106] The mapping matrix of the multi-mode remote sensing image is a hyperspectral mapping matrix and a multi-spectral mapping matrix.
[0107] The specific process is:
[0108] The spatial mapping constraint term in step two is:
[0109]
[0110] The three terms of the spatial mapping constraint term in step two are binomially expanded, and then converted into matrix multiplication and matrix trace form, the first term , the second term and the third term are rearranged as follows:
[0111]
[0112] Among them, , represents introducing a hyperspectral intermediate representation matrix, represents the similarity matrix of the hyperspectral remote sensing image, , represents introducing a multi-spectral intermediate representation matrix, represents the similarity matrix of the multi-spectral remote sensing image, , , and is a diagonal matrix;
[0113] Therefore, the objective function of the discriminative distribution adaptive multi-modal remote sensing data collaborative classification method with structure preservation is as follows:
[0114]
[0115] wherein, , , , is a sub-matrix depending on the number of multispectral and hyperspectral image samples, 、 、 is a penalty coefficient for balancing the importance of the constraint term; 、 、 The term multiplied by the coefficient is a constraint term, and the importance thereof is set by the penalty coefficient;
[0116] An alternating optimization method is used to iterate the objective function to obtain a mapping matrix of the multi-modal remote sensing image.
[0117] The multi-modal remote sensing image includes a multispectral remote sensing image and a hyperspectral remote sensing image.
[0118] The mapping matrix of the multi-modal remote sensing image is a hyperspectral mapping matrix and a multispectral mapping matrix.
[0119] The other steps and parameters are the same as one of the first to seventh embodiments.
[0120] The ninth embodiment is different from one of the first to eighth embodiments in that an alternating optimization method is used to iterate the objective function to obtain a mapping matrix of the multi-modal remote sensing image.
[0121] The optimization process is as follows:
[0122] 1) Fix , and , update , and the objective function is re-expressed as follows:
[0123]
[0124] Take the derivative of to obtain , and let , and can be obtained:
[0125]
[0126] wherein, denotes the identity matrix matrix; is the multiplication sign;
[0127] 2) fixing , and updating , the objective function can be re-expressed as follows:
[0128]
[0129] Taking the derivative of obtains , and letting , we can get :
[0130]
[0131] wherein, denotes the identity matrix matrix;
[0132] 3) fixing , and updating , the objective function can be re-expressed as follows:
[0133]
[0134]
[0135] wherein, denotes the intermediate matrix, ;
[0136] 4) fixing , and updating , the objective function can be re-expressed as follows:
[0137]
[0138]
[0139] wherein, denotes the intermediate matrix, ;
[0140] 5) repeating 1) to 4) until convergence;
[0141] When convergence is reached, the iteration is stopped, and the multi-modal mapping matrix is obtained;
[0142] The multi-modal mapping matrix comprises a hyperspectral mapping matrix and a multispectral mapping matrix, is a multispectral mapping matrix, is a hyperspectral mapping matrix.
[0143] The other steps and parameters are the same as one of embodiments 1-8.
[0144] Embodiment 10: Different from one of embodiments 1-9, in step 5, the labeled samples in the hyperspectral and multispectral remote sensing images covering the same geographic area obtained in step 1 are respectively multiplied by the hyperspectral mapping matrix and the multispectral mapping matrix obtained in step 4 to obtain mapped hyperspectral data and mapped multispectral data as training data.
[0145] The training data is used to train the classifier, and a trained classifier (the classifier uses nearest neighbor 1NN) is obtained.
[0146] The specific process is as follows:
[0147] Let be the multispectral remote sensing image covering the same geographic area input in step 1, be the hyperspectral image covering the same geographic area input in step 1;
[0148] The mapped hyperspectral data and the mapped multispectral data are as follows:
[0149]
[0150] wherein, and are the mapped multispectral data and the mapped hyperspectral data, respectively;
[0151] The mapped hyperspectral data and the mapped multispectral data are used as training data.
[0152] The training data is used to train the classifier, and a trained classifier is obtained.
[0153] The other steps and parameters are the same as one of embodiments 1-9.
[0154] The beneficial effects of the present application are verified by the following embodiments:
[0155] Embodiment 1:
[0156] The discriminative distribution adaptive multi-modal remote sensing data collaborative classification method based on structure preservation is prepared according to the following steps:
[0157] The multi-modal dataset used in the experiment includes a set of hyperspectral images and multispectral images collected in the Yellow River Delta National Nature Reserve. Among them, the hyperspectral image is acquired by GF-5 satellite on November 1, 2018, which contains 330 bands, the wavelength is from 400 nm to 2500 nm, and the spatial resolution is 30 meters per pixel. After removing the noise band and the water absorption band, a total of 295 bands are reserved. The multispectral data is acquired by the wide camera of Gaofen-1 satellite on October 31, 2018, which has 4 bands, the spatial resolution is 16 meters, and the total number of pixels is 1066*1108. The nearest neighbor interpolation method is used to upsample the hyperspectral image to the same spatial resolution as the multispectral image, and the overlapping area has a spatial size of 404*448. The ground objects in the observation scene are divided into 8 categories, which are: reed, Spartina alterniflora, Suaeda salsa, water, saline beach, tamarisk, bare beach and tidal reed. Figure 2a and Figure 2b respectively show the false color composite images of the hyperspectral image and the multispectral image, Figure 3a and Figure 3b show the ground truth of the ground objects in the overlapping area and the ground truth of the ground objects in the non-overlapping area in the observation scene. In the experiment, the classification effect of the original large scene multispectral image is taken as the baseline method, and four kinds of representative collaborative classification methods are compared, Figure 4 the classification results are given, and four indexes of classification accuracy of each class, average classification accuracy (AA), overall classification accuracy (OA) and kappa coefficient are used to evaluate the experimental results. Figure 4 It can be seen that the method (SPDDA) achieves the highest classification accuracy on four kinds of categories, and the AA, OA and Kappa coefficient are higher than those of other comparison methods. The experimental results verify the effectiveness of the structure-preserving discriminative distribution adaptive multi-modal remote sensing image collaborative classification method proposed in the present application for fine classification of large scene remote sensing images.
[0158] The present application can also have other various embodiments, and those skilled in the art can make various corresponding changes and modifications according to the present application without departing from the spirit and essence of the present application, but these corresponding changes and modifications should all belong to the protection scope of the claims attached to the present application.
Claims
1. A discriminative distribution adaptation multi-modal remote sensing image collaborative classification method based on structure preservation, characterized in that: The method specifically comprises the following steps: Step 1: obtaining multispectral remote sensing images and hyperspectral remote sensing images covering the same geographic area, and corresponding ground object label maps, and constructing a classification error constraint term; Step two, multiply the hyperspectral remote sensing image sample in the labeled sample set in step one by the hyperspectral mapping matrix to obtain the mapped hyperspectral data structure information Multiplying the multispectral remote sensing image sample in the labeled sample set in step one by the multispectral mapping matrix to obtain mapped multispectral data structure information Based on the structure information of the mapped hyperspectral data and the structure information of the mapped multispectral data, a spatial mapping constraint term is constructed; The spatial mapping constraint term comprises a hyperspectral spatial mapping constraint term, a multispectral spatial mapping constraint term and a spatial mapping constraint term between the multispectral and the hyperspectral; Step three, based on the hyperspectral remote sensing image sample in the labeled sample set in step one, the multispectral remote sensing image sample in the labeled sample set in the multispectral remote sensing image, the hyperspectral mapping matrix , the multispectral mapping matrix , construct a probability adaptation constraint term; The probability adaptation constraint term comprises an edge probability and a conditional probability; Step 4: using the constraint terms obtained in steps 1, 2 and 3 to form a target function, and using an alternating iteration to solve the target function, so as to obtain a mapping matrix of the multi-mode remote sensing images; The multi-mode remote sensing images comprise multispectral remote sensing images and hyperspectral remote sensing images; The mapping matrix of the multi-mode remote sensing images is a hyperspectral mapping matrix and a multispectral mapping matrix; Step 5: multiplying the labeled samples in the hyperspectral and multispectral remote sensing images obtained in step 1 by the hyperspectral mapping matrix and the multispectral mapping matrix obtained in step 4, so as to obtain mapped hyperspectral data and mapped multispectral data as training data; Using the training data to train a classifier, a trained classifier is obtained; Step 6: multiplying the to-be-detected multispectral remote sensing images by the multispectral mapping matrix obtained in step 4, so as to obtain mapped to-be-detected multispectral remote sensing images, and using the trained classifier to classify the mapped to-be-detected multispectral remote sensing images, so as to obtain a classification result of the multispectral remote sensing images.
2. The structural preserving discriminative distribution adaptive multi-modal remote sensing image co-classification method according to claim 1, characterized in that: In step 1, the multispectral remote sensing images and the hyperspectral remote sensing images covering the same geographic area are obtained, and corresponding ground object label maps are obtained, and a classification error constraint term is constructed; the specific process is as follows: Obtaining pixel points in a common coverage area of the hyperspectral remote sensing images and the multispectral remote sensing images, each pixel point being a sample; Let and respectively represent the set of labeled samples in the hyperspectral remote sensing image and the multispectral remote sensing image covering the same geographic area; represents a label of an i-th sample in a set of labeled samples in a hyperspectral remote sensing image represents a label of an i-th sample in a set of labeled samples in a hyperspectral remote sensing image represents a label of an i-th sample in a set of labeled samples in a multispectral remote sensing image; represents a label of an i-th sample in a set of labeled samples in a multispectral remote sensing image; and respectively represent the matrix composed of label one-hot vector encoding of labeled samples in the input hyperspectral and multispectral remote sensing images with the same covered geographic area; wherein, represents the first hyperspectral remote sensing image sample in the labeled sample set in the hyperspectral remote sensing image, represents the first hyperspectral remote sensing image sample in the labeled sample set in the hyperspectral remote sensing image, represents the first hyperspectral remote sensing image sample in the labeled sample set in the hyperspectral remote sensing image, represents the spectral dimension of the sample in the hyperspectral remote sensing image, represents the number of hyperspectral remote sensing image samples; represents the first multispectral remote sensing image sample in the labeled sample set in the multispectral remote sensing image, represents the first multispectral remote sensing image sample in the labeled sample set in the multispectral remote sensing image, represents the first multispectral remote sensing image sample in the labeled sample set in the multispectral remote sensing image, represents the spectral dimension of the sample in the multispectral remote sensing image, represents the number of multispectral remote sensing image samples; denotes the set of real numbers; represents one-hot vector encoding of a label of an i-th hyperspectral remote sensing image sample in a set of labeled samples in a hyperspectral remote sensing image; represents one-hot vector encoding of a label of an i-th hyperspectral remote sensing image sample in a set of labeled samples in a hyperspectral remote sensing image; represents one-hot vector encoding of a label of an i-th multispectral remote sensing image sample in a set of labeled samples in a multispectral remote sensing image; represents one-hot vector encoding of a label of an i-th multispectral remote sensing image sample in a set of labeled samples in a multispectral remote sensing image; represents one-hot vector encoding of a label of a first hyperspectral remote sensing image sample in the labeled sample set in the hyperspectral remote sensing image, represents one-hot vector encoding of a label of a first hyperspectral remote sensing image sample in the labeled sample set in the hyperspectral remote sensing image, represents one-hot vector encoding of a label of a first hyperspectral remote sensing image sample in the labeled sample set in the hyperspectral remote sensing image, a one-hot vector encoding representing a label of a first multispectral remote sensing image sample in the labeled sample set, a one-hot vector encoding representing a label of a first multispectral remote sensing image sample in the labeled sample set, a one-hot vector encoding representing a label of a first multispectral remote sensing image sample in the labeled sample set, T represents transposition, and c represents the total number of ground object categories; The classification error constraint term can be expressed as: wherein denotes the discriminant target item, denotes the hyperspectral mapping matrix, denotes the multispectral mapping matrix, denotes the square of the Frobenius norm of a matrix, denotes the multispectral pullback matrix, denotes the hyperspectral pullback matrix; and is the introduced relaxation matrix; is the Hadamard product operator of matrices.
3. The structural preserving discriminative distribution adaptation based multi-modal remote sensing image co-classification method according to claim 2, characterized in that: The multispectral traction matrix ; Defining multispectral traction matrix , wherein, denotes a multispectral traction matrix whether the th sample belongs to the th class, belongs to , does not belong to ; denotes whether the th sample belongs to the th class, belongs to , does not belong to ; , ; The hyperspectral traction matrix ; Defining hyperspectral traction matrix , wherein, represents a hyperspectral pull matrix the thsample belongs to the thclass, belongs to , does not belong to ; represents the thsample belongs to the thclass, belongs to , does not belong to ; , .
4. The structural preserving discriminative distribution adaptive multi-modal remote sensing image co-classification method according to claim 3, characterized in that: the first term of the classification error constraint term is a multi-spectral data classification error; the second term of the classification error constraint term classification error for hyperspectral data; the third term in the classification error constraint and the fourth term to prevent overfitting 5. The structural preserving discriminative distribution adaptive multi-modal remote sensing image co-classification method according to claim 4, characterized in that: The step two is multiplying the hyperspectral remote sensing image sample in the labeled sample set in the step one by the hyperspectral mapping matrix to obtain the mapped hyperspectral data structure information; multiplying the multispectral remote sensing image samples in the labeled sample set in step one by the multispectral mapping matrix to obtain mapped multispectral data structure information Based on the structure information of the mapped hyperspectral data and the structure information of the mapped multispectral data, a spatial mapping constraint term is constructed; The spatial mapping constraint term comprises a hyperspectral spatial mapping constraint term, a multispectral spatial mapping constraint term and a spatial mapping constraint term between the multispectral and the hyperspectral; The specific process is as follows: wherein denotes a spatial mapping constraint term; represents the i-th hyperspectral remote sensing image sample in the labeled sample set in the hyperspectral remote sensing image, ; represents the jth hyperspectral remote sensing image sample in the labeled sample set in the hyperspectral remote sensing image, ; represents a labeled sample in a set of labeled samples in a multispectral remote sensing image, a multispectral remote sensing image sample, ; represents a labeled sample in a set of labeled samples in a multispectral remote sensing image, a multispectral remote sensing image sample, ; denotes the square of the 2-norm; denotes the transpose of denotes the transpose of denotes the transpose of represents a similarity matrix between hyperspectral remote sensing images; if , then , otherwise, , represents the label of the hyperspectral i-th sample; sample; denotes a similarity matrix between multispectral remote sensing images; if , then , otherwise , denotes the label of the multispectral i-th sample; denotes the label of the multispectral i-th sample; represents a similarity matrix between hyperspectral remote sensing images and multispectral remote sensing images; if , then , else .
6. The structural preserving discriminative distribution adaptive multi-modal remote sensing image co-classification method according to claim 5, characterized in that: The step three is based on the hyperspectral remote sensing image sample in the labeled sample set in step one, the multispectral remote sensing image sample in the labeled sample set in the multispectral remote sensing image, the hyperspectral mapping matrix , the multispectral mapping matrix , and a probability adaptation constraint term is constructed; the specific process is as follows: wherein, is a feature of the multispectral data and the hyperspectral data in the subspace; denotes a feature of the hyperspectral data, denotes a feature of the multispectral data, denotes a hyperspectral pixel belonging to the k-th class, denotes a multispectral pixel belonging to the k-th class, denotes the number of hyperspectral pixels belonging to the k-th class, denotes the number of multispectral pixels belonging to the k-th class, denotes the number of hyperspectral pixels belonging to the k-th class, denotes the number of multispectral pixels belonging to the k-th class; , denotes a matrix.
7. The structural preserving discriminative distribution adaptive multi-modal remote sensing image co-classification method according to claim 6, characterized in that: The matrix and is represented as follows: wherein, is a label; is a label; and denote the th and the th sample of the th class, , ; denotes the intersection, denotes the th class, denotes the element in the matrix denotes the element in the matrix . 8. The structural preserving discriminative distribution adaptive multi-modal remote sensing image co-classification method according to claim 7, characterized in that: In step 4, the constraint terms obtained in steps 1, 2 and 3 are used to form a target function, and an alternating iteration is used to solve the target function, so as to obtain a mapping matrix of the multi-mode remote sensing images; The multi-mode remote sensing images comprise multispectral remote sensing images and hyperspectral remote sensing images; The mapping matrix of the multi-mode remote sensing images is a hyperspectral mapping matrix and a multispectral mapping matrix; The specific process is as follows: In step 2, the spatial mapping constraint term is as follows: The three terms of the spatial mapping constraint in step two are binomially expanded and then transformed into the form of matrix multiplication and matrix trace, the first term , the second term and the third term are rearranged as follows: wherein, , denotes the introduction of a hyperspectral intermediate representation matrix, denotes a similarity matrix of the hyperspectral remote sensing image, , denotes the introduction of a multispectral intermediate representation matrix, denotes a similarity matrix of the multispectral remote sensing image, , , and is a diagonal matrix; Therefore, the target function is arranged as follows: wherein, , , , is a sub-matrix, , , is a penalty coefficient; is a similarity matrix between hyperspectral remote sensing images and multispectral remote sensing images; An alternating optimization method is used to iteratively solve the target function, so as to obtain a mapping matrix of the multi-mode remote sensing images; The multi-mode remote sensing images comprise multispectral remote sensing images and hyperspectral remote sensing images; The mapping matrix of the multi-mode remote sensing images is a hyperspectral mapping matrix and a multispectral mapping matrix.
9. The structural preserving discriminative distribution adaptation based multi-modal remote sensing image co-classification method according to claim 8, characterized in that: The alternating optimization method is used to iteratively solve the target function, so as to obtain a mapping matrix of the multi-mode remote sensing images; The optimization process is as follows: 1) fixed , and , update , the objective function is re-expressed as follows: For derivation, we obtain and let , we have : wherein denotes the identity matrix matrix; is the multiplication sign; 2) fixed , and , update , the objective function can be re-expressed as follows: For derivation, we obtain and let We can obtain : wherein denotes the identity matrix matrix; 3) fixed , and , update , the objective function can be re-expressed as follows: wherein represents an intermediate matrix, ; 4) fixed , and , update , the objective function can be re-expressed as follows: wherein represents an intermediate matrix, ; 5) repeat 1) to 4) until convergence; When convergence, stop iteration, get multi-modal mapping matrix; The multi-modal mapping matrix includes a hyperspectral mapping matrix and a multispectral mapping matrix, is a multispectral mapping matrix, is a hyperspectral mapping matrix.
10. The structural preserving discriminative distribution adaptive multi-modal remote sensing image co-classification method according to claim 9, characterized in that: The labeled samples in the hyperspectral and multispectral remote sensing images of the same coverage geographical area obtained in step one are multiplied by the hyperspectral mapping matrix and the multispectral mapping matrix obtained in step four respectively in step five, to obtain mapped hyperspectral data and mapped multispectral data as training data; The training data is used to train the classifier, and a trained classifier is obtained; The specific process is: Let a multispectral remote sensing image covering the same geographic area as the input of step one, a hyperspectral image covering the same geographic area as the input of step one; The mapped hyperspectral data and the mapped multispectral data are: wherein, and are the mapped multispectral data and the mapped hyperspectral data, respectively; The mapped hyperspectral data and the mapped multispectral data are used as training data; The training data is used to train the classifier, and a trained classifier is obtained.