A feature-adaptive mutual-guiding multi-source information fusion classification method and system

Abstract

The present invention discloses a feature adaptive mutual-guided multi-source information fusion classification method and system, the method comprising: obtaining a hyperspectral image and a laser radar image data set and performing data preprocessing to obtain a training set; training a preset feature adaptive mutual-guided multi-source information fusion network based on the training set to obtain a fully trained feature adaptive mutual-guided multi-source information fusion network; obtaining an image to be tested and classifying the image to be tested based on the fully trained feature adaptive mutual-guided multi-source information fusion network to obtain an image classification result. The system comprises: a preprocessing part, a network training part and a task classification part. By using the present invention, the feature complementarity of different information source data can be fully explored, and feature data fusion of multiple data source types can be compatible, which has a strong practical application value. The present invention can be widely used in the field of multi-source information classification.

Description

Technical Field

[0001] The present invention relates to the field of multi-source information fusion classification, and in particular to a feature adaptive mutual-guiding multi-source information fusion classification method and system. Background Art

[0002] At present, the research on the method of feature extraction based on multi-source fusion data has attracted extensive attention from researchers. The mainstream machine learning multi-source data feature extraction methods include multi-kernel learning methods, dimensionality reduction-based models, subspace association algorithms, etc., but the above methods are highly based on the quality of manual feature annotation, which limits the application of complex scenes. Methods based on deep learning can autonomously learn semantic features from raw data, but this type of method generally uses different network structures to independently extract features from information from different data sources, and then concentrates the features for classification, such as fusion structures based on deep residual networks, block-to-block convolutional neural network structures for unsupervised feature extraction, etc. In summary, the existing methods have the following shortcomings: lack of attention to the specificity of hyperspectral channels and the complementarity of spatial information of hyperspectral and lidar data. Secondly, feature-level fusion networks usually extract spatial features of multiple resources independently, often ignoring the complementary relationship between different spatial expressions of features. Thirdly, deep networks generate hierarchical semantic features from low to high; however, these hierarchical features are usually combined with simple and fixed element-by-element addition or connection strategies. In addition, multimodal features at the same level have almost no interaction and guidance, which limits their semantic relevance. Summary of the invention

[0003] The purpose of the present invention is to provide a feature-adaptive and mutually guided multi-source information fusion classification method and system to solve the problem of inaccurate classification accuracy of multi-source fusion data sets with complex spatial structure features.

[0004] The first technical solution adopted by the present invention is: a feature adaptive mutual guidance multi-source information fusion classification method, comprising the following steps:

[0005] Obtain hyperspectral image and lidar image data sets and perform data preprocessing to obtain training sets;

[0006] Based on the training set, a preset feature adaptive mutual guided multi-source information fusion network is trained to obtain a fully trained feature adaptive mutual guided multi-source information fusion network;

[0007] The feature adaptive mutual guidance multi-source information fusion network includes a spectrum-space feature learning module, a spectrum-space mutual guidance module and a multimodal data fusion classification module;

[0008] The image to be tested is obtained and classified based on a fully trained feature adaptive mutual guided multi-source information fusion network to obtain the image classification result.

[0009] Furthermore, the step of obtaining a hyperspectral image and a lidar image data set and performing data preprocessing to obtain a training set specifically includes:

[0010] Obtain a hyperspectral image and its corresponding lidar image covering the same area of ​​the Earth’s surface;

[0011] Reduce the dimension of the hyperspectral image to obtain a hyperspectral image with low-dimensional principal components;

[0012] Training sample pairs are constructed based on hyperspectral images and lidar images, and a training set is constructed in combination with true labels.

[0013] Furthermore, the step of training a preset feature adaptive mutual-guided multi-source information fusion network based on the training set to obtain a fully trained feature adaptive mutual-guided multi-source information fusion network specifically includes:

[0014] Based on the spectral-spatial feature learning module, the Involution operator feature learning operation is performed on the hyperspectral image, the spatial structure of the hyperspectral image and the lidar image is fused, and the spectral feature information and spatial feature information are extracted;

[0015] Based on the spectrum-space mutual guidance module, multi-scale feature extraction is performed on the spectral feature information and spatial feature information to obtain low-level features, middle-level features and high-level features;

[0016] Based on the multimodal data fusion classification module, the low-level features, middle-level features and high-level features are classified by weighted probability calculation to obtain the training classification results;

[0017] The network parameters are adjusted according to the training classification results and the true labels to obtain a fully trained feature-adaptive mutually guided multi-source information fusion network.

[0018] Furthermore, the spectral-spatial feature learning module includes a spectral feature extractor, a spectral weight encoder and an adaptive spatial weight encoder. The step of performing an Involution operator feature learning operation on the hyperspectral image based on the spectral-spatial feature learning module, fusing the spatial structure of the hyperspectral image and the lidar image, and extracting spectral feature information and spatial feature information specifically includes:

[0019] The hyperspectral image is processed based on the spectral feature extractor to obtain the main features;

[0020] The spectral weight encoder uses the Involution method to generate the convolution kernel and combines the main features to obtain the spectral feature information;

[0021] The adaptive spatial weight encoder performs adaptive spatial fusion of the spatial structures of hyperspectral images and lidar images according to preset weights, and obtains spatial feature information by combining the main features.

[0022] Further, the step of performing multi-scale feature extraction on the spectral feature information and the spatial feature information based on the spectral-spatial mutual guidance module to obtain low-level features, middle-level features and high-level features specifically includes:

[0023] The spectral feature information and the spatial feature information are subjected to tensor concatenation operations to obtain low-level local texture features;

[0024] Based on the first SSM unit, the spectral feature information and spatial features are integrated and extracted to obtain the middle-level interpretable features;

[0025] Based on the second SSM unit, the middle-level interpretable features are re-extracted to obtain high-level global semantic features.

[0026] Furthermore, the multimodal data fusion classification module classifies the low-level features, the middle-level features and the high-level features according to the weighted probability calculation to obtain the training classification result, which specifically includes:

[0027] Based on the dimension matching function, the feature map dimensions of low-level local texture features, middle-level interpretable features and high-level global semantic features are matched, and fused by dimension to obtain a fused feature tensor;

[0028] Convert the fused features into final fused features according to preset weights;

[0029] The final fusion features are input into the multimodal data fusion classification module to obtain the training classification results.

[0030] Furthermore, the loss calculation formula for adjusting the network parameters according to the training classification results and the true labels is as follows:

[0031]

[0032] In the above formula, one_hot(·) represents the thermal probability constructor, represents element-wise tensor multiplication operation, Represents the training classification result, y i represents the true label, n + =0.9, n - =0.1, μ represents a hyperparameter.

[0033] The second technical solution adopted by the present invention is: a feature adaptive mutual guidance multi-source information fusion classification system, comprising:

[0034] The preprocessing part is used to obtain the hyperspectral image and lidar image data sets and perform data preprocessing to obtain the training set;

[0035] In the network training part, a preset feature adaptive mutual guidance multi-source information fusion network is trained based on a training set to obtain a fully trained feature adaptive mutual guidance multi-source information fusion network, wherein the feature adaptive mutual guidance multi-source information fusion network includes a spectral-spatial feature learning module, a spectral-spatial mutual guidance module and a multimodal data fusion classification module;

[0036] The task classification part is used to obtain the image to be tested and classify the image to be tested based on a fully trained feature adaptive mutual guided multi-source information fusion network to obtain the image classification result.

[0037] The beneficial effects of the method and system of the present invention are: the present invention adopts a new type of Involution operator to explore the channel correlation between spectral channels, collaboratively explores spatial information with the features of multi-modal input, and better solves and characterizes the interactive transmission of feature information through a learning process guided by multi-scale and adaptive features. In addition, the multi-source feature information fusion classification method proposed in the present invention is a fast and efficient multi-source image classification method, which can be well applied to the practical application or development process of different types of data fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flow chart of the steps of a feature adaptive mutual guided multi-source information fusion classification method of the present invention;

[0039] Figure 2 It is a framework diagram of a multi-source information fusion network featuring adaptive mutual guidance according to a specific embodiment of the present invention;

[0040] Figure 3 is a schematic diagram comparing the Convolution operator and the Involution operator in a specific embodiment of the present invention;

[0041] Figure 4 It is a structural block diagram of a feature adaptive mutual-guiding multi-source information fusion classification system of the present invention. DETAILED DESCRIPTION

[0042] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0043] The present invention provides a solution for accurate classification of multimodal data by constructing a multi-source information fusion classification framework with feature adaptive mutual guided learning.

[0044] Reference Figure 1 The present invention provides a feature adaptive mutual guidance multi-source information fusion classification method, the method comprising the following steps:

[0045] S1, obtain hyperspectral image and lidar image data sets and perform data preprocessing to obtain a training set;

[0046] S2, based on the training set, the preset feature adaptive mutual guidance multi-source information fusion network is trained to obtain a fully trained feature adaptive mutual guidance multi-source information fusion network (AM 3 Net);

[0047] The feature adaptive mutual guidance multi-source information fusion network includes a spectrum-space feature learning module, a spectrum-space mutual guidance module and a multimodal data fusion classification module;

[0048] Specifically, the framework diagram of the feature adaptive mutual guidance multi-source information fusion network is shown in Figure 2 .

[0049] S3, obtain the image to be tested and use the multi-source information fusion network (AM) based on the fully trained feature adaptive mutual guidance 3 Net) to classify the image to be tested and obtain the image classification result.

[0050] As a further preferred embodiment of the method, the step of obtaining a hyperspectral image and a lidar image data set and performing data preprocessing to obtain a training set specifically includes:

[0051] S11. Obtain hyperspectral images covering the same area on the Earth's surface and its corresponding lidar image

[0052] Specifically, M and N represent the width and height of the two images, respectively, and D represents the number of original hyperspectral channels.

[0053] S12, reducing the dimension of the hyperspectral image to obtain a hyperspectral image with low-dimensional principal components;

[0054] S13. Construct training sample pairs based on hyperspectral images and lidar images, and construct a training set based on true labels.

[0055] Specifically, after the standard dimensionality reduction operation is performed on the input hyperspectral image, an image training block with a hyperspectral channel number of d is obtained. For hyperspectral and lidar training sample data pairs T represents the total number of training samples, and From the total sample X hsi and X lid The i-th hyperspectral and lidar sample block (block size is m×n) selected from is the true value label corresponding to X, y i ∈{1,2,…,C}, C represents the number of true value labels.

[0056] As a further preferred embodiment of the method, the step of training a preset feature adaptive mutual-guided multi-source information fusion network based on a training set to obtain a fully trained feature adaptive mutual-guided multi-source information fusion network specifically includes:

[0057] S21, based on the spectral-spatial feature learning module, the Involution operator feature learning operation is performed on the hyperspectral image, the spatial structure of the hyperspectral image and the lidar image is fused, and the spectral feature information and the spatial feature information are extracted;

[0058] S22, performing multi-scale feature extraction on the spectral feature information and the spatial feature information based on the spectral-spatial mutual guidance module to obtain low-level features, middle-level features and high-level features;

[0059] S23, based on the multimodal data fusion classification module, the low-level features, the middle-level features and the high-level features are classified according to the weighted probability calculation to obtain the training classification result;

[0060] S24. Adjust the network parameters according to the training classification results and the true labels to obtain a fully trained feature-adaptive mutually guided multi-source information fusion network.

[0061] As a further preferred embodiment of the method, the spectral-spatial feature learning module includes a spectral feature extractor F hsi , spectral weight encoder and adaptive spatial weight encoder The step of performing an Involution operator feature learning operation on the hyperspectral image based on the spectral-spatial feature learning module, fusing the spatial structure of the hyperspectral image and the lidar image, and extracting spectral feature information and spatial feature information specifically includes:

[0062] Based on spectral feature extractor F hsi Process the hyperspectral image to obtain the main features;

[0063] Specifically, the spectral feature extractor F hsi Includes three hyperspectral data X hsiThe convolutional layers for extracting spectral spatial features include 16, 32, and 64 filters, respectively, and all convolutions include zero padding and batch normalization (BN) operations. hsi Finally, a training block of size m×n×64 is output, and then the main features are multiplied by the spectral weight filter and spatial weight filter To obtain spectral information and spatial information respectively.

[0064] The spectral weight encoder uses the Involution method to generate the convolution kernel and combines the main features to obtain the spectral feature information;

[0065] Hyperspectral images contain rich spectral band information and have the key ability to represent and describe pixel features. Machine learning-based methods usually use dimensionality reduction and band selection. Existing deep learning methods often use 1×1 convolutional layers or self-attention mechanisms to unify the number of channels for each feature. However, the above methods have limited ability to characterize high-dimensional spectral channel features. In contrast, the Involution convolution kernel has the characteristics of independent spatial features and shared channel features, which can effectively ensure the adaptive allocation of corresponding channel weights to different spatial positions. Therefore, the spectral weight encoder design uses the Involution algorithm to extract spectral features. Given a pixel Its involution weight operator is expressed as Shared on channels. G represents the number of groups that share the same convolution kernel. For each group, there are k filters,

[0066]

[0067] In the above formula, represents the kernel generating function. Then use Multiply by feature F hsi The spectral characteristics obtained are as follows:

[0068]

[0069] In the above formula, Represents an element-wise tensor multiplication operation.

[0070] The present invention uses the Involution operator to replace the 1×1 convolution of the traditional deep convolutional neural network Convolution, which can fully consider the dimensional characteristics between different pixel spectral channels, thereby obtaining more representative spectral feature information. For comparison, refer to the schematic diagram Figure 3 .

[0071] The adaptive spatial weight encoder performs adaptive spatial fusion of the spatial structures of hyperspectral images and lidar images according to preset weights, and obtains spatial feature information by combining the main features.

[0072] The present invention uses the spatial information of the hyperspectral image and the depth and elevation information of the radar image to adaptively obtain the spatial information of the ground object. The traditional convolution method is used to obtain the initial features. Assume that the kth filter kernels of the hyperspectral and lidar data are Where p×p is the size of each filter kernel. Then, the adaptive spatial weight fusion layer is used to learn the fusion feature weights, and the formula is as follows:

[0073]

[0074] In the above formula, α1 and α2 represent the different spatial weights of hyperspectral and lidar images, respectively, which can be adaptively adjusted according to the loss function in the learning process. represents the weight of the hyperspectral and lidar data for the kth filter, represents the global weight of the fusion of hyperspectral and lidar data, Finally, the spectral spatial feature F hsi Multiply by the spatial feature weight Get the final space-level features as follows:

[0075]

[0076] Different from the traditional mode of independently acquiring spatial information of multi-source data, the present invention adaptively learns the spatial information contained in the hyperspectral spectrum and the depth and elevation information of the radar image, and can adaptively and jointly acquire the spatial feature information of the ground target.

[0077] As a further preferred embodiment of the method, the step of performing multi-scale feature extraction on the spectral feature information and the spatial feature information based on the spectral-spatial mutual guidance module to obtain low-level features, middle-level features and high-level features includes:

[0078] The spectral feature information and the spatial feature information are subjected to tensor concatenation operations to obtain low-level local texture features;

[0079] Specifically, the low-level features F L From the spectral information and spatial information The series connection is produced; it is recorded as:

[0080]

[0081] In the above formula, Represents a tensor concatenation operation along the channels axis.

[0082] Based on the first SSM unit, the spectral feature information and spatial features are integrated and extracted to obtain the middle-level interpretable features;

[0083] Specifically, using F spe and F spa As the input of the first SSM unit to generate the middle-level features F M The SSM module proposed in the invention can effectively share and integrate spectral and spatial information. The SSM module uses 96 and 128 kernel convolution layers to extract high-level spectral features respectively. and spatial characteristics Then, multimodal hierarchical feature pairs (e.g. and ) performs SSM operation to strengthen the feature representation mechanism. Finally, a max-pooling layer follows the feature pair layer in order to reduce the computational complexity of the model and expand the receptive field information. M It is expressed as follows:

[0084]

[0085] In the above formula, SSM(·) represents the SSM operation, pooling(·) represents the average pooling operation, and Represent the middle-level features F M spectral and spatial components.

[0086] Based on the second SSM unit, the middle-level interpretable features are re-extracted to obtain high-level global semantic features.

[0087] Similarly, using F' spe and F' spa As the input of the second SSM unit to generate high-level features F H , where F' spe Indicates the mid-level features of the spectrum, F' spa In order to extract more detailed information from the input features, 192 and 256 kernel convolution layers are used to extract features again, and the obtained and As the output result, the SSM unit uses multimodal hierarchical features to and ) to calculate the high-level features F H ,as follows:

[0088]

[0089] In the above formula, and Represent the high-level semantic features F H spectral and spatial components.

[0090] The present invention uses two SSM units to learn spectral features from hyperspectral data and spatial features from hyperspectral and lidar images in a deep and cross-learning manner.

[0091] As a further preferred embodiment of the method, the step of calculating the probability of classifying low-level features, middle-level features and high-level features according to weights based on the multimodal data fusion classification module to obtain a training classification result specifically includes:

[0092] Based on the dimension matching function, the feature map dimensions of low-level local texture features, middle-level interpretable features and high-level global semantic features are matched, and fused by dimension to obtain a fused feature tensor;

[0093] Specifically, to ensure that the features have the same number of feature map dimensions, the dimension matching function g(·) is used to match the dimensions of the input multi-level semantic features. The global network fusion feature tensor O is expressed as follows:

[0094] O=[(g1(F L ),g2(F M ),g3(F H )); λ]

[0095] In the above formula, g1(·), g2(·) and g3(·) are F L , F M and F H The dimension matching function of Produced by a convolution operation (kernel size is 1×1, stride is 4, number of filters is 512), Produced by convolution operation (kernel size is 1×1, step size is 2, number of filters is 512), Produced by a convolution operation with kernel size 1×1, stride 1, and number of filters 512.

[0096] Convert the fused features into final fused features according to preset weights;

[0097] The obtained global network fusion feature tensor O is transformed into:

[0098]

[0099] In the above formula, λ=[λ1,λ2,λ3]represents F L , F M and F HSimilar to α, the invention adopts λ to adaptively adjust the composition of the final fusion features.

[0100] The final fusion features are input into the multimodal data fusion classification module to obtain the training classification results.

[0101] For the i-th sample, the final fusion feature O i It is fed into the multimodal data classification module to classify the class label of the input pixel. After a global average pooling and a fully connected (FC) layer, O i is converted into a vector of size (1, C), where C represents the total number of true value labels. Then, the softmax function is used to convert the processed vector into a probability distribution to obtain the predicted vector Finally, based on the prediction vector to classify.

[0102] As a further preferred embodiment of the method, the loss function of adjusting the network parameters according to the training classification results and the true labels is The calculation formula is as follows:

[0103]

[0104] In the above formula, one_hot(·) represents the thermal probability constructor, represents element-wise tensor multiplication operation, Represents the training classification result, y i represents the true label, n + =0.9, n - =0.1, μ is a hyperparameter used to control the network back propagation process.

[0105] like Figure 4 As shown, a feature adaptive mutual guidance multi-source information fusion classification system comprises:

[0106] The preprocessing part is used to obtain the hyperspectral image and lidar image data sets and perform data preprocessing to obtain the training set;

[0107] In the network training part, a preset feature adaptive mutual guidance multi-source information fusion network is trained based on a training set to obtain a fully trained feature adaptive mutual guidance multi-source information fusion network, wherein the feature adaptive mutual guidance multi-source information fusion network includes a spectral-spatial feature learning module, a spectral-spatial mutual guidance module and a multimodal data fusion classification module;

[0108] The task classification part is used to obtain the image to be tested and classify the image to be tested based on a fully trained feature adaptive mutual guided multi-source information fusion network to obtain the image classification result.

[0109] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0110] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A feature-adaptive mutual-guided multi-source information fusion classification method, characterized in that: The following steps are involved: Obtain hyperspectral image and lidar image data sets and perform data preprocessing to obtain training sets; Based on the training set, a preset feature adaptive mutual guided multi-source information fusion network is trained to obtain a fully trained feature adaptive mutual guided multi-source information fusion network; The feature adaptive mutual guidance multi-source information fusion network includes a spectrum-space feature learning module, a spectrum-space mutual guidance module and a multimodal data fusion classification module; Obtain the image to be tested and classify it based on the fully trained feature adaptive mutual guided multi-source information fusion network to obtain the image classification result; The step of training a preset feature adaptive mutual-guided multi-source information fusion network based on a training set to obtain a fully trained feature adaptive mutual-guided multi-source information fusion network specifically includes: Based on the spectral-spatial feature learning module, the Involution operator feature learning operation is performed on the hyperspectral image, the spatial structure of the hyperspectral image and the lidar image is fused, and the spectral feature information and spatial feature information are extracted; Based on the spectrum-space mutual guidance module, multi-scale feature extraction is performed on the spectral feature information and spatial feature information to obtain low-level features, middle-level features and high-level features; Based on the multimodal data fusion classification module, the low-level features, middle-level features and high-level features are classified by weighted probability calculation to obtain the training classification results; The network parameters are adjusted according to the training classification results and the true labels to obtain a fully trained feature-adaptive mutually guided multi-source information fusion network.

2. According to the feature adaptive mutual guidance multi-source information fusion classification method of claim 1, it is characterized by: The step of obtaining a hyperspectral image and a lidar image data set and performing data preprocessing to obtain a training set specifically includes: Obtain a hyperspectral image and its corresponding lidar image covering the same area of ​​the Earth’s surface; Reduce the dimension of the hyperspectral image to obtain a hyperspectral image with low-dimensional principal components; Training sample pairs are constructed based on hyperspectral images and lidar images, and a training set is constructed in combination with true labels.

3. According to the feature adaptive mutual guidance multi-source information fusion classification method of claim 2, it is characterized by: The spectral-spatial feature learning module includes a spectral feature extractor, a spectral weight encoder and an adaptive spatial weight encoder. The step of performing an Involution operator feature learning operation on the hyperspectral image based on the spectral-spatial feature learning module, fusing the spatial structure of the hyperspectral image and the lidar image, and extracting spectral feature information and spatial feature information specifically includes: The hyperspectral image is processed based on the spectral feature extractor to obtain the main features; The spectral weight encoder uses the Involution method to generate the convolution kernel and combines the main features to obtain the spectral feature information; The adaptive spatial weight encoder performs adaptive spatial fusion of the spatial structures of hyperspectral images and lidar images according to preset weights, and obtains spatial feature information by combining the main features.

4. According to claim 3, a feature-adaptive mutual-guided multi-source information fusion classification method is characterized in that: The step of performing multi-scale feature extraction on spectral feature information and spatial feature information based on the spectral-spatial mutual guidance module to obtain low-level features, middle-level features and high-level features specifically includes: The spectral feature information and the spatial feature information are subjected to tensor concatenation operations to obtain low-level local texture features; Based on the first SSM unit, the spectral feature information and spatial features are integrated and extracted to obtain the middle-level interpretable features; Based on the second SSM unit, the middle-level interpretable features are re-extracted to obtain high-level global semantic features.

5. According to the feature adaptive mutual guidance multi-source information fusion classification method of claim 4, it is characterized in that: The step of calculating the probability of classifying low-level features, middle-level features and high-level features according to weights based on the multimodal data fusion classification module to obtain the training classification result specifically includes: Based on the dimension matching function, the feature map dimensions of low-level local texture features, middle-level interpretable features and high-level global semantic features are matched, and fused by dimension to obtain a fused feature tensor; Convert the fused features into final fused features according to preset weights; The final fusion features are input into the multimodal data fusion classification module to obtain the training classification results.

6. According to claim 5, a feature-adaptive mutual-guided multi-source information fusion classification method is characterized in that: The loss calculation formula for adjusting network parameters based on the training classification results and the true labels is as follows: In the above formula, one_hot(·) represents the thermal probability constructor, represents element-wise tensor multiplication operation, Represents the training classification result, y i represents the true label, n + =0.9, n - =0.1, μ represents the hyperparameter, and max(·) is used to find the maximum value.

7. A feature-adaptive mutual-guided multi-source information fusion classification system, characterized in that: The method for performing the feature adaptive mutual guidance multi-source information fusion classification method as claimed in claim 1 comprises: The preprocessing part is used to obtain the hyperspectral image and lidar image data sets and perform data preprocessing to obtain the training set; In the network training part, a preset feature adaptive mutual guidance multi-source information fusion network is trained based on a training set to obtain a fully trained feature adaptive mutual guidance multi-source information fusion network, wherein the feature adaptive mutual guidance multi-source information fusion network includes a spectral-spatial feature learning module, a spectral-spatial mutual guidance module and a multimodal data fusion classification module; The task classification part is used to obtain the image to be tested and classify the image to be tested based on a fully trained feature adaptive mutual guided multi-source information fusion network to obtain the image classification result.

Images