Hyperspectral image and LiDAR data semantic synergy joint classification method

Through deep learning technology and language model, the geographic category descriptor text is generated, combined with hyperspectral images and LiDAR data, and the problem of insufficient deep feature relationship and feature interpretation in traditional methods is solved, achieving efficient multi-source data fusion and geographic data Item classification improves classification accuracy and interpretability.

CN119027727APending Publication Date: 2024-11-26HARBIN UNIV OF SCI & TECH
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411061275.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-03
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the field of remote sensing geographic classification, traditional methods ignore deep feature relationships and feature interpretability, and it is difficult to effectively utilize the heterogeneous features of hyperspectral images and LiDAR data, resulting in insufficient classification accuracy and interpretability.

Method used

Deep learning technology is adopted, combined with convolutional neural networks and Transformer models, and detailed geographic category descriptor text is generated through large language models, and advanced abstract representation is jointly constructed to realize multi-dimensional representation and joint classification of hyperspectral images and LiDAR data.

Benefits of technology

It significantly improves the accuracy of geographic classification and model interpretation capabilities, realizes deep fusion and information complementarity of multi-source heterogeneous data, and enhances the understanding and classification performance of geographic essential characteristics.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027727A_ABST
    Figure CN119027727A_ABST
Patent Text Reader

Abstract

The invention relates to a hyper-spectral image (HSI) and LiDAR data semantic collaboration joint classification method, and belongs to the technical field of image classification. The method sequentially comprises the following steps of: a, inputting a to-be-classified HSI image and LiDAR data; b, a class descriptor text (Text) is constructed by providing an instruction to the large language model; c, performing patch extraction on the HSI and the LiDAR to form a training sample set, a verification sample set and a test sample set; step d, inputting the HSI data and the LiDAR data into a ModaUnion encoder, and extracting common characteristics; e, inputting the HSI, the LiDAR common characteristics and the Text into a hybrid expert encoder to obtain characteristic representation; and f, inputting the three feature representations into a joint classifier based on integrated HSI and Text comparative learning and HSI and LiDAR comparative learning, and carrying out joint classification training to obtain a collaborative classification result. According to the method, HSI, LiDAR and Text representation information is fully mined, more comprehensive and fine multi-dimensional representation for ground feature attributes is constructed in a collaborative manner, and an excellent collaborative classification effect is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention discloses a joint classification method of hyperspectral image and LiDAR data semantic collaboration, belonging to the technical field of image classification. Background Art

[0002] Hyperspectral images (HSI) are composed of hundreds of continuous bands of spectra in the same area. They have the unique advantages of high spectral resolution and unified spectra. Their rich spectral information can be used to identify the composition and internal structure of different objects. LiDAR provides high-precision three-dimensional structural information, but is relatively limited in the expression of spectral features. Collaborative classification of hyperspectral images and LiDAR data can give full play to the advantages of both, achieve information complementarity, and realize more accurate object classification.

[0003] In the field of remote sensing object classification, traditional methods based on category names often ignore the potential deep-level feature relationships and feature interpretability. With the advancement of deep learning and natural language processing technology, language, as a high-level abstract tool, can better simulate the human cognitive process of objects. Therefore, it is envisioned to use language models as a bridge between hyperspectral image features and object categories to enhance the classification ability and interpretability of the model by automatically generating detailed feature descriptions. This method not only avoids the time-consuming and scalability problems of manually defined descriptions, but also utilizes the knowledge reserve of large language models to generate detailed descriptions rich in remote sensing expertise for each object category, thereby improving the accuracy of classification and the interpretability of the model.

[0004] Deep learning provides an efficient path for the collaborative classification of remote sensing multimodal data. Compared with traditional manual feature extraction methods, end-to-end deep learning architectures can autonomously learn the feature expressions of images and other data, and effectively explore the intrinsic relationships between different data sources. Convolutional neural networks (CNNs) learn high-level semantic information of images through multi-level feature extraction, while Transformers use self-attention mechanisms to capture long-distance dependencies between data. The combination of these two models can maximize their respective advantages, effectively solve the problem of feature expression of heterogeneous data, achieve deep fusion and information complementarity of multi-source data, and significantly improve the accuracy of ground object classification. Summary of the invention

[0005] This invention fully exploits the high-level abstract representation information of the description of the object category by HSI, LiDAR and language model, and collaboratively constructs a more comprehensive and detailed multi-dimensional representation of the object attributes. This innovative method profoundly promotes the understanding and insight into the essential characteristics of the object, achieves a more outstanding collaborative classification effect, and has important scientific and practical value for improving the expression ability and classification performance of multi-source heterogeneous data.

[0006] The object of the present invention is achieved in that:

[0007] A joint classification method for semantic collaboration of hyperspectral image and LiDAR data includes the following steps:

[0008] Step a, inputting the hyperspectral image (HSI) and LiDAR data to be classified;

[0009] Step b, constructing a hyperspectral category descriptor text set (Text) by giving instructions to a large language model;

[0010] Step c: extract patches from HSI and LiDAR data to form their own training, verification and test samples;

[0011] Step d: Input the patch data of HSI and LiDAR into the ModaUnion encoder composed of InceptionNeXt, and extract multimodal common features through parameter sharing mechanism;

[0012] Step e: Input the common features of HSI and LiDAR, and the Text obtained in step b, into the mixed expert encoder (MoE-EN) module to obtain the feature representation of HSI, LiDAR, and Text;

[0013] Step f: Input the feature representations of HSI, LiDAR, and Text into a joint classifier based on integrated HSI and Text comparative learning and HSI and LiDAR comparative learning, perform joint classification training, and obtain a collaborative classification result.

[0014] The above-mentioned joint classification method for alignment and fusion of hyperspectral image and LiDAR data features is characterized in that step b specifically sends instructions to a large language model to generate detailed description Texts for different types of land objects, and these descriptor text sets cover rich semantic information.

[0015] The above-mentioned joint classification method for alignment and fusion of hyperspectral image and LiDAR data features is characterized in that step c specifically comprises performing patch extraction on training and test samples with a single window size of 11×11.

[0016] The above-mentioned joint classification method of hyperspectral image and LiDAR data semantic collaboration, step d specifically includes the following steps:

[0017] Step d1, perform online data enhancement on HSI training samples, the enhancement methods are: min-max normalization, random noise, multi-scale transformation, random horizontal mirroring;

[0018] Step d2: Perform online data enhancement on the LiDAR training samples by using the following enhancement methods: min-max normalization, random noise addition, multi-scale transformation, and random horizontal mirroring;

[0019] Step d3: Input the HSI and LiDAR enhanced training samples into the ModaUnion encoder, and obtain the common features between different modalities through the parameter sharing mechanism.

[0020] The above-mentioned joint classification method of hyperspectral image and LiDAR data semantic collaboration, step e specifically includes the following steps:

[0021] Step e1, input the common features into the HSI encoder to obtain HSI encoding vectors respectively;

[0022] Step e2, input the common features into the LiDAR encoder to obtain LiDAR encoding vectors respectively;

[0023] Step e3: Input the Text into the Text encoder in the hybrid expert encoder to obtain the Text encoding vectors respectively.

[0024] The above-mentioned joint classification method of hyperspectral image and LiDAR data semantic collaboration, step f specifically includes the following steps:

[0025] Step f1, input the HSI code vector and the LiDAR code vector into the comparative learning branch, perform comparative learning of the two domain features, and obtain the HSI and LiDAR comparative learning loss function;

[0026] Step f2: Input the HSI encoding vector and the Text encoding vector into the comparative learning branch, perform comparative learning of the two domain features, and obtain the HSI and Text comparative learning loss function;

[0027] Step f3, the HSI encoding vector, LiDAR encoding vector and Text encoding vector are feature fused to form an HSI-LiDAR-Text encoder, and the HSI and LiDAR comparison learning loss function, HSI and Text comparison loss function, and HSI-LiDAR classification loss function are combined to perform classification and obtain the final object classification result.

[0028] Beneficial effects:

[0029] The invention discloses a joint classification method for semantic collaboration of hyperspectral image and LiDAR data, which sequentially performs the following steps: step a, inputting hyperspectral image (HSI) and LiDAR (Light Detection and Ranging, LiDAR) data to be classified; step b, constructing a hyperspectral category descriptor text set (Text) by proposing instructions to a large language model; step c, performing patch extraction on HSI and LiDAR data to form respective training, verification and test sample sets; step d, inputting the patch data of HSI and LiDAR into a ModaUnion encoder composed of InceptionNeXt, and extracting multimodal common features through a parameter sharing mechanism; step e, inputting the common features of HSI and LiDAR, and the Text obtained in step b, into a mixed expert encoder (MoE-EN) module to obtain feature representations of HSI, LiDAR and Text; step f, inputting the feature representations of HSI, LiDAR and Text into a joint classifier based on integrated HSI and Text contrast learning and HSI and LiDAR contrast learning, performing joint classification training, and obtaining a collaborative classification result. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of a joint classification method for aligning and fusing hyperspectral images and LiDAR data features according to the present invention.

[0031] Figure 2 It is a schematic diagram of the network principle of a joint classification method for aligning and fusing hyperspectral images and LiDAR data features in the method of the present invention.

[0032] Figure 3 It is a schematic diagram of the principle of constructing a hyperspectral category descriptor in the method of the present invention.

[0033] Figure 4 It is a schematic diagram of the principle of the ModaUnion encoder in the method of the present invention.

[0034] Figure 5 It is a schematic diagram of the principle of the HSI encoder in the MoE-EN in the method of the present invention.

[0035] Figure 6 It is a schematic diagram of the principle of the LiDAR encoder in the MoE-EN in the method of the present invention.

[0036] Figure 7 This is a classification result diagram of the method used in the Houston 2013 data set in the method of the present invention.

[0037] Figure 8It is a classification result diagram of the method adopted by the Trento data set in the method of the present invention.

[0038] Fig. 9 It is a classification result diagram of the method adopted by the MUUFL Gulfport data set in the method of the present invention. DETAILED DESCRIPTION

[0039] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0040] A joint classification method for aligning and fusing hyperspectral images and LiDAR data features in this specific implementation is shown in the flowchart. Figure 1 As shown, the network principle diagram is as follows Figure 2 As shown, the following steps are included:

[0041] Step a: Input the hyperspectral image and LiDAR data to be classified.

[0042] In this specific implementation, two public datasets are used, namely Houston2013, Trento and MUUFL Gulfport datasets.

[0043] (1) Houston 2013 dataset: This dataset contains hyperspectral images and LiDAR-derived DSMs, both of which consist of 349×1905 pixels with a spatial resolution of 2.5 m. The LiDAR DSM data were obtained by the NSF-funded Center for Airborne Laser Mapping (NCALM) at the University of Houston 2013 campus and adjacent urban areas. The HSI was obtained by the AVIRIS sensor over the Indian Pines Agricultural Experiment Station in northwestern Indiana, USA. The original data has a total of 224 bands, 4 zero bands and 20 water absorption bands (104-108, 150-163, 220) are removed, and the remaining 200 bands are used for experimental research, ranging from 0.4 to 2.5 μm, with a spatial size of 145×145 pixels. The HSI has 144 spectral bands in the 380-1050 nm region, including 15 categories.

[0044] (2) Trento dataset: This dataset contains hyperspectral images and LiDAR-derived DSM, both of which consist of 600×166 pixels with a spatial resolution of 1m. The LiDAR DSM data is generated using the first and last point cloud pulses obtained by the Optech ALTM 3100EA sensor, and the HSI data is acquired by the AISA Eagle sensor. The HSI has 63 spectral bands, covering the 402.89-989.09nm region and including 6 categories.

[0045] (3) MUUFL Gulfport Dataset: This dataset was collected in November 2010 using the Reflection Optical System Imaging Spectrometer (ROSIS) sensor over the University of Southern Mississippi campus. It contains a hyperspectral image (HSI) with a size of 325×220 pixels and 72 spectral bands. In addition, the LiDAR image includes elevation data provided by two raster layers, with a total of 64 bands after removing the initial and final 8 bands affected by noise. The dataset covers 11 urban land cover categories, with 53,687 ground truth pixels annotated. The number of pixels used for training is 100 for each category, and the remaining 52,587 pixels are used for testing.

[0046] Step b: The principle of constructing hyperspectral category descriptors in this method is as follows: Figure 3 As shown, by giving instructions to a large language model, a hyperspectral category descriptor text set (Text) is constructed, which specifically describes the spectral features unique to distinguishing the category in the hyperspectral image.

[0047] Step c: perform patch extraction on the HSI image and LiDAR data with a window size of 11×11 to form respective training samples, verification samples, and test samples.

[0048] Step d: Input the HSI and LiDAR patch data into the ModaUnion encoder composed of InceptionNeXt, and extract multimodal common features through the parameter sharing mechanism. Specifically:

[0049] Step d1: Perform online data enhancement on the HSI training samples using the following enhancement methods: min-max normalization, random noise addition, multi-scale transformation, and random horizontal mirroring.

[0050] The specific implementation is:

[0051] Min-max normalization is to perform min-max normalization on the input samples. The formula is: where x i is the i-th sample, is the normalized sample, and after normalization, the sample data is mapped to (0,1).

[0052] Noise addition is to add noise to the input sample: where x i is the i-th sample, μ is the mean, δ is the variance, x i ′ is the sample after adding noise.

[0053] Multi-scale transformation is to perform multi-scale transformation operations on input samples: the training samples are multi-scale transformed with four scale window sizes, and the scale window sizes are 11×11, 9×9, 7×7, and 5×5.

[0054] Horizontal mirroring is to perform a horizontal mirroring operation on the input sample: where x i is the i-th sample, w is the sample image width, x i ′ is the sample after mirroring.

[0055] Step d2: Perform online data enhancement on the LiDAR training samples using the following methods: min-max normalization, random noise addition, multi-scale transformation, and random horizontal mirroring. The enhancement method is consistent with the processing of HSI images.

[0056] Step d3: Input the HSI and LiDAR enhanced training samples into the ModaUnion encoder, and obtain the common features between different modalities through the parameter sharing mechanism;

[0057] ModaUnion encoder details are as follows Figure 4 As shown in the figure, the encoder is based on the InceptionNeXt architecture and realizes the collaborative processing of HSI and LiDAR data through a parameter sharing mechanism. Specifically, the ModaUnion encoder is composed of InceptionNeXt modules, which are inspired by the original Inception architecture and aim to promote the effective extraction of information of different scales and directions by decomposing large convolution kernels into parallel small convolution kernels and orthogonal banded convolution kernels. This architecture can not only efficiently extract the low-frequency common features of the two modal data, but also significantly improve the model calculation efficiency and data processing capabilities through a parallel processing mechanism. In addition, the application of the global response normalization (GRN) technology further optimizes information transmission and realizes the deep integration of heterogeneous data.

[0058] Step e: Input the common features of HSI and LiDAR, as well as the Text obtained in step b, into the Hybrid Expert Encoder (MoE-EN) module to obtain the feature representation of HSI, LiDAR, and Text. Specifically, the following steps are included:

[0059] Step e1, input the common features into the HSI encoder encoder to obtain HSI encoding vectors respectively;

[0060] The specific steps are as follows:

[0061] according to Figure 5The HSI encoder consists of a spectral-spatial perception branch and a global spectral understanding branch. The spectral-spatial perception branch uses the local spatial window multi-headed dual attention (Spatial Window Multi-headed Self-attention, SW-MHSA) mechanism and the channel group multi-headed dual attention (Channel Group Multi-headed Self-attention, CG-MHSA) mechanism, and is combined with the global spectral understanding branch to capture local spectral-spatial features and global spectral semantic information, which comprehensively improves the model's understanding and expression capabilities of HSI features.

[0062] Specifically, the SW-MHSA mechanism divides the input hyperspectral image into multiple local image regions, and then defines a limited spatial window for each region, within which attention weights are calculated and assigned. Under the multi-head architecture, different heads focus on learning diverse local spatial dependency patterns. This feature enables SW-MHSA to model local spatial information in a more efficient way, significantly enhancing the model's ability to capture and learn spatial context information.

[0063] At the same time, the CG-MHSA mechanism subdivides the channel dimension of the input features into multiple independent groups, and performs self-attention calculations within each group to learn and understand the interdependencies between channels within the group. This careful design enables the model to focus on exploring the intrinsic connections between spectral channels, thereby deepening the understanding of the correlation between spectra in HSI.

[0064] In addition, through the global spectral understanding branch, the model is able to learn and analyze the dependencies between spectral features, thereby obtaining the contextual representation of the central pixel in the global spectral perspective, further enriching the grasp of the contextual semantic information between spectral features.

[0065] The HSI encoder enables accurate processing of spatial, spectral global and local correlation information in hyperspectral images. This not only enhances the model's ability to capture spatial features, but also promotes collaborative learning of spectral correlation information, ultimately significantly improving the model's overall expressiveness and classification accuracy.

[0066] Step e2, input the common features into the LiDAR encoder to obtain LiDAR encoding vectors respectively;

[0067] The specific steps are as follows:

[0068] according to Figure 6,The LiDAR encoder utilizes the frequency domain attention mechanism (Spectrum Former). Since LiDAR data has rich elevation information such as building boundaries and vegetation shapes, the encoder fully learns the elevation features of LiDAR data, which can greatly improve the collaborative classification performance.

[0069] Specifically, Spectrum-LiDAR innovatively combines the Transformer encoder architecture with the Fourier transform-enhanced self-attention mechanism to achieve efficient processing of LiDAR data. By mapping the data to the frequency domain and strengthening its high-frequency components, the system can keenly focus on the key terrain features necessary to build overall scene understanding. This process not only significantly enhances the robustness of the model, enabling it to robustly handle data fluctuations caused by perspective differences and environmental changes, but also ensures that the model can produce more accurate and stable representations even under noisy or challenging acquisition conditions. Furthermore, Spectrum-LiDAR uses an inverse Fourier transform to restore the frequency-domain optimized data to the spatial domain. This step lays the foundation for subsequent multimodal feature fusion and ensures that information obtained from different sensors can be effectively integrated.

[0070] Step e3: Input the Text into the Text encoder in the hybrid expert encoder to obtain the Text encoding vectors respectively.

[0071] The specific steps are as follows:

[0072] A text embedding with hyperspectral feature meaning is created, and more feature knowledge is introduced into the label space through a text encoder. The semantic information contained is more detailed and richer than traditional single-hot or multi-hot label vectors. The RemoteCLIP model is able to grasp the subtle connection between remote sensing visual content and professional narrative by combining remote sensing imagery with matching professional text information (such as feature classification and environmental characteristics description marked in satellite images). This process embeds deep remote sensing expertise into the model, greatly improving the ability to recognize complex features and phenomena in remote sensing images, such as accurately distinguishing different vegetation communities, analyzing changes in land use patterns, and accurately identifying water characteristics. Compared with the widely applicable CLIP model, RemoteCLIP demonstrates a higher level of accuracy and performance in specialized classification and recognition tasks of remote sensing images, thanks to its specialized learning and understanding of remote sensing environments.

[0073] Step f: Input the feature representations of HSI, LiDAR and Text into a joint classifier based on integrated HSI and Text contrast learning and HSI and LiDAR contrast learning, perform joint classification training, and obtain collaborative classification results. Specifically, the following steps are included:

[0074] Step f1, input the HSI code vector and the LiDAR code vector into the comparative learning branch, perform comparative learning of the two domain features, and obtain the HSI and LiDAR comparative learning loss function;

[0075] Step f2: Input the HSI encoding vector and the Text encoding vector into the comparative learning branch, perform comparative learning of the two domain features, and obtain the HSI and Text comparative learning loss function;

[0076] Step f3, the HSI encoding vector, LiDAR encoding vector and Text encoding vector are feature fused to form an HSI-LiDAR-Text encoder, and the HSI and LiDAR comparison learning loss function, HSI and Text comparison loss function, and HSI-LiDAR classification loss function are combined to perform classification and obtain the final object classification result.

[0077] The specific steps are as follows:

[0078] The heterogeneity of multi-source data is a key bottleneck that limits the performance improvement of multimodal object classification using hyperspectral and LiDAR data. To solve this problem, this patent introduces an innovative strategy, which is to construct a specially designed joint training loss function system to guide the model to deeply explore and effectively integrate shared features in multi-source data. In general, the optimization framework of this training phase includes three core components: first, the contrastive learning loss function (HLBCL) for HSI and LiDAR, second, the contrastive learning loss function (HTBCL) for HSI and Text, and finally, the cross-entropy loss function for object classification. In summary, the entire optimization objective can be formally expressed as follows:

[0079]

[0080] Where λ is a hyperparameter for balancing the cross entropy loss (CE Loss) and multiple binary contrastive losses (BCL loss). ω is a learnable weight used to summarize the contribution of different modal sources to the classification of objects. N represents the number of samples, and K represents the total number of categories. ik Indicates whether the i-th sample belongs to the K-th ground feature category. p(x ik ) represents the probability that the model predicts that the i-th sample belongs to the K-th type of ground object.

[0081] The contrast loss aims to reduce the distance between samples of the same type of feature in the feature space, while increasing the distance between samples of different types of features, so as to strengthen the model's deep understanding and accurate distinction of the intrinsic properties of cross-modal data. Specifically, for N pairs of instances of two different modalities, two encoders are used to normalize the i-th pair of instances by L2 norm to obtain the feature embedding x i and t i The specific formula of BCL can be expressed as:

[0082]

[0083] where sim(·) represents the feature embedding x i and t i At the same time, a temperature parameter τ is introduced, which fine-tunes the model's attention to "difficult" negative samples and directly affects the balance between training efficiency and recognition performance.

[0084] Tables 1, 2 and 3 show the detailed classification results of the proposed method and other comparative methods on three datasets: Houston 2013, Trento and MUUFL Gulfport. The comparative methods include: Two-Branch, which uses a two-branch convolutional network for object classification; EndNet, which uses a deep encoder-decoder network to fuse multimodal features and classify objects; MDL-Middle, which uses the middle layer of multi-source features to perform feature fusion network for image classification; MAHiDFNet, which uses a multi-attention hierarchical dense fusion network for object classification; FusAtNet, which combines self-attention and cross-attention mechanisms to extract spatial-spectral features from HSI and LiDAR data for object classification; CALC, a classification method based on coupled adversarial learning for object classification; SepG-ResNet50, which proposes a SepDGConv module to automatically learn the grouped convolution structure in a single-stream neural network to achieve object classification based on the deep single-stream CNN model of this module; and the network proposed in this patent. It is obvious that the classification performance of the method of the present invention is better than that of all the comparison methods, and its comprehensive classification indicators on the three data sets have reached the highest level.

[0085] In order to subjectively evaluate the classification effect, Figure 7 , Figure 8 and Fig. 9 The classification diagrams of the present invention and various methods in Houston2013, Trento and MUUFL Gulfport datasets are shown respectively. It can be seen that the classification results obtained by the method of the present invention are closer to the actual distribution of objects, and the area of ​​misclassification is greatly reduced, which further proves the effectiveness of the invention in the joint classification of hyperspectral and LiDAR.

[0086] Table 1 Comparison of classification accuracy of different classification methods for Houston 2013 dataset (%)

[0087]

[0088] Table 2 Comparison of classification accuracy of different classification methods on Trento dataset (%)

[0089]

[0090] Table 2 Comparison of classification accuracy of different classification methods for MUUFL Gulfport dataset (%)

[0091]

[0092]

Claims

1. A joint classification method for hyperspectral image and LiDAR data semantic collaboration, characterized in that: The following steps are involved: Step a, inputting the hyperspectral image (HSI) and LiDAR data to be classified; Step b, constructing a hyperspectral category descriptor text set (Text) by giving instructions to a large language model; Step c: extract patches from HSI and LiDAR data to form their own training, verification and test samples; Step d: Input the patch data of HSI and LiDAR into the ModaUnion encoder composed of InceptionNeXt, and extract multimodal common features through parameter sharing mechanism; Step e: Input the common features of HSI and LiDAR, as well as the Text obtained in step b, into the mixed expert encoder (MoE–EN) module to obtain the feature representation of HSI, LiDAR, and Text; Step f: Input the feature representations of HSI, LiDAR, and Text into a joint classifier integrating HSI and Text comparative learning and HSI and LiDAR comparative learning, perform classification training, and obtain a collaborative classification result.

2. According to the joint classification method of hyperspectral image and LiDAR data semantic collaboration described in claim 1, step b is specifically to send instructions to a large language model to generate detailed description Text for different ground object categories, and these descriptor text sets cover rich semantic information.

3. According to the joint classification method of hyperspectral image and LiDAR data semantic collaboration described in claim 1, step c is specifically, by performing patch extraction on training and test samples, patch extraction is performed on training and test samples with a single window size, and the window size is 11×11.

4. According to the method for joint classification of hyperspectral image and LiDAR data semantic collaboration as described in claim 1, step d specifically comprises the following steps: Step d1, perform online data enhancement on HSI training samples, the enhancement methods are: min-max normalization, random noise, multi-scale transformation, random horizontal mirroring; Step d2: Perform online data enhancement on the LiDAR training samples by using the following enhancement methods: min-max normalization, random noise addition, multi-scale transformation, and random horizontal mirroring; Step d3: Input the HSI and LiDAR enhanced training samples into the ModaUnion encoder, and obtain the common features between different modalities through the parameter sharing mechanism.

5. According to the method for joint classification of hyperspectral image and LiDAR data semantic collaboration as described in claim 1, step e specifically comprises the following steps: Step e1, input the common features into the HSI encoder in the hybrid expert encoder to obtain HSI encoding vectors respectively; Step e2, inputting the common features into the LiDAR encoder in the hybrid expert encoder to obtain LiDAR encoding vectors respectively; Step e3: Input the Text into the Text encoder in the hybrid expert encoder to obtain the Text encoding vectors respectively.

6. According to the method for joint classification of hyperspectral image and LiDAR data semantic collaboration as described in claim 1, step f specifically comprises the following steps: Step f1, input the HSI code vector and the LiDAR code vector into the comparative learning branch, perform comparative learning of the two domain features, and obtain the HSI and LiDAR comparative learning loss function; Step f2: Input the HSI encoding vector and the Text encoding vector into the comparative learning branch, perform comparative learning of the two domain features, and obtain the HSI and Text comparative learning loss function; Step f3, the HSI encoding vector, LiDAR encoding vector and Text encoding vector are feature fused to form an HSI-LiDAR-Text encoder, and the HSI and LiDAR comparison learning loss function, HSI and Text comparison loss function, and HSI-LiDAR classification loss function are combined to perform classification and obtain the final object classification result.

Citation Information

Cited By

  • Mineral oxygen element analysis method based on multi-mode LIBS

    CN120084777A