Hyperspectral-laser radar image classification method based on coarse and fine granularity fusion network

Through coarse and fine-grained fusion network and adaptive loss fusion technology, the problems of insufficient feature mining and difficult information balance in traditional methods are solved, and efficient classification of hyperspectral and lidar images is achieved, and classification accuracy and performance are improved.

CN118968113APending Publication Date: 2024-11-15QIQIHAR UNIVERSITY
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202410784826.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-18
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The traditional hyperspectral image and lidar image fusion classification methods rely on hand-designed features. Deep feature mining is insufficient, and it is impossible to effectively fit the complex nonlinear relationship between land features in HSI and LiDAR data, and it is difficult to balance spectrum-spatial information. and the contribution of elevation information limits classification accuracy.

Method used

Using image classification method based on coarse and fine particle fusion network, through coarse and fine mutual learning strategy and adaptive loss fusion, combining Transformer model and multi-head cross attention residual block, HSI and LiDAR features are extracted, local and global information are balanced, and the feature is achieved. Excellent integration.

Benefits of technology

It improves feature expression capabilities, reduces calculation costs, realizes efficient fusion of HSI and LiDAR data, improves classification accuracy and performance, and verifies the effectiveness of the method through experiments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968113A_ABST
    Figure CN118968113A_ABST
Patent Text Reader

Abstract

The invention discloses a hyperspectral-laser radar image classification method based on a coarse and fine granularity fusion network. The method comprises the following steps: step 1, setting an image classification system; 2, setting a simulation experiment; 3, designing an experiment; 4, carrying out a simulation experiment; according to the method, through experiments carried out on three groups of HSI and LiDAR common data sets and comparison with an existing classification method, the result shows that the proposed method has a good classification effect and excellent performance, a series of ablation experiments verify the effectiveness of dynamic convolution, a CT method and a CFM module in a CFML method on multi-source remote sensing classification, and the method has the advantages of being high in accuracy and high in reliability. Coarse-grained feature extraction is carried out by using double-branch dynamic convolution, the feature expression ability is greatly improved, the additional calculation cost is also reduced, a Transform based on cross marking attention is designed, interactive learning of multi-source remote sensing information is realized, complementarity between different modes is fully utilized, and the method is suitable for large-scale popularization and application. Therefore, fine-grained fusion features with discrimination are obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field, and in particular to a hyperspectral-lidar image classification method based on a coarse-fine granularity fusion network. Background Art

[0002] With the continuous evolution of remote sensing imaging technology, the availability of multi-source remote sensing imagery data covering the same geographic area has gradually increased, providing researchers with a broader and richer data resource. Different sensor technologies are capable of capturing the diverse characteristics of land cover materials, thus generating diverse remote sensing data. For example, hyperspectral imagery provides rich spectral and geospatial information, while light detection and ranging imagery provides elevation and object height information. Meanwhile, synthetic aperture radar imagery provides geometric information about the amplitude and phase of ground objects. By integrating these heterogeneous data, we can obtain more detailed information, enabling us to construct a complete feature representation. Among these data types, HSI and LiDAR data have attracted significant attention due to their complementary advantages. Hyperspectral imagery captures rich spectral and spatial information and demonstrates unique potential for classifying land cover objects of similar depth and intensity but different materials, such as sidewalks and lawns. However, when it comes to classifying certain objects, such as roads and rooftops, the limitation of similar reflectance spectra poses a challenge to accurate classification. LiDAR data, on the other hand, provides precise measurements of target distance and elevation information, making it well-suited for distinguishing objects of varying heights, such as roads and rooftops. However, distinguishing objects of the same height, such as grass and roads, becomes more challenging. By combining the spatial-spectral information of HSI with the elevation information of LiDAR, leveraging the advantages of each data and overcoming their respective limitations, this joint classification method has become an important trend in current research.

[0003] With the continuous development of ground observation methods and technologies, it is possible to obtain multi-source data more efficiently in the same geographical area, thereby significantly improving the accuracy of land cover identification. Among them, hyperspectral imagery and lidar image data have received great attention due to their excellent complementarity.

[0004] However, the traditional hyperspectral image and lidar image fusion classification method has the following disadvantages:

[0005] Traditional machine learning methods usually rely on hand-designed features, lack deep feature mining, and cannot well fit the complex nonlinear relationship between ground object features in HSI and LiDAR data, which may limit their ability to achieve higher classification accuracy in HSI and LiDAR fusion tasks.

[0006] At the same time, efficient fusion of HSI and LiDAR data is challenging due to their unrelated physical properties and different data structures. Furthermore, over-reliance on one type of information can reduce classification effectiveness. However, existing methods struggle to balance the contributions of spectral-spatial information from HSI and elevation information from LiDAR. Summary of the Invention

[0007] The purpose of the present invention is to provide a hyperspectral-lidar image classification method based on a coarse- and fine-grained fusion network to solve the problem that the traditional machine learning methods proposed in the above background technology usually rely on manually designed features, lack deep feature mining, cannot well fit the complex nonlinear relationship between the ground object features in HSI and LiDAR data, and are difficult to balance the contribution of HSI and LiDAR information; this may limit its ability to achieve higher classification accuracy in HSI and LiDAR fusion tasks.

[0008] To achieve the above object, the present invention provides the following technical solution: a hyperspectral-lidar image classification method based on a coarse-fine granularity fusion network, comprising the following steps:

[0009] Step 1: Establish an image classification system: Design a coarse-fine mutual learning strategy to fuse coarse and fine decisions and use adaptive loss to continuously adjust the weights of the two.

[0010] Step 2: Set up the simulation experiment: Set up the equipment for the simulation experiment and collect the data and equipment information for the simulation experiment in batches;

[0011] Step 3: Design the experiment: set the experimental evaluation indicators, experimental configuration and experimental parameters;

[0012] Step 4: Conduct simulation experiment: Expressed as X H ∈R MXNXB The hyperspectral image is represented by X L ∈R MXNX1 The corresponding LiDAR image of the image is obtained by multiplying the image by 1, where M and N refer to the width and height of the two images, and B represents the number of original HSI channels. After performing PCA dimensionality reduction on the input HSI, the number of compressed HSI channels is recorded as X. H ∈R MXNXb In order to make full use of the spatial information of the image, the s×s dimensional adjacent cube centered at each pixel is used for classification. For each pixel, 3D and 2D block extraction are performed to obtain a small cube. and small pieces The index of the center pixel is used to label each patch, and for edge pixels, a width For the padding operation, we assume that there are T labeled training samples, denoted as The corresponding label is where y i ∈{1,2,...,C} and C represents the number of land cover types in the remote sensing data. After removing the pixel blocks with zero labels, the remaining sample blocks are divided into training set, validation set and test set;

[0013] Step 5: Analysis and application: Analyze the results of the simulation experiment and apply them after modifying the parameters.

[0014] As a preferred technical solution of the present invention, the image classification system in step one includes a coarse-grained feature learning module, a fine-grained feature learning module, and a coarse-fine mutual guidance module. The coarse-grained feature learning module inputs the coarse-grained features into the Transformer model, the fine-grained feature learning module is used to extract the fusion information and fine-grained information of heterogeneous remote sensing data, and the coarse-fine mutual guidance module balances the coarse-grained and fine-grained information with the local and global information.

[0015] As a preferred technical solution of the present invention, the fine-grained feature learning module includes hyperspectral radar feature cross-tokenization and a Transformer encoder. The hyperspectral radar feature cross-tokenization continues the dual-branch strategy on the HSI branch, uses LiDAR features as supplementary information, abstracts LiDAR features into external classification tokens, and performs corresponding operations on the LiDAR branch. The Transformer encoder includes a multi-head cross-attention residual block and a multi-layer perceptron residual block, respectively. The cross-attention residual block starts with a layer normalization operation, followed by a CTA layer, and then the output is added to the input of the LN in an element-by-element manner.

[0016] As a preferred technical solution of the present invention, the cross-labeling of the hyperspectral radar features will expand the HSI and LiDAR feature maps by and Represented as follows, where uh and νh are the height and width of the HSI feature map, ul and vl are the height and width of the LiDAR feature map, zh and zl are the number of feature channels of the HSI and LiDAR feature maps, and the HSI cross feature marker is represented by T LH ∈R whxzh Indicates that w h =n h +1,n h is the number of HSI tokens, and LiDAR feature tokens are denoted as T HL ∈R wlxzl , where w l =n l +1,n lis the number of LiDAR tokens. The cross-tokenization process uses four learnable weights, where Wa and Wb are multiplied by the secondary input features to extract complementary features, while Wc and Wb are multiplied by the primary input features to extract salient features. The following equation summarizes the process: T LH =cls l ∪T H (5), where (.) represents the transpose operation, Wa∈R 64x1 ,Wb∈R 1xn ,cls L ∈R 1x64 ,On the lidar branch, the operation is similar to that of the HSI branch. The whole steps are summarized as follows: T HL =cls H ∪T L (8), where, in order to preserve the semantic texture information of the image cube within the image patch, the trained position embedding is added to the image patch embedding. The position embedding is added to all the tokens in an element-by-element manner. This process is followed by a dropout layer to help reduce the effect of network overfitting. The above process is summarized by the following equation: Where Dro is the dropout layer and pos is the learnable position embedding.

[0017] As a preferred technical solution of the present invention, the coarse-fine mutual guidance module includes adaptive coarse-fine decision fusion and adaptive loss fusion. The adaptive coarse-fine decision fusion represents different categories of information by fusing coarse-grained features and fine-grained features, and fuses the two different granularity decision information to enhance the robustness of the model. The adaptive loss fusion guides the feature learning process by reasonably fusing the local information extracted by the convolutional neural network with the global information.

[0018] As a preferred technical solution of the present invention, the adaptive loss fusion specifically adopts the cross entropy loss function as the basis and introduces an adaptive parameter β to dynamically adjust the trade-off between local information and global information. For the i-th training sample, our loss function is expressed by the following mathematical formula: L total =β·L c +(1-β)·L s (13), where y i is the real feature category to which the pixel belongs, represents the output of the rough prediction, Represents the final predicted output, L totalis the final fusion loss function. Once the proposed CFML network is well trained, for any test pixel X test and its corresponding prediction vector The pixel X is determined based on the maximum probability as follows test Tags:

[0019] In this way, all pixels follow the same rules, thus obtaining the entire classification map.

[0020] As a preferred technical solution of the present invention, the experimental evaluation indicators in step 3 are specifically to calculate and analyze the total accuracy, average accuracy, Kappa coefficient and classification accuracy in the experiment in order to evaluate the classification performance of the proposed framework and other existing models.

[0021] As an optimal technical solution of the present invention, the experimental parameters in step three are specifically learning rate, patch size, number of labels, encoder depth, number of attention heads, coarse and fine fusion parameters, and loss fusion parameters.

[0022] Compared with the existing technology, the beneficial effects of the present invention are: first, dual-branch dynamic convolution is used for coarse-grained feature extraction. Compared with traditional CNN, this method not only greatly improves the feature expression ability, but also reduces the additional computational cost. Secondly, in order to fuse multi-source remote sensing information in a labeled manner, a Transformer based on cross-label attention is designed. This design realizes the interactive learning of multi-source remote sensing information and makes full use of the complementarity between different modalities, thereby obtaining discriminative fine-grained fusion features. In addition, a coarse-fine adaptive mutual learning strategy is adopted to achieve the optimal balanced classification between local information and global information, as well as coarse and fine-grained information. Through experiments on three groups of HSI and LiDAR public datasets and comparison with existing classification methods, the results show that the proposed method has good classification effect and superior performance. A series of ablation experiments verify the effectiveness of dynamic convolution, CT method and CFM module in the CFML method for multi-source remote sensing classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart of the present invention;

[0024] Figure 2 Schematic diagram of the architecture of the image classification system of the present invention;

[0025] Figure 3 Detailed structural diagram of the coarse-fine granularity mutual learning fusion network of the present invention;

[0026] Figure 4 A diagram of the cross-labeling process of hyperspectral and lidar features of the present invention;

[0027] Figure 5 This is an illustration of the Transformer with multi-head cross-labeled attention of the present invention;

[0028] Figure 6 This is the Houston dataset diagram of the present invention;

[0029] Figure 7 This is the Trento dataset diagram of the present invention;

[0030] Figure 8 This is the MUUFL dataset diagram of the present invention;

[0031] Figure 9 Graph showing the impact of different parameters on overall accuracy in the three data sets of the present invention;

[0032] Figure 10 This is a diagram showing the influence of different parameters on the overall accuracy of the present invention;

[0033] Figure 11 This is one of the influence diagrams of the weight sum of the proposed method on three data sets in the present invention;

[0034] Figure 12 This is the second influence diagram of the weight sum of the proposed method on three data sets;

[0035] Figure 13 is the full pixel classification map of the Houston2013 dataset of the present invention;

[0036] Figure 14 is the full pixel classification map of the Trento dataset of the present invention;

[0037] Figure 15 is the full pixel classification map of the MUUFL dataset of the present invention;

[0038] Figure 16 This is the feature space distribution diagram of CFML under different conditions given by the t-SNE algorithm of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0040] See also Figure 1-16 The present invention provides a hyperspectral-lidar image classification method based on a coarse-fine granularity fusion network, comprising the following steps:

[0041] Step 1: Establish an image classification system: Design a coarse-fine mutual learning strategy to fuse coarse and fine decisions and use adaptive loss to continuously adjust the weights of the two.

[0042] Step 2: Set up the simulation experiment: Set up the equipment for the simulation experiment and collect the data and equipment information for the simulation experiment in batches;

[0043] Step 3: Design the experiment: set the experimental evaluation indicators, experimental configuration and experimental parameters;

[0044] Step 4: Conduct simulation experiment: Expressed as X H ∈R MXNXB The hyperspectral image is represented by X L ∈R MXNX1 The corresponding LiDAR image of the image is obtained by multiplying the image by 1, where M and N refer to the width and height of the two images, and B represents the number of original HSI channels. After performing PCA dimensionality reduction on the input HSI, the number of compressed HSI channels is recorded as X. H ∈R MXNXb In order to make full use of the spatial information of the image, the s×s dimensional adjacent cube centered at each pixel is used for classification. For each pixel, 3D and 2D block extraction are performed to obtain a small cube. and small pieces The index of the center pixel is used to label each patch, and for edge pixels, a width For the padding operation, we assume that there are T labeled training samples, denoted as The corresponding label is where y i ∈{1,2,...,C} and C represents the number of land cover types in the remote sensing data. After removing the pixel blocks with zero labels, the remaining sample blocks are divided into training set, validation set and test set;

[0045] Step 5: Analysis and application: Analyze the results of the simulation experiment and apply them after modifying the parameters.

[0046] In step 1, the image classification system includes a coarse-grained feature learning module, a fine-grained feature learning module, and a coarse-fine mutual guidance module. The coarse-grained feature learning module inputs the coarse-grained features into the Transformer model. The fine-grained feature learning module is used to extract the fusion information and fine-grained information of heterogeneous remote sensing data. The coarse-fine mutual guidance module balances the coarse-grained and fine-grained information with the local and global information.

[0047] The fine-grained feature learning module includes hyperspectral radar feature cross-tokenization and Transformer encoder. Hyperspectral radar feature cross-tokenization continues the dual-branch strategy on the HSI branch, using LiDAR features as supplementary information, abstracting LiDAR features into external classification tokens, and performing corresponding operations on the LiDAR branch. The Transformer encoder includes a multi-head cross-attention residual block and a multi-layer perceptron residual block. The cross-attention residual block starts with a layer normalization operation, followed by a CTA layer, and then the output is added to the input of the LN in an element-by-element manner.

[0048] Hyperspectral radar feature cross-labeling will expand the HSI and LiDAR feature maps by and Represented as follows, where uh and νh are the height and width of the HSI feature map, ul and vl are the height and width of the LiDAR feature map, zh and zl are the number of feature channels of the HSI and LiDAR feature maps, and the HSI cross feature marker is represented by T LH ∈R whxzh Indicates that w h =n h +1,n h is the number of HSI tokens, and LiDAR feature tokens are denoted as T HL ∈R wlxzl , where w l =n l +1,n l is the number of LiDAR tokens. The cross-tokenization process uses four learnable weights, where Wa and Wb are multiplied by the secondary input features to extract complementary features, while Wc and Wb are multiplied by the primary input features to extract salient features. The following equation summarizes the process: T LH =cls l ∪T H (5), where (.) represents the transpose operation, Wa∈R 64x1 ,Wb∈R 1xn ,cls L ∈R 1x64 ,On the lidar branch, the operation is similar to that of the HSI branch. The whole steps are summarized as follows: T HL =cls H ∪T L(8), where, in order to preserve the semantic texture information of the image cube within the image patch, the trained position embedding is added to the image patch embedding. The position embedding is added to all the tokens in an element-by-element manner. This process is followed by a dropout layer to help reduce the effect of network overfitting. The above process is summarized by the following equation: Where Dro is the dropout layer and pos is the learnable position embedding.

[0049] The coarse-fine mutual guidance module includes adaptive coarse-fine decision fusion and adaptive loss fusion. The adaptive coarse-fine decision fusion combines coarse-grained features and fine-grained features to represent different categories of information, and fuses the two different granularity decision information to enhance the robustness of the model. The adaptive loss fusion guides the feature learning process by reasonably fusing the local information extracted by the convolutional neural network with the global information.

[0050] Adaptive loss fusion specifically uses the cross entropy loss function as the basis and introduces an adaptive parameter β to dynamically adjust the trade-off between local information and global information. For the i-th training sample, our loss function is expressed as the following mathematical formula: L total =β·L c +(1-β)·L s (13), where y i is the real feature category to which the pixel belongs, represents the output of the rough prediction, Represents the final predicted output, L total is the final fusion loss function. Once the proposed CFML network is well trained, for any test pixel X test and its corresponding prediction vector The pixel X is determined based on the maximum probability as follows test Tags:

[0051] In this way, all pixels follow the same rules, thus obtaining the entire classification map.

[0052] The experimental evaluation indicators in step three are specifically: In order to evaluate the classification performance of the proposed framework and other existing models, the total accuracy, average accuracy, Kappa coefficient and classification accuracy were calculated and analyzed in the experiment.

[0053] The experimental parameters in step 3 are learning rate, patch size, number of tags, encoder depth, number of attention heads, coarse and fine fusion parameters, and loss fusion parameters.

[0054] In this invention, a coarse-fine mutual learning strategy is designed to fuse the coarse decision and the fine decision, and use the adaptive loss to continuously adjust the weight of the image classification system; set the equipment for the simulation experiment and collect the data and equipment information of the simulation experiment in batches; set the experimental evaluation index, experimental configuration and experimental parameters; combine the appendix of the manual to Figure 3 , denoted as X H ∈R MXNXB The hyperspectral image is represented by X L ∈R MXNX1 The corresponding LiDAR image of the image is obtained by multiplying the image by 1, where M and N refer to the width and height of the two images, and B represents the number of original HSI channels. After performing PCA dimensionality reduction on the input HSI, the number of compressed HSI channels is recorded as X. H ∈R MXNXb In order to make full use of the spatial information of the image, the s×s dimensional adjacent cube centered at each pixel is used for classification. For each pixel, 3D and 2D block extraction are performed to obtain a small cube. and small pieces The index of the center pixel is used to label each patch, and for edge pixels, a width For the padding operation, we assume that there are T labeled training samples, denoted as The corresponding label is where y i ∈{1,2,...,C} and C represents the number of land cover types in the remote sensing data. After removing the pixel blocks with zero labels, the remaining sample blocks are divided into training set, validation set and test set. According to the simulation experimental results, they are analyzed and the parameters are modified before application. 3D dynamic convolution and 2D dynamic convolution are used as consecutive sequential layers to extract distinctive spatial and spectral features from HSI data. Each HSI patch cube of size s×s×b in the training patch set is The training data is used as input for the 3D dynamic convolution layer. In the 3D dynamic convolution layer, the size of the convolution kernel is set to 3×3×3. After the 3D dynamic convolution operation, the generated feature cube flattens the spatial dimension into a vector and serves as the input of the next layer. Then, a 2D dynamic convolution operation is performed using the 3×3 convolution kernel to obtain 64 2D feature maps as output. Different from the HSI data processing, the LiDAR data uses two 2D dynamic convolution layers to extract elevation features. Each LiDAR patch of size s×s It is used as the input of two 2D dynamic convolutional layers. In both layers, the size of the convolution kernel is set to 3×3. Batch normalization (BN) and ReLU activation layers are applied continuously after each dynamic convolutional layer. BN is used to alleviate the overfitting caused by the scarcity of training samples and improve the training performance. ReLU plays an important role by introducing nonlinearity into the output feature map to achieve effective and smooth propagation of loss gradients. The HSI and LiDAR features obtained by the dynamic convolution component are represented as follows: Among them, DyConv 2D Represents a 2D dynamic convolution component, DyConv, consisting of 2D dynamic convolution, BN, and ReLU. 3D Represents a 3D dynamic convolution component consisting of 3D dynamic convolution, BN, and ReLU. and Represents heterogeneous coarse-grained feature information obtained through two different layers of dynamic convolution, combined with the attached manual Figure 4 , the cross-labeling of hyperspectral radar features will expand the HSI and LiDAR feature maps by and Represented as follows, where uh and νh are the height and width of the HSI feature map, ul and vl are the height and width of the LiDAR feature map, zh and zl are the number of feature channels of the HSI and LiDAR feature maps, and the HSI cross feature marker is represented by T LH ∈R whxzh Indicates that w h =n h +1,n h is the number of HSI tokens, and LiDAR feature tokens are denoted as T HL ∈R wlxzl , where w l =n l +1,n l is the number of LiDAR tokens. The cross-tokenization process uses four learnable weights, where Wa and Wb are multiplied by the secondary input features to extract complementary features, while Wc and Wb are multiplied by the primary input features to extract salient features. The following equation summarizes the process: T LH =cls l ∪T H , where (.) represents the transpose operation, Wa∈R 64x1 ,Wb∈R 1xn ,cls L ∈R 1x64 ,On the lidar branch, the operation is similar to that of the HSI branch. The whole steps are summarized as follows: THL =cls H ∪T L , where, in order to preserve the semantic texture information of the image cube within the image patch, the trained position embedding is added to the image patch embedding. The position embedding is added to all tokens in an element-by-element manner. This process is followed by a dropout layer to help reduce the effect of network overfitting. The above process is summarized by the following equation: Among them, Dro is the dropout layer, pos is the learnable position embedding, combined with the instructions attached Figure 5 , shows the Transformer encoder image caption with multi-head cross-label attention using LiDAR feature abstraction as CLS token, where cls L As the only q, As k and v, then, in order to learn the relationship between feature tokens, three learnable weights are predefined, including Wq, Wk and Wv. By multiplying k, q, and v with the three learnable weights, three different matrices are obtained, namely query Q, key K and value V. The attention score is obtained by multiplying Q by K and multiplying The softmax function is used to convert the score into a weighted probability. In short, the cross attention is expressed by the following formula: Q = cls L W q , where cls L Represents the classification label of LiDAR data, To fuse the two remote sensing data, h d Represents the dimension of K. When mapping features to Wq, Wk, and Wv, the MHCTA module uses multiple sets of weights and uses the same calculation process to obtain the MHCTA score. All these attention scores are concatenated together as follows: MHCTA = Concat(CTA1, CTA2, ..., CTA h )W, where W is the parameter matrix, where d w = n (number of tokens), the other branch uses HSI features as significant features and LiDAR features as supplementary features. Other operations are similar to the above steps. At this time, we get Y LH and Y HL Two kinds of fine-grained feature information of multi-source data are fused. Adaptive loss fusion specifically uses the cross entropy loss function as the basis and introduces an adaptive parameter β to dynamically adjust the trade-off between local information and global information. For the i-th training sample, our loss function is expressed by the following mathematical formula: L total =β·L c +(1-β)·Ls , where y i is the real feature category to which the pixel belongs, represents the output of the rough prediction, Represents the final predicted output, L total is the final fusion loss function. Once the proposed CFML network is well trained, for any test pixel X test and its corresponding prediction vector The pixel X is determined based on the maximum probability as follows test Tags: In this way, all pixels follow the same rules, thus obtaining the entire classification map.

[0055] In this section, extensive experiments are conducted on three widely used HSI and LiDAR datasets, which are introduced below: Houston2013: The first dataset was collected by the ITRES CASI-1500 imaging sensor on a university campus and its surrounding rural areas, and was released for IEEE GRSSDFC 2013. The dataset consists of two data sources, of which the hyperspectral data contains 144 spectral bands with a spectral resolution ranging from 0.38 to 1.05 μm, and the LiDAR data only provides 1 band. Both the hyperspectral and LiDAR data consist of 349 × 1905 pixels with a spatial resolution of 2.5 meters, resulting in a total of 15 different categories. Detailed information is shown in Table 1 and the Appendix of the manual. Figure 6 Trento: The second dataset contains hyperspectral and lidar data. The hyperspectral data is captured by the AISA Eagle sensor, and the lidar DSM data is collected by the Optech ALTM 3100EA sensor. This dataset is a subset of a larger dataset captured in a rural area south of the city of Trento, Italy. It contains six categories, with an image size of 600 × 166 pixels and a spatial resolution of 1.0 m. Detailed information is provided in Table 1 and the appendix of the manual. Figure 7 MUUFL: The third dataset was acquired in November 2010 by the Reflective Optical System Imaging Spectrometer sensor in the campus area of ​​the University of Southern Mississippi Gulf Park. In this dataset, the hyperspectral data includes 72 spectral bands ranging from 0.38 to 1.05 μm, while the lidar data contains two gratings with a wavelength of 1.06 μm. Due to excessive noise, the first and last 8 spectral bands were deleted, leaving 64 spectral bands as the new HSI data. The dataset consists of 325×220 pixels with a spatial resolution of 1.0 m and a total of 11 different land cover categories. Detailed information is shown in Table 1 and the appendix of the manual. Figure 8 ;

[0056]

[0057]

[0058] Table 1. Size of training, validation, and test samples for the Houston2013, Trento, and MUUFL datasets.

[0059] Figure 6 (a) Pseudo-color composite image based on HSI bands 64, 30, and 20; (b) grayscale image based on lidar; (c) ground truth map, Figure 7 In the figure, (a) pseudo-color composite image based on HSI bands 30, 15, and 7; (b) grayscale image based on lidar; (c) ground truth map, Figure 8 (a) Pseudo-color composite image based on HSI bands 30, 15, and 7; (b) grayscale image based on lidar; (c) ground truth map.

[0060] In order to evaluate the classification performance of the proposed framework and other existing models, the overall accuracy (OA), average accuracy (AA), Kappa coefficient (Kappa) and classification accuracy were calculated and analyzed in the experiments. For each indicator, the higher the value, the more accurate the classification. The experiments of the proposed method and other deep learning methods were implemented on the PyTorch platform. The laptop used was equipped with an AMD Ryzen 76800H (with Radeon Graphics 3.20GHz CPU), 16GB memory and NVIDIA GeForce RTX 3060Laptop GPU (with 6GB VRAM), in order to optimize the network, Adam optimizer is selected as the initial optimizer. In the training stage, the mini-batch size and the number of training epochs are set to 64 and 100 respectively. Several parameters that affect the classification performance and training process are analyzed, including learning rate, patch size s of input data, number of labels, encoder depth, number of attention heads, coarse and fine fusion parameters, and loss fusion parameters. All parameter analyses are based on the control variable method. Below we will introduce the impact analysis of these hyperparameters in detail. (1) Learning rate: The learning rate is a key parameter in the deep learning model. It has an important impact on the training process and the final classification results. By setting the appropriate learning rate, the model parameters can be effectively guided to update in the optimal direction, helping the objective function to converge quickly to the local minimum. In order to evaluate the effect of the learning rate, we selected different learning rates from the candidate set {0.01, 0.005, 0.001, 0.0005, 0.0001, 0.00005} for experiments. The instruction manual is attached. Figure 9(a) shows the OA obtained by setting different learning rates for the proposed method on three datasets. The results show that the best learning rates for the Houston2013, Trento and MUUFL datasets are 0.01, 0.001 and 0.0001 respectively; Patch size: In the field of remote sensing image classification, accurately determining the appropriate patch size is crucial for effectively capturing the features of the ground objects. In the experiment, we screened the most suitable patch size from the candidate set {5, 7, 9, 11, 13, 15}. The instructions are attached. Figure 9 As shown in (b), we studied the effect of different patch sizes on OA. When the patch size increases from to , OA shows an increasing trend. When the patch size is 7, the three datasets all reach the optimal OA value. However, as the patch size further increases, OA shows a decreasing trend. This is because an excessively large patch size will increase the number of pixels that are inconsistent with the central pixel label, making it difficult to extract small-scale ground feature extraction, thereby reducing classification performance. Number of markers: Different datasets have different features and data distributions, so it is necessary to determine the appropriate number of markers. Figure 9 In (c), we show the impact of the number of markers on OA on three datasets. In the parameter experimental analysis, the number of markers for LiDAR and HSI is set to be equal, and the number of markers is selected from the candidate set {2, 4, 6, 8, 10, 12}. The results show that on the Houston2013 dataset and the MUUFL dataset, the best OA value is achieved when the number of markers is 8, while on the Trento dataset, the OA value is best when the number of markers is 4. Except for datasets that are easy to classify such as Trento, the OA values ​​of other datasets show a significant downward trend after the number of markers is 8. This is because too many markers cause the model to pay too much attention to some irrelevant features and ignore features that are more important to the classification task, resulting in a decrease in classification performance. The depth of the encoder will affect the feature representation ability. In order to verify the impact of the encoder depth on the classification performance, the candidate set is set to {1, 2, 3, 4, 5}. The instruction manual is attached. Figure 10 (a) shows the OA of different encoder depths in the three datasets. When the encoder depth is 2, the proposed network achieves the best classification effect. As the encoder depth increases, the performance of the network on the three datasets decreases. This phenomenon shows that deeper networks do not bring better performance. Increasing the number of attention heads improves the representation ability of the model, but it also increases the computational requirements and the possibility of overfitting. Therefore, it is necessary to determine the appropriate number of attention heads. According to the attached manual, Figure 10(b) Results of testing different numbers of attention heads on three datasets. The candidate set of the number of attention heads is {2, 4, 8, 16, 32}. Experiments show that on the Houston2013 dataset and the MUUFL dataset, when the number of attention heads is 8, the model reaches the optimal OA value, while on the Trento dataset, the optimal number of attention heads is 4. The graphical results show that before reaching the optimal number of attention heads, OA shows an upward trend. However, after reaching the optimal number of attention heads, increasing the number of attention heads will cause the OA value to fluctuate, which indicates that increasing the number of attention heads will improve the representation ability and information interaction ability of the model, but will also increase the computational cost and the risk of overfitting.

[0061] Instructions attached Figure 11 The influence of coarse-grained and fine-grained decision-making on three data sets is shown. In the figure, train_acc represents the training accuracy, val_acc represents the verification accuracy, represents the coarse-grained decision weight, and represents the fine-grained decision weight. Figure 11 As shown in (a) of the Houston dataset, when the epoch number is close to 20, the training accuracy gradually stabilizes. However, when the epoch number is greater than 20, the verification accuracy fluctuates slightly and is still lower than the training accuracy. It can be inferred that the network has a slight overfitting phenomenon. When the network stabilizes, it is observed that it is less than , which means that fine-grained decision-making provides more effective representation information than coarse-grained decision-making. Figure 11 (b) shows the various parameters on the Trento dataset. When the training accuracy and validation accuracy tend to be stable, their sizes are close to equal, which shows that the network has successfully learned features on the Trento dataset. In this process, they first decrease and then increase, and finally tend to be equal, which shows that both coarse-grained and fine-grained decisions play an important role in land feature classification. Figure 11 (c) shows the various parameters on the MUUFL dataset. When the difference between and increases, the training accuracy and verification accuracy gradually stabilize, which shows that ADF dynamically adjusts the weights of coarse-grained and fine-grained information to achieve better classification results. Figure 12 The influence of the composition of loss function on network performance on three datasets is shown. Among them, represents the local loss based on dynamic convolution, and represents the loss of the network's final decision. The fusion of the two aims to balance local and global information to train the best network performance. Figure 12 In (a), when the network tends to be stable, it is observed that it is greater than, which indicates that on the Houston dataset, the network needs to obtain more local information to improve the accuracy of object classification. Figure 12 (b) and the appendix to the instructions Figure 12In (c), for the Trento and MUUFL datasets, as the network accuracy stabilizes, the difference between and gradually increases and is less than, which suggests that the network gradually weakens the influence of local loss. Under the action of ALF, the network achieves a suitable balance between local and global information on different datasets, thereby achieving a higher classification effect.

[0062] To verify the effectiveness of the proposed framework, we conducted a series of experiments and compared it with several representative classification methods, including SVM, CoupledCNN, the recently proposed OSDN, SSFTT, AM3Net

[33] , HCT, and S2Enet. Among them, SVM, OSDN, and SSFTT are respectively machine learning methods, convolutional neural networks, and Transformer are representatives of hyperspectral classification methods, CoupledCNN is a representative of early multi-source remote sensing classification methods, and AM3Net, HCT, and S2Enet are representatives of the most advanced fusion networks. All compared methods used default parameters and were trained with the provided original files. Each experiment was repeated 10 times independently, and the average value was reported as the final result, as shown in Table I. To ensure a fair comparison, we randomly sampled the training sample sets, validation sample sets, and test sample sets of all methods. We first compared the performance of various methods on the Houston2013 dataset. The classification results are shown in Table 2, and the full classification diagram is shown in Figure 13, the best values ​​of OA, AA, Kappa and classification accuracy are highlighted in bold in the table. Overall, compared with other methods, the proposed CFML network achieved good classification results in most cases on the four evaluation indicators. Specifically, the OA obtained by the deep learning-based method is much higher than that of SVM (81.20%). Specifically, the C12 (59.65%) and C13 (22.41%) classes are difficult to be classified by SVM, while other deep learning methods have an accuracy of more than 70% and 80% for these two classes, respectively. This shows that the introduction of spatial features and high-level features helps to improve the recognition rate of ground objects. The accuracy of the CFML network proposed by us for these two classes is the best level among all the compared algorithms. This is because the CNN-Transformer structure enables the network to obtain more outstanding spatial features and high-level feature extraction capabilities. OSDN provides a higher OA (90.88%) than SVM. This is because Because its two polarization attentions make full use of spatial-spectral features, OSDN is 14.37% lower than SSFTT in terms of accuracy of class C13, which shows that Transformer has a stronger ability to extract higher-level features. However, SSFTT's OA is lower than OSDN, which is caused by fewer training samples. Observing the classification accuracy of various methods, we see that, in general, the CoupledCNN, AM3Net, HCT, S2Enet and CFML methods using HSI-LiDAR input have higher accuracy than the classification method using HSI input alone, which shows that the introduction of LiDAR information effectively improves the classification accuracy. In the comparison of fusion classification methods, the proposed CFML obtains the highest OA, and more than half of the classification accuracy reaches the highest. This is because the proposed feature fusion method fully combines the heterogeneous information between HSI and LiDAR to obtain more discriminative features. Figure 13 As can be seen from the figure, the classification map generated by the proposed CFML is smoother and has fewer misclassifications than other classification maps. Since SVM only uses the spectral features of each pixel for classification, the classification result map is full of a lot of salt and pepper noise. Other methods introduce spatial information and generate full classification maps with less salt and pepper noise, but some methods still have serious misclassifications. For example, AM3Net loses a lot of spatial and elevation features. On the contrary, due to the use of carefully designed multi-source features, decision fusion and adaptive strategies, the classification performance of most land cover in the CFML classification map is more consistent than that of AM3Net.

[0063]

[0064] Table 2. Performance of various methods in Houston2013 dataset classification

[0065] In the experiment of this dataset, there are few training sample categories and sufficient training samples. The specific classification results and the full classification diagram are shown in Table 3 and Figure 14 As shown in the figure, observing the classification accuracy of each method, SVM obtained the lowest OA (82.43%), and the OA of other single-source methods, OSDN and SSFTT were 96.28% and 96.26% respectively. In addition, the OA of all multi-source fusion methods were greater than 98%, which shows that the introduction of LiDAR data is beneficial to improving the recognition rate of ground objects. Further observation of the classification accuracy of each category shows that with the introduction of spatial information, the accuracy of C02 and C06 increased from 70% to 80%, and with the introduction of elevation information, the accuracy of C02 and C06 increased from 80% to 90%, which shows that the addition of LiDAR data is beneficial to the classification of these two categories. Of course, this is not absolute. Compared with the OSDN and SSFTT methods that only use a single HSI input, the accuracy of C01 and C03 in CoupledCNN, AM3Net and HCT methods has decreased. This shows that for C01 and C03, the addition of elevation information will affect the performance of spatial information. Figure 14 , we see that the apple tree (C01) is surrounded by the ground (C03). In these two categories, CoupledCNN, AM3Net and HCT methods have some misclassification situations. This shows that only by fully integrating the features of the two data can the advantages of the two data be brought into play, thereby steadily improving the recognition rate of ground objects. In this regard, the proposed method CFML performs well, and OA, AA, Kappa and classification accuracy all reach a good level. Observation Figure 13 It can be seen that the full classification map of CFML is cleaner and smoother, especially the boundaries of the regions corresponding to C01 and C03 are very clear, which further proves the classification performance of the proposed method;

[0066]

[0067] Table 3. Performance of various methods in Trento dataset classification;

[0068] To further verify the performance of the proposed method, we used the MUUFL dataset for experiments. The dataset contains complex terrain landscapes and has an imbalance in the number of labeled samples. The classification results and the full classification map are shown in Table 4 and Figure 15,Observing the classification accuracy of each method, SVM obtains the lowest OA (83.27%), and the proposed method CFML obtains the highest OA (89.92%), with a difference of 6.65%. ,In addition, the OA of CFML is only 0.4% different from the second-best method. ,This is because the MUUFL dataset has a large number of samples, which makes most methods better capture the ,distribution characteristics of the data. ,Further observing the accuracy of each class, it is found that C03, C07, C09 and C10 are difficult for most methods to ,classify. However, on C03 and C10, the proposed method CFML performs best. ,This is because the coarse-grained and fine-grained decision fusion ,strategy fuses the coarse and fine-grained information and continuously optimizes the ,weight representation according to the accuracy. Figure 15 As can be seen from the figure, the full classification map generated by the CFML method is cleaner and smoother than that of other methods, which is consistent with the classification results, indicating that CFML makes better use of HSI and LiDAR data;

[0069]

[0070]

[0071] Table 4. Performance of various methods in MUUFL dataset classification

[0072] We present ablation analyses of different data sources and components to verify their effectiveness. In addition, we visualize important ablation experimental results to intuitively demonstrate their impact.

[0073] (1) Ablation analysis of different data source inputs: Considering the impact of heterogeneous data on the classification performance of the model, we conducted three sets of experiments: single HSI data input, single LiDAR data input, and HSI-LiDAR input. The experimental results are shown in Table 5. From the experimental results, we can see that after removing the LiDAR data, the OA, AA, and κ of the proposed method on the three datasets have a certain degree of decline, which shows that LiDAR data is effective in improving the accuracy of ground object recognition. However, after removing only the HSI data, the proposed method has a low accuracy on the Houston2013, Trento, and MUUFL datasets. The classification performance of all datasets has dropped significantly, with OA dropping by 49.98%, 6.54%, and 26.69% respectively. On the Trento dataset, the accuracy drop is not obvious, while on the Houston and MUUFL datasets, the accuracy drops significantly. This phenomenon indicates that the classification of high-difficulty datasets such as Houston and MUUFL relies more on spectral-spatial information to provide more discriminative information. This verifies that the two types of data play a positive role in improving classification performance, and also verifies that the designed network fully utilizes the complementary information of different remote sensing data.

[0074]

[0075] Table 5. Ablation analysis of different data source inputs (the best results are shown in bold)

[0076] (2) Ablation analysis of different components: Since the proposed network benefits from multiple components, we analyze the contribution of each component to the joint classification performance in the ablation experiment. Specifically, the proposed framework is divided into three main parts: Dynamic Conv 3D and Dynamic Conv 2D, which are used to process HSI and LiDAR data, the cross tokenization (CT) method is used for feature fusion of the two types of data, and the CFM module, which contains two components, ADF and ALF. According to the results in Table 6, we show five variants of the proposed method on three datasets, which delete different combinations of components. We analyze the classification accuracy of these variants to evaluate the impact of different components on model performance. On the three datasets, the first variant that deletes dynamic convolution reduces OA by 1.37%, 0.46% and 0.73% respectively compared with the full-component model, which confirms the importance of dynamic convolution in the feature extraction stage. It can enhance the feature extraction ability of the CFML network in the coarse feature learning stage. The second variant lacks the ALF component, which means the lack of CFM component. The accuracy of this variant is reduced by The low verification verifies that ALF can effectively balance local and global information to achieve better classification results. The third variant is without ADF components, which means there is no CFM component. Compared with the full component, the classification effect of this variant is significantly reduced, which verifies that ADF can effectively fuse coarse-grained and fine-grained information to obtain comprehensive and accurate classification information. The fourth variant does not have all components of the CFM module, and its classification accuracy is the worst. This is enough to prove that the CFM module plays a key role in reducing information loss in the feature abstraction process and thus improving classification accuracy. The accuracy of the fifth variant decreases without using CT, which shows that CT plays an effective role in overcoming the obstacles of heterogeneous data structures and irrelevant physical properties. This analysis further confirms the effectiveness of the proposed network framework and demonstrates the positive role of all its main components.

[0077]

[0078]

[0079] Table 6. Ablation analysis of different components of the proposed model on three datasets (the best results are marked in bold)

[0080] (3) Visual evaluation and analysis: To more intuitively analyze the excellent feature learning ability of the proposed CFML, we use the t-distributed stochastic neighbor embedding (t-SNE) algorithm to map high-dimensional sample features into a two-dimensional space and determine the separability between different categories by observation. The results without the use of lidar visualization are shown in Figure 2. Figure 16 (a), (b), (c) show the results of visualization without using CFM module. Figure 16 (d), (e), and (f) are shown, while the results of using full component visualization are shown in Figure 16 (g), (h), (i) show, compare Figure 16 (a)-(c) and Figure 16 (d)-(f) show that, in these three datasets, after adding LiDAR, the separable boundaries between objects are more obvious. This is because HSI alone is not sensitive enough to the three-dimensional morphological information of the objects, while LiDAR provides high-precision morphological and elevation features. The combination of HSI and LiDAR obtains more comprehensive and detailed three-dimensional spatial information while retaining rich spectral information. Figure 16 As can be seen from (g)-(i), the distances between similar classes are relatively close. This is because CFM reduces the information loss in the feature abstraction process to obtain more comprehensive and accurate information.

[0081] (4) Complexity Analysis: The following are the parameters and floating-point operations (FLOPs) of different methods on three datasets. As shown in Table 7, the parameter values ​​and FLOPs values ​​are calculated when the batch size is 1. All data in the table are calculated using the method provided by the thop library. The size of the observed dataset also affects the number of parameters and FLOPs. In particular, the results of all methods on the Trento dataset are similar to those on the MUUFL dataset. This is because the data sizes of the two datasets are similar. In contrast, the data of AM3Net does not seem to be affected by the size of the dataset. This is because the network uses a (1×1) convolution for dimensionality transformation in the early stage. The CFML we proposed performs moderately in terms of parameters, but performs well in terms of FLOPs. This is due to the use of dynamic convolution and efficient network structure, which achieves higher representation capabilities with smaller FLOPs.

[0082]

[0083] Table 7. Comparison of parameter count and FLOPs.

[0084] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A hyperspectral-lidar image classification method based on a coarse-grained and fine-grained fusion network, characterized in that: The following steps are involved: Step 1: Establish an image classification system: Design a coarse-fine mutual learning strategy to fuse coarse decisions with fine decisions and use adaptive loss to continuously adjust the weights of the two image classification systems; Step 2: Set up the simulation experiment: set up the equipment for the simulation experiment and collect the data and equipment information for the simulation experiment in batches; Step 3: Design the experiment: set the experimental evaluation indicators, experimental configuration and experimental parameters; Step 4: Conduct simulation experiment: represented by X H ∈R MXNXB The hyperspectral image is represented by X L ∈R MXNX1 The corresponding LiDAR image of the image is shown in Figure 2, where M and N refer to the width and height of the two images, B represents the number of original HSI channels, and after performing PCA dimensionality reduction on the input HSI, the number of compressed HSI channels is recorded as X. H ∈R MXNXb In order to make full use of the spatial information of the image, an s×s dimensional adjacent cube centered on each pixel is used for classification. For each pixel, 3D and 2D block extraction is performed to obtain a small cube and small pieces The index of the center pixel is used to label each patch, and for edge pixels, a width For the padding operation, we assume that there are T labeled training samples, denoted as The corresponding label is where y i ∈{1,2,...,C} and C represents the number of land cover types in remote sensing data. After removing the pixel blocks with zero labels, the remaining sample blocks are divided into training set, validation set and test set; Step 5: Analysis and application: Analyze the simulation results and apply them after modifying the parameters.

2. The hyperspectral-lidar image classification method based on coarse-fine granularity fusion network according to claim 1 is characterized in that: The image classification system in step 1 includes a coarse-grained feature learning module, a fine-grained feature learning module, and a coarse-fine mutual conduction module. The coarse-grained feature learning module inputs the coarse-grained features into the Transformer model, the fine-grained feature learning module is used to extract the fusion information and fine-grained information of heterogeneous remote sensing data, and the coarse-fine mutual conduction module balances the coarse-grained and fine-grained information with the local and global information.

3. The hyperspectral-lidar image classification method based on coarse-grained and fine-grained fusion network according to claim 2 is characterized in that: The fine-grained feature learning module includes hyperspectral radar feature cross-tokenization and a Transformer encoder. The hyperspectral radar feature cross-tokenization continues the dual-branch strategy on the HSI branch, uses LiDAR features as supplementary information, abstracts LiDAR features into external classification tokens, and performs corresponding operations on the LiDAR branch. The Transformer encoder includes a multi-head cross-attention residual block and a multi-layer perceptron residual block, respectively. The cross-attention residual block starts with a layer normalization operation, followed by a CTA layer, and then the output is added to the input of the LN in an element-by-element manner.

4. The hyperspectral-lidar image classification method based on coarse-grained and fine-grained fusion network according to claim 3 is characterized in that: The hyperspectral radar feature cross-labeling will expand the HSI and LiDAR feature maps by and where uh and νh are the height and width of the HSI feature map, ul and vl are the height and width of the LiDAR feature map, zh and zl are the number of feature channels of the HSI and LiDAR feature maps, and the HSI cross feature marker is represented by T LH ∈R whxzh Indicates that w h =n h +1,n h is the number of HSI tokens, and LiDAR feature tokens are denoted as T HL ∈R wlxzl , where w l =n l +1,n l is the number of LiDAR tokens. The cross-tokenization process uses four learnable weights, where Wa and Wb are multiplied by the secondary input features to extract complementary features, while Wc and Wb are multiplied by the primary input features to extract salient features. The following equation summarizes the process: T LH =cls l ∪T H (5), where (.) represents the transpose operation, Wa∈R 64x1 ,Wb∈R 1xn ,cls L ∈R 1x64 ,On the LiDAR branch, the operation is similar to that of the HSI branch. The whole steps are summarized as follows: T HL =cls H ∪T L (8), where, in order to preserve the semantic texture information of the image cube within the image patch, the trained position embedding is added to the image patch embedding. The position embedding is added to all the tokens in an element-by-element manner. This process is followed by a dropout layer, which helps to reduce the effect of network overfitting. The above process is summarized by the following equation: Where Dro is the dropout layer and pos is the learnable position embedding.

5. The hyperspectral-lidar image classification method based on coarse-grained and fine-grained fusion network according to claim 2 is characterized in that: The coarse-fine mutual guidance module includes adaptive coarse-fine decision fusion and adaptive loss fusion. The adaptive coarse-fine decision fusion represents different categories of information by fusing coarse-grained features and fine-grained features, and fuses two types of decision information of different granularities to enhance the robustness of the model. The adaptive loss fusion guides the feature learning process by reasonably fusing local information and global information extracted by convolutional neural networks.

6. The hyperspectral-lidar image classification method based on coarse-grained and fine-grained fusion network according to claim 5 is characterized by: The adaptive loss fusion specifically uses the cross entropy loss function as the basis and introduces an adaptive parameter β to dynamically adjust the trade-off between local information and global information. For the i-th training sample, our loss function is expressed by the following mathematical formula: L total =β·L c +(1-β)·L s (13), where y i is the real feature category to which the pixel belongs, represents the output of the rough prediction, Represents the final predicted output, L total is the final fusion loss function. Once the proposed CFML network is well trained, for any test pixel X test and its corresponding prediction vector The pixel X is determined based on the maximum probability as follows test Tags: In this way, all pixels follow the same rule, thus obtaining the entire classification map.

7. The hyperspectral-lidar image classification method based on coarse-grained and fine-grained fusion network according to claim 1 is characterized in that: The experimental evaluation indicators in step 3 are specifically: In order to evaluate the classification performance of the proposed framework and other existing models, the total accuracy, average accuracy, Kappa coefficient and classification accuracy were calculated and analyzed in the experiment.

8. The hyperspectral-lidar image classification method based on coarse-grained and fine-grained fusion network according to claim 1 is characterized in that: The experimental parameters in step 3 are specifically learning rate, patch size, number of tags, encoder depth, number of attention heads, coarse and fine fusion parameters, and loss fusion parameters.

Citation Information

Cited By

  • Multi-modal hyperspectral image classification method and system

    CN119888333A

  • Method for deciding polishing process parameters based on image quality indexes of neural network

    CN120068675A

  • Method for determining polishing process parameters based on image quality indexes based on neural network

    CN120068675B

  • Visual language navigation method and system based on collaborative alignment and adaptive fusion

    CN120427010A