A hyperspectral and LiDAR data joint classification method based on text semantic guided space-frequency domain fusion network

Abstract

A hyperspectral and LiDAR data joint classification method based on text semantic guided space-frequency domain fusion network belongs to the field of remote sensing image classification. The present application aims at the problems of insufficient utilization of class semantics and insufficient expression of spatial and frequency domain information in joint classification, and constructs a text semantic guided space-frequency domain fusion network. The method obtains hyperspectral spatial-spectral features and LiDAR elevation features through a visual feature extraction module, strengthens the local structure and global frequency domain representation through a space-frequency feature enhancement module, and further realizes the complementary interaction of hyperspectral features and LiDAR features through a bidirectional cross-modal fusion module to form a fusion visual feature; meanwhile, a text semantic prototype is generated by using a class text description, and a visual-text alignment loss is used to constrain multi-modal feature learning. Finally, the fusion visual feature is input into a classification head to obtain the land cover class results of the region to be classified. The method can be applied to the land cover classification task of hyperspectral images and LiDAR data.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image classification, specifically involving a joint classification method for hyperspectral and LiDAR data based on a text semantic-guided spatial-frequency domain fusion network. Background Technology

[0002] With the development of remote sensing observation technology and intelligent interpretation methods, multi-source remote sensing data has been widely used in tasks such as land cover classification, urban management, agricultural monitoring, and ecological environment assessment. Hyperspectral images possess continuous and rich spectral information, enabling them to characterize the spectral differences of different ground features; LiDAR data provides elevation, structural, and geometric information, making it suitable for describing the spatial structure of ground features. The two are complementary in their information representation; therefore, joint classification of hyperspectral and LiDAR data has become an important research direction in the field of multi-source remote sensing image classification.

[0003] Existing joint hyperspectral and LiDAR classification methods typically extract features from both types of data using convolutional neural networks, attention mechanisms, or Transformer structures, and employ feature concatenation, weighted fusion, or cross-attention mechanisms to achieve multimodal fusion, providing an effective technical foundation for classifying complex surface scenes. However, under conditions of complex land cover boundaries, similar categories, and limited samples, relying solely on visual features and discrete label supervision is still insufficient to fully express the semantic differences between categories. Furthermore, the texture, boundaries, and regional structures in remote sensing images often contain both spatial and frequency domain information; therefore, it is necessary to enhance feature representation from a spatial-frequency domain perspective.

[0004] Based on the aforementioned needs, textual semantic guidance, spatial-frequency domain feature enhancement, and bidirectional cross-modal fusion have significant application value. Category text descriptions can provide semantic priors such as category names, color attributes, and geographic context; spatial-frequency domain enhancement helps strengthen local structure and global frequency representation; and bidirectional cross-modal fusion can promote complementary interaction between hyperspectral spatial-spectral information and LiDAR elevation structure information. Therefore, the discriminative power, semantic consistency, and land cover classification accuracy of multimodal features can be improved. Summary of the Invention

[0005] The purpose of this invention is to address the problems of insufficient utilization of category semantic information, inadequate expression of spatial-frequency domain structural features, and insufficient mining of cross-modal associations between heterogeneous data in traditional joint classification methods of hyperspectral and LiDAR data, which lead to limited classification accuracy of complex land cover. Therefore, this invention proposes a joint classification method of hyperspectral and LiDAR data based on a text semantic-guided spatial-frequency domain fusion network.

[0006] The technical solution adopted by this invention to solve the above-mentioned technical problems is: a joint classification method for hyperspectral and LiDAR data based on a text semantic-guided spatial-frequency domain fusion network, the method specifically including the following steps:

[0007] Step 1: Obtain LiDAR data and hyperspectral image data from the dataset;

[0008] Step 2: Perform dimensionality reduction processing on the acquired hyperspectral image data to obtain dimensionality-reduced hyperspectral image data;

[0009] Step 3: Slice the LiDAR data and the dimensionality-reduced hyperspectral image data to obtain LiDAR data blocks and hyperspectral image data blocks. Then divide the sliced ​​LiDAR data blocks and hyperspectral image data blocks into training set and test set.

[0010] Step 4: Construct category text descriptions based on land cover categories. The category text descriptions include category name descriptions, color attribute descriptions, and geographic context descriptions.

[0011] Step 5: Construct a text semantic-guided space-frequency domain fusion network, and train the constructed joint classification network using the training set until the maximum number of training iterations is reached or the joint classification network meets the preset convergence condition, and then stop training to obtain the trained joint classification network.

[0012] The text semantic-guided spatial-frequency domain fusion network includes a visual feature extraction module, a spatial-frequency domain feature enhancement module, a bidirectional cross-modal fusion module, a text encoding module, a visual-text semantic alignment module, and a classification head. The visual feature extraction module includes a hyperspectral feature extraction block and a LiDAR feature extraction block, and its working process is as follows:

[0013] The dimensionality-reduced hyperspectral image data block is input into the hyperspectral feature extraction block, and hyperspectral local spatial-spectral features are extracted through a 3×3×3 three-dimensional convolutional layer, a three-dimensional batch normalization layer, and a ReLU activation function layer. Then, the output data is reconstructed to merge the channel-dimensional features and spectral-dimensional features into a two-dimensional hyperspectral feature map. The number of channels is adjusted through a two-dimensional channel projection convolutional block to obtain the hyperspectral spatial-spectral features.

[0014] The LiDAR data block is input into the LiDAR feature extraction block, and the LiDAR elevation structure features are extracted through a 3×3 two-dimensional convolutional layer, a batch normalization layer, and a ReLU activation function layer.

[0015] Hyperspectral spatial-spectral features and LiDAR elevation structure features are respectively input into the spatial-frequency domain feature enhancement module. The spatial enhancement branch enhances local boundaries, texture details and structural change information, while the frequency domain enhancement branch models frequency energy distribution and phase structure information. The two types of enhanced features are then concatenated and integrated through the channel fusion convolution module to obtain the spatial-frequency domain enhanced hyperspectral features and LiDAR features.

[0016] The hyperspectral and LiDAR features enhanced in the spatial-frequency domain are reconstructed into one-dimensional sequence features and input into a bidirectional cross-modal fusion module. The two types of features are interactively updated through bidirectional cross-attention mechanism, residual connection, layer normalization and nonlinear transformation. The fused visual features are then obtained through global aggregation and splicing.

[0017] The category text description is input into the text encoding module for semantic modeling to generate a category-level text semantic prototype. Hyperspectral features, LiDAR features and fused visual features are mapped to a unified semantic space and visual-textual alignment is performed with the category-level text semantic prototype.

[0018] During training, the classification loss is used to constrain the category discrimination result of the fused visual features, and the network is jointly optimized by hyperspectral visual-text alignment loss, LiDAR visual-text alignment loss and fused visual-text alignment loss. Finally, the fused visual features are input into the classification head to obtain the classification result.

[0019] Step 6: Use the trained joint classification network to jointly process the hyperspectral image and LiDAR data of the region to be classified to obtain the land cover classification result of the region to be classified.

[0020] The beneficial effects of this invention are:

[0021] This invention can extract spatial-spectral features from hyperspectral image data and obtain elevation structure features from LiDAR data. It enhances local boundaries, texture details, and global frequency domain structure information through a spatial-frequency domain feature enhancement module, improving the discriminative ability of multi-source remote sensing features. Through a bidirectional cross-modal fusion module, hyperspectral and LiDAR features can complement each other, reducing the information underutilization caused by simple stitching or one-way fusion. Simultaneously, this invention utilizes category-level textual descriptions to generate category-level textual semantic prototypes and applies semantic constraints to hyperspectral, LiDAR, and fused visual features through visual-text alignment loss, improving the semantic consistency and inter-class separability of multimodal features, thereby enhancing the accuracy and stability of joint classification of hyperspectral and LiDAR data. Attached Figure Description

[0022] Figure 1 This is a structural diagram of a text semantic-guided space-frequency domain fusion network;

[0023] Figure 2 This is a training flowchart based on a text semantic-guided space-frequency domain fusion network;

[0024] Figure 3 This is a structural diagram of the visual feature extraction module;

[0025] Figure 4 This is a structural diagram of the space-frequency domain feature enhancement module;

[0026] Figure 5 This is a structural diagram of a bidirectional cross-modal fusion module;

[0027] Figure 6 This is a structural diagram of the text encoding module and the visual-text semantic alignment module;

[0028] Figure 7 This is a table showing the land cover categories and sample size statistics for the Trento dataset;

[0029] Figure 8 This is a statistical table of land cover categories and sample sizes for the Augsburg dataset;

[0030] Figure 9 This is a statistical table of land cover categories and sample size for the Houston2013 dataset;

[0031] Figure 10 It is the Trento dataset;

[0032] In the figure, (a) is the hyperspectral false color image, (b) is the DSM grayscale image, and (c) is the ground truth image;

[0033] Figure 11 It is the Augsburg dataset;

[0034] In the figure, (a) is the hyperspectral false color image, (b) is the DSM grayscale image, and (c) is the ground truth image;

[0035] Figure 12 It is the Houston 2013 dataset;

[0036] In the figure, (a) is the hyperspectral false color image, (b) is the DSM grayscale image, and (c) is the ground truth image;

[0037] Figure 13 It is a subjective classification result of different datasets based on a text semantic-guided spatial-frequency domain fusion network;

[0038] In the image, (a) is Trento, (b) is Augsburg, and (c) is Houston 2013. Detailed Implementation

[0039] Specific implementation method one: Combining Figure 1 and Figure 2 This embodiment describes a joint classification method for hyperspectral and LiDAR data based on a text semantic-guided spatial-frequency domain fusion network. The method specifically includes the following steps:

[0040] Step 1: Obtain LiDAR data and hyperspectral image data from the dataset;

[0041] For each hyperspectral image obtained from the dataset, LiDAR data of the corresponding region of the hyperspectral image was also obtained simultaneously.

[0042] Step 2: Perform dimensionality reduction processing on the acquired hyperspectral image data to obtain dimensionality-reduced hyperspectral image data;

[0043] Step 3: Slice the LiDAR data and the dimensionality-reduced hyperspectral image data to obtain LiDAR data blocks and hyperspectral image data blocks. Then divide the sliced ​​LiDAR data blocks and hyperspectral image data blocks into training set and test set.

[0044] During the partitioning process, LiDAR data blocks and hyperspectral image data blocks corresponding to the same spatial location are simultaneously partitioned into the training set or the test set to ensure that the two types of data blocks maintain a spatial correspondence when subsequently input into the joint classification network.

[0045] Step 4: Construct category text descriptions based on land cover categories. The category text descriptions include category name descriptions, color attribute descriptions, and geographic context descriptions.

[0046] Step 5: Construct a text semantic-guided space-frequency domain fusion network, and train the constructed joint classification network using the training set until the maximum number of training iterations is reached or the joint classification network meets the preset convergence condition, and then stop training to obtain the trained joint classification network.

[0047] The text semantic-guided spatial-frequency domain fusion network includes a visual feature extraction module, a spatial-frequency domain feature enhancement module, a bidirectional cross-modal fusion module, a text encoding module, a visual-text semantic alignment module, and a classification head. The visual feature extraction module includes a hyperspectral feature extraction block and a LiDAR feature extraction block, and its working process is as follows:

[0048] The dimensionality-reduced hyperspectral image data block is input into the hyperspectral feature extraction block, and hyperspectral local spatial-spectral features are extracted through a 3×3×3 three-dimensional convolutional layer, a three-dimensional batch normalization layer, and a ReLU activation function layer. Then, the output data is reconstructed to merge the channel-dimensional features and spectral-dimensional features into a two-dimensional hyperspectral feature map. The number of channels is adjusted through a two-dimensional channel projection convolutional block to obtain the hyperspectral spatial-spectral features.

[0049] The LiDAR data block is input into the LiDAR feature extraction block, and the LiDAR elevation structure features are extracted through a 3×3 two-dimensional convolutional layer, a batch normalization layer, and a ReLU activation function layer.

[0050] Hyperspectral spatial-spectral features and LiDAR elevation structure features are respectively input into the spatial-frequency domain feature enhancement module. The spatial enhancement branch enhances local boundaries, texture details and structural change information, while the frequency domain enhancement branch models frequency energy distribution and phase structure information. The two types of enhanced features are then concatenated and integrated through the channel fusion convolution module to obtain the spatial-frequency domain enhanced hyperspectral features and LiDAR features.

[0051] The hyperspectral and LiDAR features enhanced in the spatial-frequency domain are reconstructed into one-dimensional sequence features and input into a bidirectional cross-modal fusion module. The two types of features are interactively updated through bidirectional cross-attention mechanism, residual connection, layer normalization and nonlinear transformation. The fused visual features are then obtained through global aggregation and splicing.

[0052] The category text description is input into the text encoding module for semantic modeling to generate a category-level text semantic prototype. Hyperspectral features, LiDAR features and fused visual features are mapped to a unified semantic space and visual-textual alignment is performed with the category-level text semantic prototype.

[0053] During training, the classification loss is used to constrain the category discrimination result of the fused visual features, and the network is jointly optimized by hyperspectral visual-text alignment loss, LiDAR visual-text alignment loss and fused visual-text alignment loss. Finally, the fused visual features are input into the classification head to obtain the classification result.

[0054] Step 6: Use the trained joint classification network to jointly process the hyperspectral image and LiDAR data of the region to be classified to obtain the land cover classification result of the region to be classified.

[0055] Before being input into the classification network, both the hyperspectral image and LiDAR data of the region to be classified need to be preprocessed. Specifically, the hyperspectral image needs to be reduced in dimension and sliced, and the LiDAR data needs to be sliced.

[0056] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that the dimensionality reduction processing of the acquired hyperspectral image data is performed using principal component analysis.

[0057] Principal component analysis is used to reduce the spectral dimension of hyperspectral image data. This reduces the impact of redundant bands on model training and classification while preserving the main spectral discrimination information, thus lowering the computational complexity of subsequent feature extraction.

[0058] The other steps and parameters are the same as in Specific Implementation Method 1.

[0059] Specific implementation method three: Combining Figure 3 This embodiment is described below. The difference between this embodiment and specific embodiments one or two is that the visual feature extraction module operates as follows:

[0060] The visual feature extraction module includes a hyperspectral feature extraction block and a LiDAR feature extraction block;

[0061] The dimensionality-reduced hyperspectral image data block is input into the hyperspectral feature extraction block, and hyperspectral local spatial-spectral features are extracted through a 3D convolutional layer with a kernel size of 3×3×3, a 3D batch normalization layer, and a ReLU activation function layer.

[0062] The data output by the ReLU activation function layer is reconstructed to form a two-dimensional hyperspectral feature map. The channel-dimensional features and spectral-dimensional features are merged into a two-dimensional hyperspectral feature map. The number of channels is then adjusted by a two-dimensional channel projection convolution block to obtain the hyperspectral spatial-spectral features.

[0063] The LiDAR data block is input into the LiDAR feature extraction block, and the LiDAR elevation structure features are extracted through a two-dimensional convolutional layer with a kernel size of 3×3, a batch normalization layer, and a ReLU activation function layer.

[0064] Other steps and parameters are the same as in specific implementation method one or two.

[0065] The visual feature extraction module extracts hyperspectral spatial-spectral features and LiDAR elevation structure features, respectively, providing basic features for subsequent spatial-frequency domain enhancement and bidirectional cross-modal fusion.

[0066] Specific implementation method four: Combination Figure 4 This embodiment is described below. The difference between this embodiment and one of the specific embodiments one to three is that the working process of the spatial-frequency domain feature enhancement module is as follows:

[0067] Hyperspectral spatial-spectral features and LiDAR elevation structure features are input into the spatial-frequency domain feature enhancement module, respectively; for any input feature, spatial enhancement branch and frequency domain enhancement branch are constructed respectively.

[0068] In the spatial enhancement branch, the gradient responses of the input features in the horizontal and vertical directions are extracted by the Scharr edge gradient extraction layer and then weighted and fused to obtain the edge enhancement features. These edge enhancement features are then input into the first spatial convolutional module for boundary mapping, and finally added to the original input features as residuals to obtain the spatial residual enhancement features. These residual features are then input into the second spatial convolutional module for feature refinement to obtain the spatial enhancement features. Both the first and second spatial convolutional modules consist of a 3×3 two-dimensional convolutional layer, a batch normalization layer, and a SiLU activation function layer connected sequentially.

[0069] In the frequency domain enhancement branch, a two-dimensional real-valued fast Fourier transform is performed on the input features to obtain the complex form of the frequency domain features, which are then decomposed into amplitude and phase spectra. The amplitude spectrum is input into an amplitude convolution module consisting of a 3×3 two-dimensional convolutional layer, a batch normalization layer, and a SiLU activation function layer to obtain the enhanced amplitude spectrum. The phase spectrum is input into a phase spectrum refinement sub-branch consisting of a 3×3 two-dimensional convolutional layer, a batch normalization layer, and a Tanh activation function layer to obtain a phase correction term, which is multiplied by a learnable scaling parameter and then superimposed onto the original phase spectrum to obtain the enhanced phase spectrum.

[0070] The complex frequency domain features are reconstructed based on the enhanced amplitude and phase spectra, and then restored to the spatial domain using a two-dimensional inverse real-valued fast Fourier transform. The restored features are then input into a frequency domain post-processing convolution module to obtain frequency domain enhanced features. The spatial and frequency domain enhanced features are concatenated along the channel dimension, and then channel compression and cross-domain information integration are performed using a 1×1 channel fusion convolution module to obtain spatial-frequency domain enhanced features.

[0071] The above process was performed on the hyperspectral spatial-spectral features and the LiDAR elevation structure features respectively to obtain the spatially-frequency-enhanced hyperspectral features and the spatially-frequency-enhanced LiDAR features.

[0072] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0073] The spatial-frequency domain feature enhancement module can simultaneously enhance local boundaries, texture details, frequency energy distribution, and phase structure information, thereby improving the structural representation and discrimination capabilities of hyperspectral and LiDAR features.

[0074] Specific Implementation Method Five: Combining Figure 5 This embodiment is described below. The difference between this embodiment and one of the specific embodiments one to four is that the working process of the bidirectional cross-modal fusion module is as follows:

[0075] The spatial-frequency enhanced hyperspectral and LiDAR features are reconstructed into one-dimensional sequence features and then subjected to layer normalization. Hyperspectral query features and LiDAR query features are generated through a hyperspectral query mapping layer and a LiDAR query mapping layer, respectively. Key features and value features corresponding to the two modes are generated through a key-value mapping layer.

[0076] In the direction of hyperspectral feature update, LiDAR query features are used as the query end, and hyperspectral key features and hyperspectral value features are used as the key end. Cross-modal attention response is obtained through matrix multiplication, scale normalization and softmax function, and matrix multiplication is performed with hyperspectral value features to obtain hyperspectral interactive features that absorb LiDAR elevation structure information.

[0077] In the LiDAR feature update direction, hyperspectral query features are used as the query end, and LiDAR key features and LiDAR value features are used as the key end. Cross-modal attention response is obtained through matrix multiplication, scale normalization and softmax function, and matrix multiplication is performed with LiDAR value features to obtain LiDAR interactive features that absorb hyperspectral spatial-spectral information.

[0078] The hyperspectral and LiDAR interactive features are respectively mapped through the output mapping layer and residually connected with the corresponding original sequence features. Then, they are refined nonlinearly through layer normalization and multilayer perceptron to obtain the updated hyperspectral and LiDAR sequence features. Finally, global average pooling is performed on both and they are concatenated to form the fused visual features.

[0079] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0080] The bidirectional cross-modal fusion module enables complementary updates of hyperspectral and LiDAR features through feature interaction in two directions, thereby enhancing the joint representation capability of heterogeneous remote sensing data.

[0081] Specific Implementation Method Six: Combination Figure 6 This embodiment is described below. The difference between this embodiment and any one of embodiments one through five is that the working process of the text encoding module and the visual-text semantic alignment module is as follows:

[0082] A category text description is constructed based on the land feature category, and the category text description includes a category name description, a color attribute description, and a geographic context description;

[0083] The category text description is input into the text encoding module, and after word segmentation and modeling by a text encoder based on the Transformer structure, the category semantic features are obtained.

[0084] The category semantic features are mapped to a shared semantic space and normalized to obtain the category-level text semantic prototype;

[0085] The hyperspectral and LiDAR features enhanced by spatial-frequency domain are subjected to global average pooling and mapped to the shared semantic space through the corresponding projection layer to obtain hyperspectral semantic embedding features and LiDAR semantic embedding features.

[0086] The fused visual features output by the bidirectional cross-modal fusion module are mapped to the shared semantic space through the fusion feature projection layer to obtain fused semantic embedding features;

[0087] Calculate the similarity between hyperspectral semantic embedding features, LiDAR semantic embedding features, fused semantic embedding features and category-level text semantic prototypes respectively;

[0088] During training, text semantic prototypes corresponding to the real category labels are selected, and one-to-one pairing similarity matrices between hyperspectral features, LiDAR features, and fusion features and text semantic prototypes are constructed respectively.

[0089] In each similarity matrix, visual features and text semantic prototypes of the same sample corresponding to the same category are taken as positive sample pairs, and visual features and text semantic prototypes of different categories are taken as negative sample pairs.

[0090] Based on three types of similarity matrices, hyperspectral visual-text alignment loss, LiDAR visual-text alignment loss, and fused visual-text alignment loss are calculated respectively, and semantic constraints are applied to hyperspectral features, LiDAR features, and fused visual features.

[0091] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0092] The text encoding module and the visual-text semantic alignment module improve the semantic consistency and inter-class separability of hyperspectral features, LiDAR features, and fused visual features by constraining multimodal feature learning through category-level text semantic prototypes.

[0093] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods One to Six in that the working process of the classification head is as follows:

[0094] The fused visual features output from the bidirectional cross-modal fusion module are input into the fusion feature projection layer to obtain the fused projection features;

[0095] After normalizing the fused projection features, they are input into the linear classification head to obtain the classification scores corresponding to each land cover category.

[0096] The linear classification head is composed of a fully connected layer;

[0097] The land cover category label corresponding to the pixel to be classified is determined based on the maximum classification score.

[0098] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0099] The classification head is used to convert fused visual features into category scores and determine the land cover category based on the highest score; textual semantic information provides constraints during the training phase through visual-text alignment loss.

[0100] Specific Implementation Method Eight: This implementation method differs from Specific Implementation Methods One to Seven in that the training and optimization process of the joint classification network is as follows:

[0101] The cross-entropy classification loss is calculated by comparing the classification score output by the linear classification head with the true class label.

[0102] The total visual-text alignment loss is obtained by weighted summing of the hyperspectral visual-text alignment loss, the LiDAR visual-text alignment loss, and the fused visual-text alignment loss.

[0103] The joint training loss is obtained by jointly weighting the cross-entropy classification loss and the total visual-text alignment loss.

[0104] The visual feature extraction module, spatial-frequency domain feature enhancement module, bidirectional cross-modal fusion module, visual-text semantic alignment module, and classification head are jointly optimized using a joint training loss.

[0105] Training stops when the maximum number of training iterations is reached or the joint classification network meets the preset convergence condition, resulting in a well-trained joint classification network.

[0106] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0107] The joint training optimization process constrains the model through cross-entropy classification loss and visual-text alignment loss, enabling the fused features to have class discrimination capabilities and enhancing the consistency between multimodal features and the semantic prototypes of class texts.

[0108] This invention proposes a joint classification method for hyperspectral and LiDAR data based on a text semantic-guided spatial-frequency domain fusion network. The implementation process is shown in Table 1.

[0109] Table 1. Flowchart of the joint classification method based on text semantic-guided spatial-frequency domain fusion network

[0110]

[0111] The specific implementation steps are as follows:

[0112] Step 1: Acquire hyperspectral image data and LiDAR data.

[0113] Step 2: Use PCA (Principal Component Analysis) to reduce the dimensionality of the hyperspectral image data, thereby reducing spectral redundancy and computational complexity.

[0114] Step 3: Preprocess the LiDAR data and the dimensionality-reduced hyperspectral image data. Divide the data into training and test sets based on the number of labeled pixel samples, and then slice each set.

[0115] Step 4: Construct category text descriptions based on land cover categories. The category text descriptions include category name descriptions, color attribute descriptions, and geographic context descriptions, which are used to generate category-level text semantic prototypes.

[0116] Step 5: Train and classify the spatial-frequency domain fusion network based on text semantic guidance.

[0117] Step 5.1: Construct a joint classification network. The joint classification network includes a visual feature extraction module, a spatial-frequency domain feature enhancement module, a bidirectional cross-modal fusion module, a text encoding module, a visual-text semantic alignment module, and a classification head. The visual feature extraction module includes a hyperspectral feature extraction block and a LiDAR feature extraction block.

[0118] Step 5.2: In the visual feature extraction module, the dimensionality-reduced hyperspectral image data block is input into the hyperspectral feature extraction block. Local spatial-spectral features are extracted through a 3×3×3 three-dimensional convolutional layer, a 3-dimensional batch normalization layer, and a ReLU activation function layer. Then, hyperspectral spatial-spectral features are obtained through feature reconstruction and a two-dimensional channel projection convolutional block. The LiDAR data block is input into the LiDAR feature extraction block, and LiDAR elevation structure features are obtained through a 3×3 two-dimensional convolutional layer, a batch normalization layer, and a ReLU activation function layer.

[0119] Step 5.3: Input the hyperspectral spatial-spectral features and LiDAR elevation structure features into the spatial-frequency domain feature enhancement module, respectively. The spatial enhancement branch enhances local boundaries, texture details and structural change information, while the frequency domain enhancement branch models amplitude spectrum and phase spectrum information. Finally, the channel fusion convolution module is used to obtain the spatial-frequency domain enhanced hyperspectral features and LiDAR features.

[0120] Step 5.4: Reconstruct the spatially-frequency-enhanced hyperspectral features and LiDAR features into one-dimensional sequence features and input them into the bidirectional cross-modal fusion module. Through cross-modal attention calculations in two directions, the hyperspectral features and LiDAR features complement each other; subsequently, through residual connections, layer normalization, multilayer perceptron refinement, global average pooling, and stitching, fused visual features are formed.

[0121] Step 5.5: Input the category text description into the text encoding module to generate a category-level text semantic prototype; map the hyperspectral features, LiDAR features, and fused visual features to the shared semantic space, and calculate the similarity with the category-level text semantic prototype to construct the hyperspectral visual-text alignment loss, LiDAR visual-text alignment loss, and fused visual-text alignment loss.

[0122] Step 5.6: Input the fused visual features into the fused feature projection layer to obtain fused projected features; after normalization, input them into the linear classification head to obtain classification scores. During training, calculate the cross-entropy classification loss between the classification scores and the true class labels, and jointly optimize it with the three-class visual-text alignment loss until the maximum number of training iterations is reached or the preset convergence condition is met, to obtain the trained joint classification network.

[0123] Step 6: Use the trained joint classification network to jointly process the hyperspectral image and LiDAR data of the region to be classified, and determine the pixel-level land cover category label based on the maximum classification score to obtain the final classification result.

[0124] The network model training and classification result verification experiments of this invention were completed on the following platform:

[0125] The network training and classification validation of this invention were performed on a Windows system, using the PyTorch deep learning framework and an NVIDIA GPU. The experiments employed three well-known multimodal remote sensing datasets: the Trento dataset, the Augsburg dataset, and the Houston 2013 dataset. The land cover categories and sample sizes for each dataset are as follows: Figure 7 , Figure 8 and Figure 9 As shown, different colors represent different land cover categories. Detailed statistical information and feature descriptions for each dataset are as follows: Figure 10 , Figure 11 and Figure 12 As shown.

[0126] 1. The Trento dataset, collected in the province of Trento, Italy, consists of hyperspectral images and LiDAR data, and is a commonly used dataset in multimodal remote sensing classification research. The dataset has a spatial size of 600×166 pixels. The hyperspectral images contain 63 continuous spectral bands, ranging from 420.89 to 989.09 nanometers, with a spatial resolution of 1 meter. The corresponding LiDAR data is a single-channel digital surface model used to characterize surface elevation information. The dataset includes six land cover categories: apple trees, buildings, ground, trees, vineyards, and roads.

[0127] 2. The Augsburg dataset, collected in the Augsburg region of Germany, includes hyperspectral images and LiDAR digital surface model data. The hyperspectral images, acquired by the HySpex sensor, contain 180 spectral bands ranging from 0.4 to 2.5 micrometers in wavelength. The LiDAR-DSM data provides elevation structure information for the corresponding area. Both datasets underwent spatial registration and downsampling, with a unified spatial resolution of 30 meters and a spatial size of 332×485 pixels. The dataset includes seven land cover categories: forest, residential areas, industrial areas, low vegetation, gardens, commercial areas, and water bodies.

[0128] 3. The Houston2013 dataset, released in 2013 as part of the IEEE GRSS Data Fusion Competition, covers the University of Houston campus and surrounding urban area. This dataset contains spatially registered hyperspectral images and LiDAR data, both with a spatial size of 349×1905 pixels and a spatial resolution of 2.5 meters. The hyperspectral images contain 144 spectral bands, ranging from 0.38 to 1.05 micrometers; the LiDAR data is single-band elevation structure data. This dataset includes 15 categories of urban land cover and can be used to validate the joint classification performance of methods in complex urban remote sensing scenarios.

[0129] In evaluating model performance, this embodiment employs three widely recognized and commonly used objective evaluation metrics in the industry: Overall Accuracy (OA), Average Accuracy (AA), and Kappa Coefficient (K). These metrics are all calculated based on the Confusion Matrix, which can comprehensively and objectively reflect the accuracy and reliability of the model in classification tasks.

[0130] The following sections will introduce these evaluation indicators in detail:

[0131] (1) Confusion Matrix: The confusion matrix is ​​an error matrix used to evaluate the performance of a classification model. It reflects the correspondence between the predicted class and the true class. The elements in the matrix represent the correct and incorrect classification of samples of different classes. It can intuitively show the degree of confusion between classes and is an important basis for calculating evaluation indicators such as overall classification accuracy, average classification accuracy and Kappa coefficient.

[0132] Table 2. Composition of the confusion matrix

[0133]

[0134] (2) Overall Classification Accuracy (OA): OA is a fundamental and crucial metric in evaluating classification models, used to measure the proportion of correctly classified samples out of the total number of samples. It reflects the accuracy of the model in classifying all categories globally, and is specifically calculated as the ratio of correctly classified samples to the total number of samples. Overall classification accuracy can intuitively reflect the overall performance of the classification algorithm and is an important reference for evaluating the model's classification effect.

[0135]

[0136] Where: TP (True Positive): Number of samples correctly predicted as positive. TN (True Negative): Number of samples correctly predicted as negative. FN (False Negative): Number of samples incorrectly predicted as negative. FP (False Positive): Number of samples incorrectly predicted as positive.

[0137] (3) Average Classification Accuracy (AA): AA measures the model's ability to classify land cover in a balanced manner across different categories. It is calculated by first counting the proportion of correctly classified samples in each category, and then averaging the classification accuracy across all categories. Unlike overall classification accuracy, AA focuses more on the model's consistency across different categories and reflects its ability to identify minority or easily confused categories.

[0138]

[0139] Where: n represents the number of rows and columns of the classification matrix, This represents the number of samples correctly classified into the i-th category. This represents the number of samples that were misclassified in the i-th category.

[0140] (4) Kappa coefficient (K): The Kappa coefficient is an important indicator for measuring classification accuracy, used to evaluate the degree of improvement of the classification results compared to completely random classification. It reflects the actual accuracy of the classification model by quantifying the difference between the actual classification results and the random classification results. As a supplementary indicator to the confusion matrix, the Kappa coefficient can more intuitively reveal the quality of classification accuracy:

[0141]

[0142] Where: OA (Overall Accuracy): Overall classification accuracy; PE (Probability of Random Agreement): Expected accuracy of random classification, calculated using the following formula:

[0143]

[0144] Where: N is the total number of samples. This represents the number of samples correctly classified into the i-th category. This represents the number of samples that were incorrectly predicted as positive for the i-th category. This represents the number of samples that were incorrectly predicted as negative for the i-th category.

[0145] The objective classification results of the network model used in this invention on three datasets are shown in Table 3. From left to right, the first column is the dataset; the second column is the OA data of the classification results; the third column is the AA data of the classification results; and the fourth column is the K × 100 data of the classification results. The subjective classification results are as follows... Figure 13 As shown, (a) represents the classification effect of the Trento dataset, (b) represents the classification effect of the Augsburg dataset, and (c) represents the classification effect of the Houston2013 dataset. The higher the accuracy, the less salt-and-pepper noise in the image.

[0146] Table 3

[0147]

[0148] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A joint classification method for hyperspectral and LiDAR data based on a text semantic-guided spatial-frequency domain fusion network, characterized in that, The method specifically includes the following steps: Step 1: Obtain LiDAR data and hyperspectral image data from the dataset; Step 2: Perform dimensionality reduction processing on the acquired hyperspectral image data to obtain dimensionality-reduced hyperspectral image data; Step 3: Slice the LiDAR data and the dimensionality-reduced hyperspectral image data to obtain LiDAR data blocks and hyperspectral image data blocks. Then divide the sliced ​​LiDAR data blocks and hyperspectral image data blocks into training set and test set. Step 4: Construct category text descriptions based on land cover categories. The category text descriptions include category name descriptions, color attribute descriptions, and geographic context descriptions. Step 5: Construct a text semantic-guided space-frequency domain fusion network, and train the constructed joint classification network using the training set until the maximum number of training iterations is reached or the joint classification network meets the preset convergence condition, and then stop training to obtain the trained joint classification network. The text semantic-guided spatial-frequency domain fusion network includes a visual feature extraction module, a spatial-frequency domain feature enhancement module, a bidirectional cross-modal fusion module, a text encoding module, a visual-text semantic alignment module, and a classification head. The visual feature extraction module includes a hyperspectral feature extraction block and a LiDAR feature extraction block, and its working process is as follows: The dimensionality-reduced hyperspectral image data block is input into the hyperspectral feature extraction block, and hyperspectral local spatial-spectral features are extracted through a 3×3×3 three-dimensional convolutional layer, a three-dimensional batch normalization layer, and a ReLU activation function layer. Then, the output data is reconstructed to merge the channel-dimensional features and spectral-dimensional features into a two-dimensional hyperspectral feature map. The number of channels is adjusted through a two-dimensional channel projection convolutional block to obtain the hyperspectral spatial-spectral features. The LiDAR data block is input into the LiDAR feature extraction block, and the LiDAR elevation structure features are extracted through a 3×3 two-dimensional convolutional layer, a batch normalization layer, and a ReLU activation function layer. Hyperspectral spatial-spectral features and LiDAR elevation structure features are respectively input into the spatial-frequency domain feature enhancement module. The spatial enhancement branch enhances local boundaries, texture details and structural change information, while the frequency domain enhancement branch models frequency energy distribution and phase structure information. The two types of enhanced features are then concatenated and integrated through the channel fusion convolution module to obtain the spatial-frequency domain enhanced hyperspectral features and LiDAR features. The hyperspectral and LiDAR features enhanced in the spatial-frequency domain are reconstructed into one-dimensional sequence features and input into a bidirectional cross-modal fusion module. The two types of features are interactively updated through bidirectional cross-attention mechanism, residual connection, layer normalization and nonlinear transformation. The fused visual features are then obtained through global aggregation and splicing. The category text description is input into the text encoding module for semantic modeling to generate a category-level text semantic prototype. Hyperspectral features, LiDAR features and fused visual features are mapped to a unified semantic space and visual-textual alignment is performed with the category-level text semantic prototype. During training, classification loss is used to constrain the category discrimination results of fused visual features, and hyperspectral visual-text alignment loss, LiDAR visual-text alignment loss and fused visual-text alignment loss are combined to optimize the network. Finally, the fused visual features are input into the classification head. After fused feature projection, normalization, and linear classification, the classification scores corresponding to each object category are obtained, and the classification results are determined based on the classification scores. Step 6: Use the trained joint classification network to jointly process the hyperspectral image and LiDAR data of the region to be classified to obtain the land cover classification result of the region to be classified.

2. The method for joint classification of hyperspectral and LiDAR data based on a text semantic-guided spatial-frequency domain fusion network according to claim 1, characterized in that, The dimensionality reduction processing of the acquired hyperspectral image data is performed using principal component analysis.

3. The method for joint classification of hyperspectral and LiDAR data based on a text semantic-guided spatial-frequency domain fusion network according to claim 2, characterized in that, The visual feature extraction module includes a hyperspectral feature extraction block and a LiDAR feature extraction block. The working process of the visual feature extraction module is as follows: Input the dimension-reduced hyperspectral image data block into the hyperspectral feature extraction block; In the hyperspectral feature extraction block, the hyperspectral image data block is input into a three-dimensional convolutional layer with a kernel size of 3×3×3 to extract the local spatial-spectral features of the hyperspectral image data; The output of the 3D convolutional layer is used as the input of the 3D batch normalization layer, and then the output of the 3D batch normalization layer is used as the input of the ReLU activation function layer. The data output by the ReLU activation function layer is reconstructed, and the channel-dimensional features and spectral-dimensional features obtained by the three-dimensional convolution are merged to obtain a two-dimensional hyperspectral feature map. The two-dimensional hyperspectral feature map is input into a two-dimensional channel projection convolution block, and the number of channels in the two-dimensional hyperspectral feature map is adjusted to obtain hyperspectral spatial-spectral features. Input the LiDAR data block into the LiDAR feature extraction block; In the LiDAR feature extraction block, the LiDAR data block is input into a two-dimensional convolutional layer with a kernel size of 3×3 to extract the local elevation structure features of the LiDAR data. The output of the two-dimensional convolutional layer is used as the input of the batch normalization layer, and then the output of the batch normalization layer is used as the input of the ReLU activation function layer to obtain the LiDAR elevation structure features.

4. The method for joint classification of hyperspectral and LiDAR data based on a text semantic-guided spatial-frequency domain fusion network according to claim 3, characterized in that, The working process of the space-frequency domain feature enhancement module is as follows: Hyperspectral spatial-spectral features and LiDAR elevation structure features are respectively input into the spatial-frequency domain feature enhancement module; For any modal feature of the input spatial-frequency feature enhancement module, the spatial-frequency feature enhancement module includes a spatial enhancement branch, a frequency enhancement branch, and a channel fusion convolution module; The spatial enhancement branch includes a Scharr edge gradient extraction layer, a first spatial convolution module, and a second spatial convolution module; The Scharr edge gradient extraction layer includes a horizontal Scharr convolutional layer and a vertical Scharr convolutional layer. Both the horizontal and vertical Scharr convolutional layers adopt a channel-wise convolutional structure with a kernel size of 3×3, and the kernel parameters are fixed and do not participate in training. Both the first spatial convolution module and the second spatial convolution module are composed of a two-dimensional convolutional layer with a kernel size of 3×3, a batch normalization layer, and a SiLU activation function layer connected in sequence; In the spatial augmentation branch, the input features are fed into the horizontal Scharr convolutional layer and the vertical Scharr convolutional layer respectively to obtain the horizontal gradient features and the vertical gradient features. The horizontal and vertical gradient features are weighted and fused to obtain edge enhancement features. The edge enhancement features are input into the first spatial convolution module for boundary feature mapping to obtain the boundary mapping features. The boundary mapping features are added to the original features of the input space-frequency domain feature enhancement module by residual addition to obtain the spatial residual enhancement features; The spatial residual enhancement features are input into the second spatial convolution module for feature refinement to obtain the spatial enhancement features; The frequency domain enhancement branch includes an amplitude spectrum enhancement sub-branch, a phase spectrum refinement sub-branch, and a frequency domain post-processing convolution module; In the frequency domain enhancement branch, the input features are first subjected to a two-dimensional real-valued fast Fourier transform to obtain the frequency domain features in complex form; The frequency domain features in complex form are decomposed into amplitude spectrum and phase spectrum, where amplitude spectrum is used to characterize frequency energy distribution and texture intensity, and phase spectrum is used to characterize spatial structure, positional layout and geometric shape. The amplitude spectrum enhancement sub-branch includes a first amplitude convolution module and a second amplitude convolution module. Both the first amplitude convolution module and the second amplitude convolution module are composed of a two-dimensional convolutional layer with a kernel size of 3×3, a batch normalization layer, and a SiLU activation function layer connected in sequence. The amplitude spectrum is input into the first amplitude convolution module to obtain the initial amplitude enhancement features; The initial amplitude enhancement features are added to the original amplitude spectrum by residual addition, and then input into the second amplitude convolution module to obtain the enhanced amplitude spectrum; The phase spectrum refinement sub-branch is composed of a two-dimensional convolutional layer with a kernel size of 3×3, a batch normalization layer, and a Tanh activation function layer connected in sequence; Input the phase spectrum into the phase spectrum refinement sub-branch to obtain the phase correction term; The phase correction term is multiplied by the learnable scaling parameter and then superimposed onto the original phase spectrum to obtain the enhanced phase spectrum; The complex frequency domain features are reconstructed based on the enhanced amplitude spectrum and the enhanced phase spectrum; A two-dimensional inverse real-valued fast Fourier transform is performed on the reconstructed complex frequency domain features to restore the frequency domain features to the spatial domain; The frequency domain features restored to the spatial domain are input into the frequency domain post-processing convolution module to obtain frequency domain enhanced features; The frequency domain post-processing convolution module consists of a two-dimensional convolutional layer with a kernel size of 3×3, a batch normalization layer, and a SiLU activation function layer connected in sequence. Spatial enhancement features and frequency domain enhancement features are concatenated along the channel dimension to obtain concatenated enhancement features; The splicing enhancement feature input channel fusion convolution module performs channel compression and cross-domain information integration to obtain spatial-frequency domain enhanced features; The channel fusion convolution module consists of a two-dimensional convolutional layer with a kernel size of 1×1, a batch normalization layer, and a SiLU activation function layer connected in sequence. The above spatial-frequency domain enhancement process was performed on the hyperspectral spatial-spectral features and the LiDAR elevation structure features respectively to obtain the spatial-frequency domain enhanced hyperspectral features and the spatial-frequency domain enhanced LiDAR features.

5. The method for joint classification of hyperspectral and LiDAR data based on a text semantic-guided spatial-frequency domain fusion network according to claim 4, characterized in that, The working process of the bidirectional cross-modal fusion module is as follows: The spatial-frequency enhanced hyperspectral features and spatial-frequency enhanced LiDAR features were reconstructed into one-dimensional sequence features, respectively. Layer normalization was performed on the hyperspectral sequence features and LiDAR sequence features respectively; Hyperspectral sequence features are processed through a hyperspectral query mapping layer to generate hyperspectral query features; The LiDAR sequence features are processed through the LiDAR query mapping layer to generate LiDAR query features; The LiDAR sequence features are processed through a key-value mapping layer to generate LiDAR bond features and LiDAR value features; Hyperspectral sequence features are processed through a key-value mapping layer to generate hyperspectral bond features and hyperspectral value features; In the direction of hyperspectral feature update, LiDAR query features are used as the query end, and hyperspectral key features and hyperspectral value features are used as the key end. Cross-modal attention response is obtained through matrix multiplication, scale normalization and softmax function. Then, the cross-modal attention response and hyperspectral value features are multiplied by matrix to obtain hyperspectral interactive features that absorb LiDAR elevation structure information. In the direction of LiDAR feature update, hyperspectral query features are used as the query end, and LiDAR key features and LiDAR value features are used as the key end. Cross-modal attention response is obtained through matrix multiplication, scale normalization and softmax function. Then, matrix multiplication operation is performed between cross-modal attention response and LiDAR value features to obtain LiDAR interactive features that absorb hyperspectral spatial-spectral information. Hyperspectral interactive features and LiDAR interactive features are respectively mapped through the output mapping layer; The mapped hyperspectral interactive features are residually connected with the original hyperspectral sequence features, and nonlinear feature refinement is performed through layer normalization and multilayer perceptron to obtain the updated hyperspectral sequence features. The mapped LiDAR interactive features are residually connected with the original LiDAR sequence features, and nonlinear feature refinement is performed through layer normalization and multilayer perceptron to obtain the updated LiDAR sequence features. Global average pooling is performed on the updated hyperspectral sequence features and LiDAR sequence features respectively, and then the resulting global hyperspectral features and LiDAR features are concatenated to form a fused visual feature.

6. The method for joint classification of hyperspectral and LiDAR data based on a text semantic-guided spatial-frequency domain fusion network according to claim 5, characterized in that, The text encoder works as follows: A category text description is constructed based on the land feature category, and the category text description includes a category name description, a color attribute description, and a geographic context description; The category text description is input into a text encoder for word segmentation, resulting in a text tag sequence; The text tag sequence is input into a Transformer-based text encoder for contextual semantic modeling. Extract the category semantic features from the text encoder output and map the category semantic features to the shared semantic space; The mapped category semantic features are normalized to obtain the category-level text semantic prototype; Category-level text semantic prototypes are used as semantic anchors to guide the semantic alignment of hyperspectral features, LiDAR features, and fused visual features.

7. The method for joint classification of hyperspectral and LiDAR data based on a text semantic-guided spatial-frequency domain fusion network according to claim 6, characterized in that, The working process of the visual-text semantic alignment module is as follows: The hyperspectral features enhanced in the spatial-frequency domain are subjected to global average pooling and then mapped to the shared semantic space through a hyperspectral projection layer to obtain hyperspectral semantic embedding features. The spatial-frequency domain enhanced LiDAR features are subjected to global average pooling and then mapped to the shared semantic space through a LiDAR projection layer to obtain LiDAR semantic embedding features. The fused visual features output by the bidirectional cross-modal fusion module are mapped to the shared semantic space through the fusion feature projection layer to obtain fused semantic embedding features; Normalization was performed on hyperspectral semantic embedding features, LiDAR semantic embedding features, fused semantic embedding features, and category-level text semantic prototypes respectively. The similarity between hyperspectral semantic embedding features and category-level text semantic prototypes, the similarity between LiDAR semantic embedding features and category-level text semantic prototypes, and the similarity between fused semantic embedding features and category-level text semantic prototypes are calculated respectively. During training, based on the true category labels of samples within the same training batch, a corresponding category text semantic prototype is selected for each sample, and a one-to-one pairing similarity matrix between hyperspectral features and text semantic prototype, a one-to-one pairing similarity matrix between LiDAR features and text semantic prototype, and a one-to-one pairing similarity matrix between fused features and text semantic prototype are constructed. In each one-to-one similarity matrix, the visual semantic embedding features and text semantic prototypes of the same sample corresponding to the same category are taken as positive sample pairs, and the visual semantic embedding features and text semantic prototypes of different samples are taken as negative sample pairs. Based on the one-to-one pairing similarity matrix between hyperspectral features and text semantic prototypes, the hyperspectral visual-text alignment loss is calculated. Based on the one-to-one pairing similarity matrix between LiDAR features and text semantic prototypes, calculate the LiDAR visual-text alignment loss; Based on the one-to-one pairing similarity matrix between fused features and text semantic prototypes, the fused visual-text alignment loss is calculated. The hyperspectral visual-text alignment loss, LiDAR visual-text alignment loss, and fused visual-text alignment loss all include alignment loss from the visual direction to the text direction and alignment loss from the text direction to the visual direction. Multi-branch semantic constraints are applied to hyperspectral features, LiDAR features, and fused visual features using the hyperspectral visual-text alignment loss, LiDAR visual-text alignment loss, and fused visual-text alignment loss.

8. The method for joint classification of hyperspectral and LiDAR data based on a text semantic-guided spatial-frequency domain fusion network according to claim 7, characterized in that, The working process of the classification head is as follows: The fused visual features output from the bidirectional cross-modal fusion module are input into the fusion feature projection layer to obtain the fused projection features; Normalize the fused projection features; The normalized fused projection features are input into the linear classification head to obtain the classification scores corresponding to each land cover category; The linear classification head is composed of a fully connected layer; The land cover category label corresponding to the pixel to be classified is determined based on the maximum classification score.

9. The method for joint classification of hyperspectral and LiDAR data based on a text semantic-guided spatial-frequency domain fusion network according to claim 8, characterized in that, The training and optimization process of the joint classification network is as follows: The cross-entropy classification loss is calculated by comparing the classification score output by the linear classification head with the true class label. The total visual-text alignment loss is obtained by weighted summing of the hyperspectral visual-text alignment loss, the LiDAR visual-text alignment loss, and the fused visual-text alignment loss. The joint training loss is obtained by jointly weighting the cross-entropy classification loss and the total visual-text alignment loss. The visual feature extraction module, spatial-frequency domain feature enhancement module, bidirectional cross-modal fusion module, visual-text semantic alignment module, and classification head are jointly optimized using a joint training loss. Training stops when the maximum number of training iterations is reached or the joint classification network meets the preset convergence condition, resulting in a well-trained joint classification network.

Images