Multi-source remote sensing image classification method fusing frequency domain attention mechanism and cross-modal Transform

Abstract

The invention discloses a multi-source remote sensing image classification method fusing a frequency domain attention mechanism and a cross-modal Transform, and relates to the technical field of remote sensing image processing. The method comprises the following steps: performing frequency domain enhancement on a hyperspectral image through a frequency domain attention mechanism; semantic enhancement is carried out on the laser radar image through depth separable convolution operation and a multi-head self-attention mechanism; respectively extracting deep semantic features corresponding to the two modal enhancement features through a Transform encoder; respectively weighting the deep semantic features of the two modals through a channel attention branch and a space attention branch, and performing multi-level feature fusion by using a learnable weight to obtain a fusion output feature; and aggregating and fusing the global semantic vectors of the output features in two modes of global average pooling and attention pooling, splicing the two global semantic vectors, and mapping the spliced global semantic vectors to a category space to obtain a ground feature classification result of the multi-source remote sensing image. According to the method, the classification precision under the conditions of complex city scenes and small samples is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image processing, and particularly relates to a multi-source remote sensing image classification method fusing a frequency domain attention mechanism and a cross-modal Transformer. BACKGROUND

[0002] In recent years, joint classification of multi-source remote sensing data has become an important research direction of remote sensing interpretation. Joint classification of multi-source remote sensing data (such as hyperspectral images (HSI) and LiDAR images) can significantly improve the accuracy and robustness of feature recognition by integrating the complementary information of different modal data, and has important application value in the fields of agriculture, forestry, urban planning, etc. The introduction of deep learning technology, especially convolutional neural networks (CNN), self-attention mechanisms (Self-Attention) and visual Transformers (ViT), has greatly promoted the development of multi-modal fusion methods and achieved remarkable results in land cover classification.

[0003] However, the existing multi-modal fusion methods have the following problems: (1) hyperspectral images have the phenomena of "same spectrum different objects" and "different objects same spectrum", and spectral interference is significant; (2) there is a modal imbalance problem between LiDAR auxiliary data and HSI data, and the feature expression dimension difference is large; (3) the existing feature fusion strategy is insufficient for modeling cross-modal context information, and it is difficult to fully utilize the complementarity between modalities, especially under complex scene and small sample conditions. The above problems result in low accuracy of multi-source remote sensing image classification. SUMMARY

[0004] Therefore, it is necessary to provide a multi-source remote sensing image classification method fusing a frequency domain attention mechanism and a cross-modal Transformer in view of the above technical problems.

[0005] The present specification adopts the following technical solutions: The present specification provides a multi-source remote sensing image classification method fusing a frequency domain attention mechanism and a cross-modal Transformer, comprising: obtaining a hyperspectral image and a LiDAR image of a target scene; performing frequency decomposition and weighted fusion of the spectral dimension and the spatial dimension of the hyperspectral image through a frequency domain attention mechanism to obtain hyperspectral features enhanced in the frequency domain; performing semantic enhancement on the LiDAR image through a depth separable convolution operation and a multi-head self-attention mechanism to obtain LiDAR features enhanced in the semantic domain; extract deep semantic features of the frequency domain enhanced hyperspectral features and the semantic enhanced lidar features respectively by the Transformer encoder; For the deep semantic features of any one modality, the deep semantic features are respectively weighted by channel attention and spatial attention to obtain channel enhanced features and spatial enhanced features, and the deep semantic features, the channel enhanced features and the spatial enhanced features are weighted by learnable weights, and the weighted features are convolved and added to the deep semantic features to obtain fusion features; The fusion features corresponding to the spectral modality and the fusion features corresponding to the lidar modality are added to obtain fusion output features; Global semantic vectors of the fusion output features are aggregated by global average pooling and attention pooling, and the two global semantic vectors obtained are spliced; The spliced global semantic vectors are mapped to the category space by a multilayer perceptron to obtain the ground object classification result of the multi-source remote sensing image.

[0006] Optionally, the hyperspectral image and the lidar image of the target scene are obtained, comprising: An initial hyperspectral image and an initial lidar image of the target scene are obtained; The initial hyperspectral image is subjected to principal component analysis to reduce its spectral dimension to a preset number; The spatial resolution of the initial hyperspectral image after adjusting the dimension is adjusted to be consistent with the spatial resolution of the initial lidar image, and the adjusted initial hyperspectral image and the initial lidar image are uniformly cropped to a fixed size image; The cropped initial hyperspectral image is determined as the hyperspectral image of the target scene, and the cropped initial lidar image is determined as the lidar image of the target scene.

[0007] Optionally, the ground object classification of the multi-source remote sensing image is realized by a multi-source remote sensing classification network model; The multi-source remote sensing classification network model comprises a frequency domain attention module, a deep separable convolution cross-modal attention module, two parallel Transformer encoders, a double-branch fusion attention module and a classification head; The frequency domain attention module is used for frequency decomposition and weighted fusion of spectral dimension and spatial dimension of the hyperspectral image to obtain frequency domain enhanced hyperspectral features; The deep separable convolution cross-modal attention module is used for extracting local structure features by deep separable convolution operation and modeling cross-modal context by multi-head attention mechanism to realize semantic enhancement of the lidar image; Two parallel Transformer encoders are used to capture long-range dependencies within a mode, extracting deep semantic features from frequency-domain enhanced hyperspectral features and semantically enhanced lidar features, respectively. The dual-branch fusion attention module is used to perform channel attention and spatial attention weighting on the deep semantic features output by the two Transformer encoders, and to perform multi-level fusion through learnable weights to obtain fused output features; The classification head is used to map the fused output features to land cover category probabilities.

[0008] Optionally, the frequency domain attention module includes a local residual fusion module and a frequency domain gating module. It performs frequency decomposition and weighted fusion of the hyperspectral image in both spectral and spatial dimensions using a frequency domain attention mechanism to obtain frequency-enhanced hyperspectral features, including: The hyperspectral image is input into the frequency domain attention module, and the hyperspectral image is initially enhanced by the local residual fusion module to obtain the preliminarily enhanced features; In the frequency domain gating module, the initially enhanced features are subjected to Fast Fourier Transform in both the spectral and spatial dimensions to convert them to the frequency domain and obtain frequency domain features. For any dimension, the low-frequency vector of the frequency domain feature is determined by using a preset low-frequency mask and a learnable weight vector, and the high-frequency vector of the frequency domain feature is determined by using a preset low-frequency mask; the low-frequency and high-frequency components are subjected to inverse Fourier transform to convert them back to the spatial domain; the low-frequency and high-frequency features in the spatial domain are adaptively fused through a gating mechanism to obtain the frequency domain enhanced features. By performing residual connection between the frequency domain enhanced features corresponding to the spectral and spatial dimensions and the features after preliminary enhancement, the hyperspectral features after frequency domain enhancement are obtained.

[0009] Optionally, the local residual fusion module includes two 1×1 convolutional layers and a 3×3 convolutional layer; the local residual fusion module performs preliminary enhancement on the hyperspectral image to obtain the preliminarily enhanced features, including: The hyperspectral image is convolved through two 1×1 convolutional layers to obtain the first convolutional feature and the second convolutional feature. The first convolutional feature and the second convolutional feature are added together to obtain the third feature, and the second convolutional feature is convolved through a 3×3 convolutional layer to obtain the fourth feature; The third and fourth features are added together to obtain the preliminarily enhanced features.

[0010] Optionally, semantic enhancement is performed on the LiDAR image through depthwise separable convolution operations and multi-head self-attention mechanisms to obtain semantically enhanced LiDAR features, including: The LiDAR image is input into a depth-separable convolutional cross-modal attention module, and the LiDAR image is channel-expanded through 1×1 convolution; The expanded features are subjected to depthwise separable convolution operations to generate query vectors, key vectors, and value vectors. After flattening the query vector, key vector, and value vector into a sequence, a multi-head self-attention mechanism is used to calculate cross-space augmentation features, and a 1×1 convolution is used to map the cross-space augmentation features and query vector to the corresponding output dimension. The mapped cross-space enhanced features and the query vector are added element by element to obtain the semantically enhanced LiDAR features.

[0011] Optionally, deep semantic features of frequency-domain enhanced hyperspectral features and semantically enhanced LiDAR features are extracted separately using a Transformer encoder, including: The frequency-domain enhanced hyperspectral features and semantically enhanced lidar features are flattened into sequences and then positional encoding is added. These sequences are then input into two independent and parallel Transformer encoders. In each Transformer encoder, the long-range dependencies of the features of the corresponding modes are captured through a multi-layer self-attention mechanism and a feedforward network, thereby obtaining the deep semantic features corresponding to the spectral modes and the lidar modes.

[0012] Optionally, the dual-branch fusion attention module includes a channel attention branch and a spatial attention branch; the implementation process of the dual-branch fusion attention module includes: Deep semantic features of spectral modes Deep semantic features of lidar modes Input is fed into the dual-branch fusion attention module; In the channel attention branch, two deep semantic features are addressed respectively. and Perform global average pooling to obtain channel statistics. and ,Will and After concatenation, the joint embedding features are obtained by sequentially applying 1x1 convolution and SiLU activation function. ;according to Channel attention weights are generated for each modality, and the deep semantic features of each modality are channel-weighted based on the channel attention weights to obtain the channel-enhanced features of each modality. In the spatial attention branch, two deep semantic features are addressed respectively. and The spatial response map is obtained by performing a 3×3 convolution. and ,Will and After concatenation, the joint spatial attention map is obtained by sequentially applying 1×1 convolution, batch normalization, and the sigmoid function. And through joint spatial attention maps Spatially weight the deep semantic features of each modality to obtain the spatial augmentation features of each modality. ; For any modality, deep semantic features, channel enhancement features, and spatial enhancement features are weighted by a learnable fusion weight vector. The weighted features are then processed by 3×3 convolution and normalization and residually connected with the corresponding deep semantic features to obtain the fusion features. The fusion features corresponding to the spectral modes and the fusion features corresponding to the lidar modes are added together to obtain the fusion output features.

[0013] Optionally, the classification head includes a global average pooling layer, an attention pooling layer, and a multilayer perceptron; the implementation process of the classification head includes: The fused output features are input into the classification head, and the fused output features are subjected to global average pooling through a global average pooling layer to obtain the first global semantic vector. The fused output features are then subjected to attention pooling through an attention pooling layer to obtain the second global semantic vector. The first and second global semantic vectors are concatenated and then input into a multilayer perceptron to obtain the land cover classification results of the multi-source remote sensing image; the multilayer perceptron includes layer normalization and fully connected layers.

[0014] This specification provides a multi-source remote sensing image classification device that integrates a frequency domain attention mechanism and a cross-modal Transformer, including: The acquisition module is used to acquire hyperspectral images and lidar images of the target scene; The classification module performs frequency decomposition and weighted fusion of the hyperspectral image in both spectral and spatial dimensions using a frequency-domain attention mechanism to obtain frequency-enhanced hyperspectral features. It then performs semantic enhancement on the LiDAR image using depthwise separable convolution and a multi-head self-attention mechanism to obtain semantically enhanced LiDAR features. A Transformer encoder extracts deep semantic features from both the frequency-enhanced hyperspectral features and the semantically enhanced LiDAR features. For any modality's deep semantic features, channel attention and spatial attention weighting are applied to obtain channel-enhanced features and... Spatial augmentation features are generated, and deep semantic features, channel augmentation features, and spatial augmentation features are weighted using learnable weights. The weighted features are then convolved and added to the deep semantic features to obtain fused features. The fused features corresponding to spectral modes and LiDAR modes are added to obtain fused output features. The global semantic vectors of the fused output features are aggregated using both global average pooling and attention pooling, and the two global semantic vectors are concatenated. Finally, a multilayer perceptron maps the concatenated global semantic vectors to the category space to obtain the land cover classification results of the multi-source remote sensing image.

[0015] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multi-source remote sensing image classification method that combines a fused frequency domain attention mechanism with a cross-modal Transformer.

[0016] This specification provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described multi-source remote sensing image classification method that combines the frequency domain attention mechanism with the cross-modal Transformer.

[0017] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects: In this invention, a frequency domain attention mechanism is used to decompose and weightedly fuse hyperspectral images in both spectral and spatial dimensions. This preserves low-frequency global structure and high-frequency details, achieving feature enhancement. Semantic enhancement of LiDAR images is achieved through depthwise separable convolution and multi-head self-attention, yielding semantically enhanced LiDAR features. Subsequently, the two enhanced features are processed by a Transformer encoder to capture intramodal long-range dependencies and enhance semantic representation. Deep semantic features are extracted from the frequency-enhanced hyperspectral features and the semantically enhanced LiDAR features. These deep semantic features from both modalities are weighted using channel attention and spatial attention to obtain channel-enhanced features and spatial-enhanced features. Learnable weights are then used to weight these deep semantic features, channel-enhanced features, and spatial-enhanced features. The weighted features are then convolved and added to the deep semantic features to improve cross-modal complementarity. The fused output features are sequentially processed through global average pooling and attention pooling to obtain a global semantic vector. Finally, a multilayer perceptron jointly utilizes different aggregated information to predict land cover categories. This method improves classification accuracy in complex urban scenes and under small sample conditions. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This document provides a flowchart illustrating a multi-source remote sensing image classification method that integrates a frequency domain attention mechanism and a cross-modal Transformer. Figure 2 This is a schematic diagram of the structure of a multi-source remote sensing classification network model provided in this specification; Figure 3 This document provides a schematic diagram illustrating the implementation process of a frequency domain attention module, FreqAttnBlock. Figure 4 This document provides a schematic diagram illustrating the implementation process of a frequency-domain gating module, FreqGate Block. Figure 5 This is a schematic diagram illustrating the spectral values ​​of a different category of objects provided in this specification. Figure 6 This document provides a schematic diagram illustrating the implementation process of a depthwise separable convolutional cross-modal attention module, DWConvCrossBlock. Figure 7 This document provides a schematic diagram illustrating the implementation process of a Transformer encoder. Figure 8This is a schematic diagram of the structure of a Transformer encoder provided in this specification; Figure 9 This document provides a flowchart of the implementation of a dual-branch fusion attention module, DuFuseAttention. Figure 10 This specification provides a flowchart for the implementation of channel attention branch and spatial attention branch. Figure 11 This is a schematic diagram of the structure of a sorting head provided in this specification; Figure 12 A schematic diagram of another multi-source remote sensing image classification method that integrates frequency domain attention mechanism and cross-modal Transformer, as provided in this specification; Figure 13 A comparison chart of classification results with other methods on the Berlin dataset provided in this embodiment of the invention; Figure 14 This is a schematic diagram of a computer device used in this specification to implement a multi-source remote sensing image classification method that integrates a frequency domain attention mechanism and a cross-modal Transformer. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this application.

[0021] In existing technologies, a remote sensing image fusion classification method based on multimodal feature alignment has been proposed, aiming to address the problems of feature misalignment and insufficient sample utilization in the fusion classification of hyperspectral imagery and LiDAR data. This method constructs a multimodal Transformer network called NMF2AMT, which comprises a hyperspectral feature extraction sub-network (NCH), a LiDAR feature extraction sub-network (NCL), a feature enhancement sub-network (NTE), a feature fusion sub-network (NTF), and a classification sub-network (Ncls). Specifically, NCH and NCL consist of multiple convolutional layers used for layer-by-layer convolution, normalization, and activation to extract local features; NTE further enhances feature representation through multi-head self-attention and forward propagation layers; NTF utilizes an improved cross-attention mechanism to achieve cross-modal feature fusion; and Ncls is responsible for the classification output. During the training phase, the method standardizes and blocks the hyperspectral and LiDAR samples, forming triplet inputs centered on pixels. These inputs are then processed through convolutional sub-networks and positional encoding to obtain initial features, which are then realized through the Transformer structure to achieve multimodal feature interaction and alignment. The patent designs a feature matching module based on the most difficult negative sample contrastive learning to improve discrimination ability in small sample and multimodal scenarios. It also effectively alleviates class imbalance by jointly optimizing weighted cross-entropy and contrastive loss. During the inference phase, the network extracts fused features from the tested hyperspectral and LiDAR images and outputs pixel-level classification results. The overall method combines the advantages of convolutional neural networks in local feature modeling with the ability of Transformers in long-range dependency modeling. Through feature alignment and hard sample mining, it improves cross-modal learning performance, providing a feasible technical approach for high-precision fusion classification of remote sensing multimodal data.

[0022] In another approach, a classification method based on multimodal remote sensing image fusion is proposed. Addressing the limitations of single-modality methods in representation and discrimination, this method leverages the complementary information from hyperspectral imagery, LiDAR data, and multispectral imagery to construct a complete workflow encompassing preprocessing, feature extraction, feature fusion, and classification output. In the data processing stage, principal component analysis and normalization are performed on hyperspectral imagery; LiDAR data is denoised and interpolated to generate a digital elevation model; and multispectral imagery's spectral features are enhanced using normalized vegetation index and radiometric correction. Spatial registration is achieved by integrating open-source map data. The feature extraction part employs 3D convolutional neural networks, 2D convolutional neural networks, and graph convolutional network structures to deeply model the spectral, spatial, and structural information of different modalities, and unifies the feature space through dimensionality reduction. To improve cross-modal interaction, this method introduces an attention mechanism and a multimodal Transformer in the fusion stage to model local and global dependencies, achieving weighted fusion of heterogeneous features. Finally, a softmax classifier is used to output pixel-level classification results. During training, a joint optimization of cross-entropy and multimodal contrastive loss is introduced to improve the model's performance in imbalanced classes and small sample sizes. This method fully combines the advantages of convolutional neural networks in local feature extraction with the capabilities of Transformers in long-range dependency modeling. Utilizing multimodal feature complementarity and deep feature alignment strategies, it effectively improves the accuracy and robustness of remote sensing image classification, demonstrating strong adaptability and application value in complex environments.

[0023] However, the above methods still have the following problems: (1) Hyperspectral images have serious "same spectrum, different objects" and "same object, different spectra" phenomena, resulting in significant spectral interference; (2) There is a modal imbalance problem between LiDAR-assisted data and HSI data, with large differences in feature expression dimensions; (3) Existing feature fusion strategies are insufficient in modeling cross-modal contextual information, making it difficult to fully utilize the complementarity between modalities, especially under complex scenes and small sample conditions where performance is limited. These problems lead to low accuracy in multi-source remote sensing image classification.

[0024] Based on this, in order to address the problems of severe spectral interference, modal imbalance and insufficient feature fusion in the existing multimodal remote sensing data joint classification, this invention provides a multi-source remote sensing image classification method that integrates frequency domain attention mechanism and cross-modal Transformer. This method has high classification accuracy in complex urban scenes and under small sample conditions, and achieves a good balance between accuracy and computational cost.

[0025] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0026] Figure 1This is a flowchart illustrating a multi-source remote sensing image classification method that integrates a frequency domain attention mechanism and a cross-modal Transformer, as described in this specification. The method specifically includes the following steps: S101, acquire hyperspectral images and lidar images of the target scene.

[0027] In one embodiment, acquiring a hyperspectral image and a lidar image of a target scene includes: acquiring an initial hyperspectral image and an initial lidar image of the target scene; performing principal component analysis on the initial hyperspectral image to reduce its spectral dimension to a preset number; adjusting the spatial resolution of the dimensionally adjusted initial hyperspectral image to match the spatial resolution of the initial lidar image, and uniformly cropping the adjusted initial hyperspectral image and the initial lidar image into images of a fixed size; determining the cropped initial hyperspectral image as the hyperspectral image of the target scene, and determining the cropped initial lidar image as the lidar image of the target scene.

[0028] S102 uses a frequency domain attention mechanism to perform frequency decomposition and weighted fusion of the spectral and spatial dimensions of the hyperspectral image to obtain the frequency-enhanced hyperspectral features.

[0029] Optionally, land cover classification in multi-source remote sensing images can be achieved using a multi-source remote sensing classification network model; such as... Figure 2 As shown, the multi-source remote sensing classification network model includes a frequency domain attention module (FreqAttnBlock), a depthwise separable convolutional cross-modal attention module (DWConvCrossBlock), two parallel Transformer encoders, a dual-branch fusion attention module (DuFuseAttention), and a classification head.

[0030] The frequency domain attention module is used to perform frequency decomposition and weighted fusion of the spectral and spatial dimensions of hyperspectral images to obtain frequency-enhanced hyperspectral features.

[0031] Specifically, such as Figure 3 As shown, the frequency domain attention module FreqAttnBlock includes a local residual fusion module and a frequency domain gating module (FreqGate Block). It performs frequency decomposition and weighted fusion of the hyperspectral image in both spectral and spatial dimensions using a frequency domain attention mechanism to obtain frequency-enhanced hyperspectral features. This includes: processing the hyperspectral image... The input is fed into the frequency domain attention module, where the hyperspectral image undergoes preliminary enhancement through the local residual fusion module, yielding the preliminarily enhanced features. In the frequency domain gating module, the initially enhanced features are subjected to Fast Fourier Transform (FFT) in both the spectral and spatial dimensions to convert them to the frequency domain, thus obtaining frequency domain features. For any dimension, the low-frequency vector of the frequency domain features is determined using a preset low-frequency mask and a learnable weight vector, and the high-frequency vector of the frequency domain features is determined using the same preset low-frequency mask. The low-frequency and high-frequency components are then subjected to Inverse Fourier Transform (IFT) to convert them back to the spatial domain. The low-frequency and high-frequency features in the spatial domain are adaptively fused through a gating mechanism to obtain the frequency domain enhanced features. The frequency domain enhanced features corresponding to the spectral and spatial dimensions are then residually concatenated with the initially enhanced features to obtain the frequency domain enhanced hyperspectral features.

[0032] In one embodiment, a local residual fusion module fuses shallow and deep local features through two 1×1 convolutional mappings and a RepVGG bottleneck structure to obtain pre-enhanced features. Specifically, the local residual fusion module includes two 1×1 convolutional layers and a 3×3 convolutional layer. The local residual fusion module performs pre-enhanced enhancement on the hyperspectral image to obtain pre-enhanced features, including: convolving the hyperspectral image through two 1×1 convolutional layers to obtain a first convolutional feature and a second convolutional feature; adding the first and second convolutional features to obtain a third feature; convolving the second convolutional feature through a 3×3 convolutional layer to obtain a fourth feature; and adding the third and fourth features to obtain the pre-enhanced features. .

[0033] The learnable weight vector includes a learnable spectral weight vector and a learnable spatial frequency weight matrix. In the frequency domain gating module FreqGate Block, as shown... Figure 4 As shown, in the spectral dimension, after rearranging the features, a one-dimensional fast Fourier transform (1DFFT) is performed on the channel dimension to obtain the first frequency domain features. ; Utilizing a preset low-frequency mask Mask spec and learnable spectral weight vector Spect weight ( S The first spectral feature is decomposed into low-frequency components. The first spectral feature is decomposed into high-frequency components using a preset low-frequency mask. The spatial domain features are obtained by performing inverse Fourier transform (IFFT) on each feature and then taking the real part. and In the spatial dimension, a two-dimensional fast Fourier transform (2DFFT) is performed on the features to obtain the second frequency domain features. And utilize spatial low-frequency masking Mask spat and learnable spatial frequency weight matrix Spat weight (P The second spectral feature Decomposed into low-frequency components Using a preset low-frequency mask Mask spat The second spectral feature is decomposed into high-frequency components. Similarly, the spatial domain features are obtained by performing an inverse Fourier transform (IFFT). and Introducing channel-level gating functions Each dimension adaptively fuses low-frequency and high-frequency components to obtain frequency domain enhancement features corresponding to the spectral and spatial dimensions. and Finally, the frequency domain enhancement features are... and Compared with the features after initial enhancement By adding residual connections and optionally applying a channel attention mechanism, the frequency-domain enhanced hyperspectral features are obtained. .

[0034] Specifically, hyperspectral images typically contain more than 100 bands. In this invention, the more than 100 bands can be randomly extracted and reduced to 30 bands using a pre-existing PCA dimensionality reduction technique to obtain hyperspectral data with dimensions of [144 30 1111], where 144 is the batch_size, representing the number of samples processed in one forward propagation, and 11 is the height and width of each spectral data.

[0035] 1. In the spectral dimension, rearrange the features and then perform a one-dimensional fast Fourier transform (frequency domain) on the channel dimension. like Figure 5 As shown, taking a hyperspectral image with 145 bands as an example, a pixel of a category (Trees) has continuous feature values ​​across all 145 bands (i.e., a spectral curve of length 145). Therefore, we perform a Fourier transform on the spectral value of one pixel to obtain 145 frequency domain coefficients. After transforming the spectral curve to the frequency domain, the low-frequency components represent the overall shape and trend of the spectrum, while the high-frequency components represent noise and abrupt changes in the spectrum. Processing in the frequency domain allows for better separation and enhancement of useful spectral features.

[0036] Processing Procedure and Results: A one-dimensional FFT is performed on the last dimension (C, i.e., channel dimension 30) to obtain a complex spectrum, still in the shape [144, 11, 11, 30]. This spectrum represents the frequency domain representation of the 30 channels at each spatial location (11x11). The 30-dimensional vector at each location is converted into 30 complex numbers, representing the spectral signal at that location in the frequency domain. In the frequency domain, each frequency component is a complex number, containing amplitude and phase. Low-frequency components correspond to smooth spectral changes, while high-frequency components correspond to details and noise.

[0037] The calculation process includes: a: Determine the number of low-frequency channels to retain: Keep_C = int(C * keep_ratio) = int(30 * 0.25) = 7 b: Create a low-frequency mask mask_spec with the same shape as the frequency domain feature x_fft_spec, with shape [B HW C]=[144, 11, 11, 30], initially all of which are 0.

[0038] Set the first 7 frequency points (i.e., indices 0 to 6) of channel C to 1, and the rest to 0. Thus, the mask shape is [144,11,11,30], but only the first 7 positions in the channel dimension are 1.

[0039] c: The learnable weight spec_freq_weight is a parameter of shape [1, 30, 1, 1] that reshapes spec_freq_weight from [1,30,1,1] to [1,1,1,30] so that it can be multiplied with x_fft_spec.

[0040] d: Calculate the low-frequency component: x_fft_low = x_fft_spec * mask_spec * spec_weight Here, mask_spec means setting the high-frequency part to 0 and retaining the low-frequency part; spec_weight: a learnable weight vector that weights the frequency components of each channel. This weight is learnable, allowing the network to adjust the importance of each channel in the frequency domain.

[0041] e: Calculate the high-frequency component: x_fft_high = x_fft_spec * (1 - mask_spec).

[0042] f. Perform inverse FFT on the low-frequency and high-frequency spectra respectively, transform them back to the spatial domain, and obtain x_spec_low and x_spec_high, both with the shape [144,11,11,30]. Then adjust them back to [144,30,11,11].

[0043] 2. In the spatial dimension, perform a two-dimensional fast Fourier transform (spatial domain) on the features. Regarding the spatial Fourier transform, this invention has 30 bands, each of which is an 11x11 two-dimensional image, and each pixel value has a spectral feature value. For band i (i from 0 to 30), there is an 11x11 image layer, which is the image of the scene in band i. When performing the spatial Fourier transform on each band, this invention performs a 2D Fourier transform on the two-dimensional image of each band. The purpose of the transform is to analyze the spatial frequency characteristics of the image of each band. For band 0, this invention has an 11x11 image, and a 2D Fourier transform is performed on this image to obtain the frequency domain representation. Similarly, a 2D Fourier transform is performed on the image of band 1, and so on. It is similar to the calculation method of "rearranging the features in the spectral dimension and then performing a one-dimensional fast Fourier transform (frequency domain) on the channel dimension".

[0044] S103 uses depthwise separable convolution operations and multi-head self-attention mechanisms to semantically enhance LiDAR images, resulting in semantically enhanced LiDAR features.

[0045] The depthwise separable convolutional cross-modal attention module is used to extract local structural features through depthwise separable convolution operations and model cross-modal context using a multi-head attention mechanism to achieve semantic enhancement of LiDAR images.

[0046] In one embodiment, such as Figure 6 As shown, Figure 6 This is a schematic diagram illustrating the implementation flow of the depthwise separable convolutional cross-modal attention module DWConvCrossBlock. Through depthwise separable convolution operations and a multi-head self-attention mechanism, semantic enhancement is performed on the LiDAR image to obtain semantically enhanced LiDAR features. This includes: transforming the LiDAR image... The input is fed into a depthwise separable convolutional cross-modal attention module, and the LiDAR image is processed by 1×1 convolution. Perform channel expansion; apply depthwise separable convolution to the expanded features to generate the query vector. Key vector Sum value vector ; query vector Key vector Sum value vector After flattening into a sequence, a multi-head self-attention mechanism is used to compute cross-spatial enhancement features. And enhance features across space through 1×1 convolution. and query vector Mapping to the corresponding output dimension; enhancing the mapped cross-space features and query vector Element-wise addition is performed to fuse local details with the global context, resulting in semantically enhanced LiDAR features. .

[0047] S104 uses a Transformer encoder to extract deep semantic features from frequency-domain enhanced hyperspectral features and semantically enhanced lidar features, respectively.

[0048] Two parallel Transformer encoders are used to capture long-range dependencies within a mode, extracting deep semantic features from frequency-enhanced hyperspectral features and semantically enhanced lidar features, respectively.

[0049] like Figure 7 As shown, the Transformer encoder extracts deep semantic features from frequency-domain enhanced hyperspectral features and semantically enhanced LiDAR features, including: frequency-domain enhanced hyperspectral features. and semantically enhanced lidar features After being flattened into sequences and having positional encodings added, the sequences are input into two independent parallel Transformer encoders. In each Transformer encoder, a multi-layer self-attention mechanism and a feedforward network are used to capture the long-range dependencies of features of the corresponding modality, thereby obtaining the deep semantic features corresponding to the spectral modality. Deep semantic features corresponding to lidar modes .

[0050] like Figure 8 As shown, Figure 8 This is a schematic diagram of the Transformer encoder structure, which specifically includes embedded image patches, pre-norm, multi-head attention mechanism, pre-norm, and feed forward network connected in sequence.

[0051] S105: For any modality's deep semantic features, channel attention and spatial attention are applied to the deep semantic features respectively to obtain channel enhancement features and spatial enhancement features. The deep semantic features, channel enhancement features, and spatial enhancement features are then weighted using learnable weights. The weighted features are then convolved and added to the deep semantic features to obtain fused features. Finally, the fused features corresponding to the spectral modality and the fused features corresponding to the lidar modality are added to obtain fused output features.

[0052] The dual-branch fusion attention module is used to perform channel attention and spatial attention weighting on the deep semantic features output by the two Transformer encoders, and then perform multi-level fusion through learnable weights to obtain fused output features.

[0053] The dual-branch fusion attention module includes a channel attention branch and a spatial attention branch; the implementation process of the dual-branch fusion attention module includes: integrating the deep semantic features of the spectral modalities. Deep semantic features of lidar modes The input is fed into the dual-branch fusion attention module; in the channel attention branch, the two deep semantic features are processed separately. and Perform global average pooling to obtain channel statistics. and ,Will and After concatenation, the joint embedding features are obtained by sequentially applying 1x1 convolution and SiLU activation function. ;according to Generate channel attention weights for each modality And based on channel attention weights Channel-weighted deep semantic features for each modality are applied to obtain channel-enhanced features for each modality; in the spatial attention branch, two deep semantic features are weighted separately. and The spatial response map is obtained by performing a 3×3 convolution. and ,Will and After concatenation, the joint spatial attention map is obtained by sequentially applying 1×1 convolution, batch normalization, and the sigmoid function. And through joint spatial attention maps Spatially weight the deep semantic features of each modality to obtain the spatial augmentation features of each modality. For any modality, a learnable fusion weight vector is used. deep semantic features Channel enhancement features and spatial enhancement features Perform weighting, and then assign the weighted features After 3×3 convolution and normalization, it is compared with the corresponding deep semantic features. Perform residual connections to obtain fused features; add the fused features corresponding to the spectral modes and the lidar modes to obtain the fused output features. .

[0054] like Figure 9 As shown, Figure 9 This is a flowchart illustrating the implementation of the DuFuseAttention dual-branch fusion attention module, which integrates deep semantic features of spectral modalities. Deep semantic features of lidar modes The input is fed into the dual-branch fusion attention module; in the channel attention branch (cAttn), the channel enhancement features of the spectral mode are obtained. Channel enhancement features of lidar modes In the spatial attention branch (sAttn), the spatial enhancement features of the spectral modes are obtained. Spatial augmentation features of lidar modes In spectral modes, through learnable fusion weight vectors , and deep semantic features Channel enhancement features and spatial enhancement features We perform weighting to obtain the weighted features. and will After 3×3 convolution and normalization, it is compared with the corresponding deep semantic features. Perform residual connections to obtain fused features. In lidar modes, through learnable fusion weight vectors , and deep semantic features Channel enhancement features and spatial enhancement features We perform weighting to obtain the weighted features. and will After 3×3 convolution and normalization, it is compared with the corresponding deep semantic features. Perform residual connections to obtain fused features. The fusion features corresponding to the spectral modes and the fusion features corresponding to the lidar modes are added together to obtain the fused output features. .

[0055] like Figure 10 As shown, Figure 10Figure (a) in the diagram is a flowchart of the implementation of the channel attention branch cAttn. Figure 10 Figure (b) shows the flowchart of the spatial attention branch sAttn. In the channel attention branch (cAttn), global average pooling is performed on the two deep semantic features respectively, resulting in two channel statistics with shape [B, C, 1, 1]. and ,Will and After concatenation, features of shape [B, 2C, 1, 1] are obtained. The number of channels is reduced from 2C to C through a 1x1 convolution, and then the SiLU activation function is used to obtain joint embedding features. ;Will Channel attention weights are obtained by applying two different 1x1 convolutions (Conv1 and Conv2) to the two convolutional features, and then passing the two features after convolution through a sigmoid activation function. and And based on channel attention weights and Channel-weighted deep semantic features for each modality are applied to obtain channel-enhanced features for the spectral modality. Channel enhancement features of lidar modes .

[0056] The formula for calculating the attention weights of the two channels is as follows: = sigmoid(Conv1( SiLU( Conv( cat(Avg_pool( ), Avg_pool( )) ) ) ); = sigmoid(Conv2( SiLU( Conv( cat(Avg_pool( ), Avg_pool( )) ) ) ).

[0057] In the spatial attention branch (sAttn), the two deep semantic features are processed respectively. and The spatial response map is obtained by performing a 3×3 convolution. and ,Will and After concatenation, the joint spatial attention map is obtained by sequentially applying 1×1 convolution, batch normalization, and the sigmoid function. And through joint spatial attention maps Generate spatial attention weights for each modality and ,in and They all originate from the same joint space attention graph It is directly derived from this, and its mathematical expression is: , , The calculation formula is = sigmoid( BN( Conv(Cat [Conv( ), Conv( )] ) ) ); and based on spatial attention weights and Spatially weighting the deep semantic features of each modality yields the spatially enhanced features of the spectral modality. Spatial augmentation features of lidar modes .

[0058] S106 aggregates and fuses the global semantic vector of the output features through two methods: global average pooling and attention pooling, and then concatenates the two global semantic vectors. The concatenated global semantic vector is then mapped to the category space through a multilayer perceptron to obtain the land cover classification results of the multi-source remote sensing image.

[0059] The classification head is used to map the fused output features to land cover category probabilities. For example... Figure 11 As shown, the classification head includes a global average pooling layer, an attention pooling layer, and a multilayer perceptron (MLP). The MLP includes a layer normalization layer (LayerNorm) and a fully connected layer. The implementation process of the classification head includes: inputting the fused output features into the classification head; performing global average pooling on the fused output features through the global average pooling layer to obtain a first global semantic vector; and performing attention pooling on the fused output features through the attention pooling layer to obtain a second global semantic vector; concatenating the first and second global semantic vectors and inputting them into the MLP to obtain the land cover classification result of the multi-source remote sensing image.

[0060] In one embodiment, the present invention provides a multi-source remote sensing image classification method that integrates frequency domain attention mechanism and cross-modal Transformer, the embodiment of which includes the following steps: S1: Acquire a multi-source remote sensing image dataset, including sample hyperspectral images (HSI) and corresponding sample LiDAR images, and preprocess the multi-source remote sensing image dataset.

[0061] S2. The preprocessed multi-source remote sensing image dataset is divided into training and testing sets. A multi-source remote sensing classification network model (DWFTD) based on frequency domain attention and dual-branch Transformer is constructed, and the model is trained and tested according to the training and testing sets.

[0062] S3 classifies land cover in multi-source remote sensing images using a trained multi-source remote sensing classification network model. The multi-source remote sensing images include HSI images and LiDAR images.

[0063] The multi-source remote sensing classification network model includes a hyperspectral feature extraction branch, a LiDAR feature extraction branch, two parallel Transformer encoders, a dual-branch fusion attention module, and a classification head.

[0064] The hyperspectral feature extraction branch includes a frequency domain attention module FreqAttnBlock, which performs frequency decomposition and adaptive weighted fusion of the input hyperspectral features in the spectral-spatial dimensions.

[0065] The LiDAR feature extraction branch includes a depthwise separable convolutional cross-modal attention module, DWConvCrossBlock, which is used to extract local structural features and model cross-modal context using a multi-head attention mechanism.

[0066] Two parallel Transformer encoders are used to serialize the features output from the two branches respectively and capture long-range dependencies within the modality.

[0067] The DuFuseAttention module, a dual-branch fusion attention module, is used to perform channel attention and spatial attention weighting on the output features of the Transformer from two modalities, and to perform multi-level fusion through learnable weights.

[0068] The classification head is used to map the fused features to land cover category probabilities.

[0069] like Figure 12 As shown, the trained multi-source remote sensing classification network model is used to classify ground features in multi-source remote sensing images, specifically including the following steps: S21, Input multi-source remote sensing images: hyperspectral image + LiDAR image.

[0070] S22, Data preprocessing: PCA dimensionality reduction and size normalization.

[0071] Specifically, principal component analysis is performed on the hyperspectral image to reduce its spectral dimension to a preset number; the spatial resolution of the adjusted hyperspectral image and LiDAR image are then aligned and uniformly cropped into image blocks of a fixed size to obtain the preprocessed hyperspectral image and LiDAR image.

[0072] S23, dual-branch feature extraction.

[0073] Two-branch feature extraction was performed on the preprocessed HSI and LiDAR images.

[0074] S24, Hyperspectral Feature Extraction Branch.

[0075] Specifically, the hyperspectral feature extraction branch includes a frequency domain attention module. The preprocessed hyperspectral image is input into the frequency domain attention module FreqAttnBlock, where enhanced hyperspectral features are obtained through local residual fusion, spectral-spatial dimension frequency domain decomposition, and gated fusion mechanisms. .

[0076] S25, LiDAR feature extraction branch.

[0077] Specifically, the preprocessed LiDAR image is input into the depthwise separable convolutional cross-modal attention module DWConvCrossBlock. In this module, semantically enhanced LiDAR features are obtained through channel expansion, depthwise convolutional feature extraction, multi-head attention enhancement, and residual fusion. .

[0078] S26, Parallel Transformer Encoder.

[0079] Specifically, the result obtained in step S24 and the result obtained in step S25 The sequences are flattened and positional encodings are added before being input into two independent parallel Transformer encoders. Through multi-layer self-attention mechanisms and feedforward networks, long-range dependencies within the modalities are captured, resulting in deep semantic features for both modalities. and .

[0080] S27, DuFuseAttention, a dual-branch fusion attention module.

[0081] Specifically, the features obtained in step S26 and The input is fed into the dual-branch fusion attention module DuFuseAttention, where the features of the two modalities are weighted through channel attention and spatial attention branches respectively. Then, multi-level feature fusion is performed using learnable fusion weights to obtain the fused output features. .

[0082] S28 enhances the discriminative output of global and weighted features.

[0083] Specifically, the fused output features obtained in step S27 Global information is aggregated using both global average pooling and attention pooling. The resulting two global semantic vectors are then concatenated. The concatenated global semantic vector is then mapped to the final category space through an MLP consisting of a LayerNorm normalization layer and a fully connected layer, thus achieving land cover classification for multi-source remote sensing images.

[0084] In one embodiment, data preparation and preprocessing are as follows: The Berlin public multi-source remote sensing dataset is selected. Principal component analysis (PCA) is performed on the hyperspectral images to reduce the dimensionality to 30 dimensions. HSI and LiDAR images are uniformly cropped into 11×11 image patches. The training and test sets are randomly divided according to a certain ratio (70%:30%).

[0085] Model building and training: building such as Figure 2 The multi-source remote sensing classification network model DWFTD is shown. The core components of the model include:

[0086] FreqAttnBlock: Used for hyperspectral feature frequency domain enhancement.

[0087] DWConvCrossBlock: Used for LiDAR feature extraction and cross-modal interaction.

[0088] Parallel Transformer encoder: Two modes are processed independently, with 4 layers and 8 heads.

[0089] DuFuseAttention: Used for dual-modal feature fusion.

[0090] Classification Header: Includes Global Average Pooling, Attention Pooling, and MLP.

[0091] During the training of the multi-source remote sensing classification network model, the cross-entropy loss function, Adam optimizer (learning rate 1e-4), batch size 144, and training were performed for 200 epochs on a GPU platform.

[0092] Testing and Results Analysis: Evaluate model performance using a test set. For example... Figure 13 As shown in Table 1, the multi-source remote sensing classification network model (DWFTD) provided by this invention outperforms the comparative methods (such as S2ATNet, MS2CANet, etc.) in overall classification accuracy (OA) on multiple datasets, proving its effectiveness. Figure 13The image provided in this embodiment of the invention shows a comparison of classification results with other methods on the Berlin dataset, where ae represent: (a) GroundTruth; (b) S2ENet; (c) CrossHL; (d) AsyFFNet; (e) ExViT; (f) MACN; (g) MS2CANet; (h) S2ATNet; and (i) DWFTD.

[0093] Table 1. Comparison of Overall Classification Accuracy (OA) of the present invention across multiple datasets. Wherein, OA is the overall accuracy, AA is the average accuracy, Kappa coefficient is the classification correlation coefficient, MacroAvgF1-score is the macro-average F1 score, Parameters is the number of learnable parameters of the model, Training Times is the time required for the model to complete training, and Testing Times is the time required for the model to make predictions on the test set. The proposed DWFTD achieved first place in all four core performance metrics (OA, AA, Kappa, F1-score) and also achieved the highest accuracy in most individual classes. This strongly demonstrates that the model outperforms all comparable models in classification performance on the "Berlin" dataset. The superior performance of the new model comes at the cost of higher model complexity (larger number of parameters) and longer computation time. Its parameter count (748180) and training / testing time are significantly higher than some lightweight models (such as s2enet).

[0094] This invention provides a multi-source remote sensing image classification method that integrates a frequency domain attention mechanism and a cross-modal Transformer. This method utilizes hyperspectral and LiDAR images, and the overall process includes two feature extraction branches and four core stages. The hyperspectral image, after dimensionality reduction via principal component analysis, is input into a frequency domain attention module. This module extracts spatial features by combining convolution and residual bottlenecks, and preserves low-frequency global structure and high-frequency details through spectral decomposition. Feature enhancement is then achieved using learnable frequency gating and channel-spatial attention. LiDAR single-band data is processed using depthwise separable convolutional cross-blocks. This module expands the channel dimension and uses channel-wise convolution and multi-head self-attention to model the local structure. Finally, channel alignment is achieved through convolution to obtain semantically enhanced LiDAR features. Subsequently, the two feature streams are input into independent Transformer encoders, where they are mapped into patch sequences under positional encoding. Multi-layer residual attention blocks capture long-range dependencies within each modality and enhance semantic representation. Deep semantic features are extracted from frequency-enhanced hyperspectral features and semantically enhanced LiDAR features. These deep semantic features from both modalities are then fed into a dual-branch fusion attention module. This module adaptively filters features in the channel dimension and generates a joint attention map in the spatial dimension to align modalities. It also enhances cross-modal complementarity by weighted fusion of original, channel-enhanced, and spatially enhanced features. After flattening, the fused features are sequentially processed through global average pooling and attention pooling to obtain a global semantic vector. Finally, the classification head jointly utilizes different aggregated information to predict the land cover category. This model performs exceptionally well in complex urban scenarios and under small sample conditions, achieving a good balance between accuracy and computational cost.

[0095] The beneficial effects achieved by this invention are: Compared to existing multimodal remote sensing image classification models, this invention demonstrates superior performance across various evaluation metrics, including overall classification accuracy, average accuracy, and Kappa coefficient, on the Berlin public dataset. The proposed DWFTD network provides more accurate results for remote sensing land cover interpretation, assisting experts in related fields to conduct more efficient and accurate land use monitoring and analysis.

[0096] DWFTD explicitly utilizes spectral information through a frequency domain attention module, effectively models intermodal interactions through a cross-modal attention module, and achieves stable complementarity through a dual-branch fusion mechanism. It can extract more discriminative features from multi-source remote sensing images, thereby achieving more accurate classification in complex scenes and under small sample conditions.

[0097] The methods provided in this specification can be implemented by a server, which can be a server set up on a business platform, or a device such as a desktop computer or laptop computer that can execute the solutions in this specification.

[0098] When applying the fusion frequency domain attention mechanism and cross-modal Transformer multi-source remote sensing image classification method provided in this manual, it is not necessary to consider... Figure 1 The steps shown are executed in sequence. The specific execution order of each step can be determined as needed, and this manual does not impose any restrictions on it.

[0099] The above describes a multi-source remote sensing image classification method that integrates a frequency domain attention mechanism and a cross-modal Transformer, as provided in one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding multi-source remote sensing image classification device that integrates a frequency domain attention mechanism and a cross-modal Transformer, the device comprising: The acquisition module is used to acquire hyperspectral images and lidar images of the target scene; The classification module performs frequency decomposition and weighted fusion of the hyperspectral image in both spectral and spatial dimensions using a frequency-domain attention mechanism to obtain frequency-enhanced hyperspectral features. It then performs semantic enhancement on the LiDAR image using depthwise separable convolution and a multi-head self-attention mechanism to obtain semantically enhanced LiDAR features. A Transformer encoder extracts deep semantic features from both the frequency-enhanced hyperspectral features and the semantically enhanced LiDAR features. For any modality's deep semantic features, channel attention and spatial attention weighting are applied to obtain channel-enhanced features and... Spatial augmentation features are generated, and deep semantic features, channel augmentation features, and spatial augmentation features are weighted using learnable weights. The weighted features are then convolved and added to the deep semantic features to obtain fused features. The fused features corresponding to spectral modes and LiDAR modes are added to obtain fused output features. The global semantic vectors of the fused output features are aggregated using both global average pooling and attention pooling, and the two global semantic vectors are concatenated. Finally, a multilayer perceptron maps the concatenated global semantic vectors to the category space to obtain the land cover classification results of the multi-source remote sensing image.

[0100] Specific limitations regarding the multi-source remote sensing image classification device that integrates frequency domain attention mechanisms and cross-modal Transformers can be found in the limitations of the multi-source remote sensing image classification method integrating frequency domain attention mechanisms and cross-modal Transformers mentioned above, and will not be repeated here. Each module in the aforementioned multi-source remote sensing image classification device integrating frequency domain attention mechanisms and cross-modal Transformers can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0101] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The proposed method integrates a frequency domain attention mechanism with a cross-modal Transformer for classifying multi-source remote sensing images.

[0102] This instruction manual also provides Figure 14 The schematic diagram of the computer device shown is as follows: Figure 14 At the hardware level, the computer device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 The proposed method integrates a frequency domain attention mechanism with a cross-modal Transformer for classifying multi-source remote sensing images.

[0103] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0104] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A multi-source remote sensing image classification method that fuses frequency domain attention mechanism and cross-modal Transformer, characterized in that, The method comprises the following steps: obtaining a hyperspectral image and a lidar image of a target scene; performing frequency decomposition and weighted fusion of the spectral dimension and the spatial dimension of the hyperspectral image through a frequency domain attention mechanism to obtain a frequency domain enhanced hyperspectral feature; performing semantic enhancement on the lidar image through a depth separable convolution operation and a multi-head self-attention mechanism to obtain a semantic enhanced lidar feature; extracting deep semantic features of the frequency domain enhanced hyperspectral feature and the semantic enhanced lidar feature through a Transformer encoder respectively; for the deep semantic features of any one modality, performing channel attention and spatial attention weighting on the deep semantic features respectively to obtain channel enhanced features and spatial enhanced features, and weighting the deep semantic features, the channel enhanced features and the spatial enhanced features through learnable weights, and then performing convolution on the weighted features and adding the convolution results to the deep semantic features to obtain fusion features; adding the fusion features corresponding to the spectral modality and the fusion features corresponding to the lidar modality to obtain fusion output features; aggregating global semantic vectors of the fusion output features through global average pooling and attention pooling, and then splicing the obtained two global semantic vectors; mapping the spliced global semantic vectors to a category space through a multilayer perceptron to obtain a ground object classification result of the multi-source remote sensing image.

2. The method of claim 1, wherein, The method comprises the following steps: obtaining an initial hyperspectral image and an initial lidar image of a target scene; performing principal component analysis on the initial hyperspectral image to reduce the spectral dimension to a preset number; adjusting the spatial resolution of the initial hyperspectral image with the adjusted dimension to be consistent with the spatial resolution of the initial lidar image, and then uniformly cropping the adjusted initial hyperspectral image and the initial lidar image into images with a fixed size; determining the cropped initial hyperspectral image as the hyperspectral image of the target scene and determining the cropped initial lidar image as the lidar image of the target scene.

3. The method of claim 1, wherein, The method comprises the following steps: obtaining a hyperspectral image and a lidar image of a target scene; performing frequency decomposition and weighted fusion of the spectral dimension and the spatial dimension of the hyperspectral image through a frequency domain attention mechanism to obtain a frequency domain enhanced hyperspectral feature; performing semantic enhancement on the lidar image through a depth separable convolution operation and a multi-head self-attention mechanism to obtain a semantic enhanced lidar feature; extracting deep semantic features of the frequency domain enhanced hyperspectral feature and the semantic enhanced lidar feature through a Transformer encoder respectively; for the deep semantic features of any one modality, performing channel attention and spatial attention weighting on the deep semantic features respectively to obtain channel enhanced features and spatial enhanced features, and weighting the deep semantic features, the channel enhanced features and the spatial enhanced features through learnable weights, and then performing convolution on the weighted features and adding the convolution results to the deep semantic features to obtain fusion features; adding the fusion features corresponding to the spectral modality and the fusion features corresponding to the lidar modality to obtain fusion output features; aggregating global semantic vectors of the fusion output features through global average pooling and attention pooling, and then splicing the obtained two global semantic vectors; mapping the spliced global semantic vectors to a category space through a multilayer perceptron to obtain a ground object classification result of the multi-source remote sensing image. The method comprises the following steps: obtaining a hyperspectral image and a lidar image of a target scene; performing frequency decomposition and weighted fusion of the spectral dimension and the spatial dimension of the hyperspectral image through a frequency domain attention mechanism to obtain a frequency domain enhanced hyperspectral feature; performing semantic enhancement on the lidar image through a depth separable convolution operation and a multi-head self-attention mechanism to obtain a semantic enhanced lidar feature; extracting deep semantic features of the frequency domain enhanced hyperspectral feature and the semantic enhanced lidar feature through a Transformer encoder respectively; for the deep semantic features of any one modality, performing channel attention and spatial attention weighting on the deep semantic features respectively to obtain channel enhanced features and spatial enhanced features, and weighting the deep semantic features, the channel enhanced features and the spatial enhanced features through learnable weights, and then performing convolution on the weighted features and adding the convolution results to the deep semantic features to obtain fusion features; adding the fusion features corresponding to the spectral modality and the fusion features corresponding to the lidar modality to obtain fusion output features; aggregating global semantic vectors of the fusion output features through global average pooling and attention pooling, and then splicing the obtained two global semantic vectors; mapping the spliced global semantic vectors to a category space through a multilayer perceptron to obtain a ground object classification result of the multi-source remote sensing image. The classification head is used for mapping the fusion output features to the ground object category probability.

4. The method of claim 3, wherein, The frequency domain attention module includes a local residual fusion module and a frequency domain gating module, and the frequency domain attention mechanism is used to perform frequency decomposition and weighted fusion on the hyperspectral image in the spectral dimension and the spatial dimension, to obtain the frequency domain enhanced hyperspectral features, including: The hyperspectral image is input into the frequency domain attention module, the local residual fusion module is used to preliminarily enhance the hyperspectral image, and the preliminarily enhanced features are obtained; In the frequency domain gating module, the preliminarily enhanced features are respectively subjected to fast Fourier transform in the spectral dimension and the spatial dimension, so as to be converted to the frequency domain, and the frequency domain features are obtained; For any dimension, the low-frequency vector of the frequency domain feature is determined through a preset low-frequency mask and a learnable weight vector, and the high-frequency vector of the frequency domain feature is determined through the preset low-frequency mask; the low-frequency and high-frequency components are subjected to inverse Fourier transform, and are converted back to the spatial domain; the low-frequency and high-frequency features in the spatial domain are adaptively fused through the gating mechanism, and the frequency domain enhanced features are obtained; The frequency domain enhanced features corresponding to the spectral dimension and the spatial dimension are connected with the preliminarily enhanced features in residual, and the frequency domain enhanced hyperspectral features are obtained.

5. The method of claim 4, wherein, The local residual fusion module includes two 1×1 convolution layers and a 3×3 convolution layer; The local residual fusion module is used to preliminarily enhance the hyperspectral image, and the preliminarily enhanced features are obtained, including: The hyperspectral image is convolved through the two 1×1 convolution layers, to obtain first convolution features and second convolution features; The first convolution features and the second convolution features are added to obtain third features, and the second convolution features are convolved through the 3×3 convolution layer to obtain fourth features; The third features and the fourth features are added to obtain the preliminarily enhanced features.

6. The method of claim 3, wherein, The depth separable convolution operation and the multi-head self-attention mechanism are used to perform semantic enhancement on the laser radar image, to obtain the semantic enhanced laser radar features, including: The laser radar image is input into the depth separable convolution cross-modal attention module, and the laser radar image is subjected to channel expansion through 1×1 convolution; The depth separable convolution operation is performed on the expanded features to generate query vectors, key vectors and value vectors; After the query vectors, the key vectors and the value vectors are flattened into sequences, the multi-head self-attention mechanism is used to calculate the cross-spatial enhanced features, and the cross-spatial enhanced features and the query vectors are mapped to the corresponding output dimensions through 1×1 convolution; The mapped cross-spatial enhanced features and the query vectors are added element by element to obtain the semantic enhanced laser radar features.

7. The method of claim 3, wherein, The depth layer semantic features of the frequency domain enhanced hyperspectral features and the semantic enhanced laser radar features are respectively extracted through the Transformer encoder, including: The frequency domain enhanced hyperspectral features and the semantic enhanced lidar features are respectively flattened into sequences, and after adding position encodings, are respectively input into two independent and parallel Transformer encoders, in each of which, through a multi-layer self-attention mechanism and a feedforward network, long-range dependencies of the features of the corresponding modal are captured, to obtain deep semantic features corresponding to the spectral modal and deep semantic features corresponding to the lidar modal.

8. The method of claim 3, wherein, The dual-branch fusion attention module includes a channel attention branch and a spatial attention branch; The implementation process of the dual-branch fusion attention module includes: Deep semantic features of spectral modalities and lidar modalities into a dual-branch fusion attention module; In the channel attention branch, global average pooling is performed on the two deep semantic features respectively to obtain channel statistics and and and After splicing, the joint embedding features are obtained by 1x1 convolution and SiLU activation function in turn ; According to , the channel attention weights are generated for each modality, and the channel weighting is performed on the deep semantic features of each modality based on the channel attention weights to obtain the channel enhanced features of each modality;​​ In the spatial attention branch, 3x3 convolution is performed on two deep semantic features and respectively to obtain spatial response maps and , and after splicing and , 1x1 convolution, batch normalization and Sigmoid function are sequentially performed to obtain a joint spatial attention map , and the deep semantic features of each modality are spatially weighted through the joint spatial attention map to obtain spatially enhanced features of each modality ; For any modal, the deep semantic features, the channel enhanced features and the spatial enhanced features are weighted through a learnable fusion weight vector, and after the weighted features are subjected to 3*3 convolution and normalization processing, the features are connected in residual with the corresponding deep semantic features to obtain fusion features; The fusion features corresponding to the spectral modal and the fusion features corresponding to the lidar modal are added to obtain fusion output features.

9. The method of claim 3, wherein, The classification head includes a global average pooling layer, an attention pooling layer and a multi-layer perceptron; the implementation process of the classification head includes: The fusion output features are input into the classification head, the fusion output features are subjected to global average pooling through the global average pooling layer to obtain a first global semantic vector, and the fusion output features are subjected to attention pooling through the attention pooling layer to obtain a second global semantic vector; The first global semantic vector and the second global semantic vector are spliced and input into the multi-layer perceptron to obtain a ground object classification result of the multi-source remote sensing image; the multi-layer perceptron includes layer normalization and a fully connected layer.

Images