Multi-source remote sensing data classification method

Abstract

The invention provides a multi-source remote sensing data classification method, which belongs to the field of hyperspectral image processing in the field of remote sensing, and comprises the following steps: acquiring remote sensing data of various target objects; extracting shallow layer features of each kind of remote sensing data; obtaining a plurality of frequency characteristics of each kind of remote sensing data; fusing the same frequency features of all remote sensing data to obtain a plurality of same frequency fusion features; splicing the plurality of same-frequency fusion features to obtain a multi-source fusion feature; enabling the multi-source fusion features to sequentially pass through a frequency modulation layer and an attention layer which are superposed to obtain fused global features and local features; and weighting the global features and the local features in the spectral dimension to obtain a prediction classification result of the target object. According to the method, directional frequency characteristic decomposition and fusion of the multi-source data are realized, the multi-source fusion characteristics are extracted based on the multi-source remote sensing data to serve as the basis of classification, more comprehensive information is provided for classification of target objects, complex ground object characteristics can be fully captured, and the classification result is more accurate.

Description

Technical Field

[0001] The invention relates to a multi-source remote sensing data classification method, and belongs to the field of hyperspectral image processing in the remote sensing field. Background Art

[0002] At present, remote sensing data presents the coexistence of multiple source data such as high, medium and low resolution, multi-spectral, hyperspectral, synthetic aperture radar SAR, street view and LiDAR laser point cloud, providing basic data guarantee for remote sensing monitoring and other multi-field applications.

[0003] Existing remote sensing images mainly include: visible light RGB remote sensing images, panchromatic remote sensing images, multi / hyperspectral remote sensing images, infrared remote sensing images, LiDAR remote sensing images and synthetic aperture radar SAR remote sensing images.

[0004] Visible light RGB remote sensing images are a special case of multispectral images. They fuse the waves of three channels: red spectrum, blue spectrum, and green spectrum. RGB remote sensing images are the most commonly used remote sensing images in real life and are usually used to identify terrain and objects.

[0005] Panchromatic remote sensing images are different from RGB remote sensing images. Panchromatic remote sensing images are black-and-white images of the entire visible light wave range obtained by the remote sensor. They are called panchromatic images and are displayed as grayscale images on the map. Panchromatic remote sensing images generally have high spatial resolution, but cannot display the color of ground objects.

[0006] Multispectral remote sensing technology can provide more color information by fusing dozens to hundreds of spectra, and can assist in determining the properties of materials on the Earth's surface, but its spatial resolution is relatively low.

[0007] Hyperspectral imaging technology generates rich spectral data by capturing the reflectance spectra of multiple bands, and can detect the unique spectral features of a single object at different spatial locations. Therefore, hyperspectral imaging technology can carefully analyze and distinguish different substances, such as plant species, soil types, and water quality. Each band provides unique spectral features, which can detect visually indistinguishable substances, making hyperspectral images unique in material identification and change detection.

[0008] Infrared remote sensing images are obtained by sensing infrared rays reflected by ground objects and radiated by themselves. Since infrared rays have a long wavelength and strong penetration in the atmosphere and are not affected by darkness and smoke, infrared remote sensing images can capture clear pictures of the ground. However, they still have the disadvantages of poor resolution, low contrast, low signal-to-noise ratio, and blurred visual effects.

[0009] LiDAR remote sensing images are obtained by solving the ground coordinates of the laser point by comparing the laser angle emitted from the air or spacecraft and the detected laser distance. LiDAR technology obtains the three-dimensional spatial information of the target object by emitting laser and measuring the reflection time, generating high-precision point cloud data. LiDAR data not only provides ground height, but also depicts the shape and structure of the object, which is of great value for applications such as forest monitoring, urban modeling and terrain analysis.

[0010] Synthetic aperture radar (SAR) is a technology that uses a small aperture antenna to achieve the measurement effect of a large aperture radar through movement and mathematical calculations. It is a high-resolution imaging radar system. Through the movement of the antenna on a mobile platform (such as a satellite or aircraft), SAR can synthesize a "synthetic aperture" that is larger than the physical antenna aperture, thereby improving the imaging resolution and being able to accurately image the ground under various weather conditions and lighting conditions. Each pixel of the remote sensing image it generates contains not only the so-called grayscale value that reflects the intensity of microwave reflection on the surface, but also the phase value related to the radar slant range. However, the latter exhibits great randomness and is generally regarded as noise, which brings inconvenience to interference analysis.

[0011] In summary, each single-source remote sensing image has its own advantages, but due to the small amount of information in single-source remote sensing data, it may not be able to fully capture the complex features of the ground objects. Therefore, it is necessary to integrate different remote sensing data to judge the properties of the earth's surface materials and use the information provided by multi-source data for forest monitoring, land use and aboveground biomass estimation. Summary of the invention

[0012] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a multi-source remote sensing data classification method to solve the problem that complex features cannot be fully captured using single-source remote sensing data.

[0013] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0014] The present invention provides a multi-source remote sensing data classification method, comprising: acquiring multiple types of remote sensing data of a target object; extracting shallow features of each type of remote sensing data respectively; inputting the shallow features into a pre-constructed frequency feature decomposition module to obtain multiple preset frequency features of each type of remote sensing data; inputting the frequency features of all remote sensing data into a pre-constructed same-frequency feature fusion module, the same-frequency feature fusion module performs feature fusion on the same frequency features to obtain corresponding multiple same-frequency fusion features; splicing and fusing the multiple same-frequency fusion features to obtain multi-source fusion features; passing the multi-source fusion features through superimposed frequency modulation layers and attention layers in sequence to obtain fused global features and local features; weighting the fused global features and local features in a spectral dimension to extract depth information, and then obtaining a predicted classification result of the target object.

[0015] Furthermore, the extraction of shallow features of each type of remote sensing data includes: dividing the remote sensing data into image patches; performing multi-scale convolution operations on the divided patches to obtain corresponding multiple-scale features; in any preset channel, superimposing the multiple-scale features along the channel dimension to obtain channel fusion features; performing element summation, averaging and maximum pooling operations on the channel fusion features along the channel dimension to obtain corresponding sum feature maps, average feature maps and maximum pooling feature maps; splicing the sum feature maps, average feature maps and maximum pooling feature maps, and using convolution to perform further channel fusion to obtain low-dimensional channel features of the channel; splicing the low-dimensional channel features of all channels to obtain shallow features of the remote sensing data.

[0016] Furthermore, the multi-scale convolution operation includes: performing 3×3 convolution, 5×5 convolution and 7×7 convolution on the divided patch blocks, and performing batch normalization operation and ReLU operation in sequence after each convolution operation to obtain corresponding three scale features.

[0017] Furthermore, the frequency feature decomposition module is constructed based on the frequency domain Transformer; the obtaining of multiple preset frequency features of each shallow feature includes: passing the input shallow feature through the convolution and normalization layers to obtain the processed shallow feature ; The processed shallow features Averaged over the spectral dimension In the head, Divide into Equal parts, of which, is the number of frequency features, so that each equal portion is used to calculate one frequency feature; is the default value.

[0018] Each frequency feature corresponds to a preset window shape; for any equal part, each head is first evenly divided into non-overlapping windows using the window shape corresponding to the frequency feature, and then the attention of each window is calculated; then the frequency feature of each head is obtained; finally, the frequency features of each head are spliced ​​to obtain the frequency feature corresponding to the equal part.

[0019] Furthermore, the frequency features include: low-frequency features, high-frequency features, vertical features and horizontal features; the window shape corresponding to the low-frequency features is: ; is the window size index, Take a positive integer; used to calculate the low-frequency feature header, each window contains tokens; the token refers to a minimum unit in the window.

[0020] The window shape corresponding to the high-frequency feature is: ; In the header used to calculate high-frequency features, each window contains tokens; the window shape corresponding to the vertical feature is: ; In the header used to calculate vertical features, each window contains tokens; the window shape corresponding to the horizontal feature is: ; In the header used to calculate horizontal features, each window contains tokens.

[0021] Furthermore, the process of obtaining the vertical feature includes: The heads are evenly divided into non-overlapping Windows , is the number of windows, , The shallow features after processing length or width; , The shallow features after processing spectral dimension;

[0022] No. The query tensor of the head , key tensor Sum value tensor The dimension is ,in, ; ; Calculate the The attention of each window in the head; The attention calculation result of the window is:

[0023] ;

[0024] In the formula, For the The attention calculation results of the windows are: To calculate the attention operation, , , Respectively The query matrix, key matrix and value matrix of each head;

[0025] According to The attention of each window in the head gets The vertical characteristics of the head are expressed as:

[0026] ;

[0027] In the formula, For the The vertical characteristics of the head, , , The first, second and The attention calculation results of the window;

[0028] The vertical features of all heads used to calculate vertical features are concatenated to obtain the vertical features of shallow features. The expression is:

[0029] ;

[0030] In the formula, is the vertical feature of the shallow feature, For splicing operation, , and Respectively , and The vertical characteristics of the head.

[0031] Furthermore, the same-frequency component fusion module performs feature fusion on the same frequency components to obtain a plurality of corresponding same-frequency fusion features, including: for any frequency component, the frequency component from all remote sensing data is added element by element to obtain the same-frequency component ;Will Perform global average pooling on the channel dimension, and then pass through the channel attention module to obtain the channel weight , the expression is:

[0032] ;

[0033] In the formula, is the channel weight, is a 1×1 convolutional layer, To obtain the maximum value, Output of global average pooling in the channel dimension.

[0034] Will Global average pooling and global maximum pooling are performed in the spatial dimension, and then the spatial weight is obtained through the spatial attention module. , the expression is:

[0035] ;

[0036] In the formula, is the spatial weight, is a 7×7 convolutional layer, for The output of global average pooling in the spatial dimension, for Output of global max pooling in the spatial dimension.

[0037] According to the broadcasting rules, the channel weights are added by addition. and spatial weight Fusion to get the rough weight ;Will and Each channel of is rearranged by the rearrangement operation, and the expression is:

[0038] ;

[0039] In the formula, is the fine weight; is the sigmoid function, For group convolution, the number of groups is set to the number of channels. For channel rearrangement operation, is the coarse weight.

[0040] According to the frequency components and detailed weights of all remote sensing data, combined with residual connection, a weighted summation method is adopted to obtain the same-frequency fusion features of this frequency feature.

[0041] Furthermore, the frequency modulation layer and the attention layer adopt a phased architecture in which the frequency modulation layer is superimposed and the attention layer is superimposed and then connected in series, and The factor controls the number of FM layers and attention layers in the total number of layers, where is the proportion of FM layers in the total number of layers.

[0042] Furthermore, the frequency modulation layer is used to capture local features, including: firstly applying a block-based fast Fourier transform The input features Transform to the frequency domain; then, introduce a learnable matrix to suppress or amplify all frequency components by multiplying the elements in the frequency domain to obtain the frequency modulation feature ; Then use the inverse Fourier transform And reconstruct to obtain refined output features , the expression is: ;

[0043] ;

[0044] ;

[0045] In the formula, is the input feature of the FM layer, is the feature obtained through the forward propagation network, is the frequency modulation feature, is the output feature of the FM layer, is layer normalization, is a 1×1 convolutional layer, is the activation function, For block operation, is element-wise multiplication, is the learnable matrix, For block merging operations, is the fast Fourier transform, is the inverse fast Fourier transform, For multi-layer perceptron operations.

[0046] The attention layer is used to capture global attributes or semantic features, including: first performing layer normalization and multi-head attention operations on the input features of the attention layer, then performing layer normalization and multi-layer perceptron operations in sequence, and finally outputting; among which, the multi-layer perceptron operation is used for channel mixing in the attention layer.

[0047] Furthermore, the fused global features and local features are weighted in the spectral dimension to obtain deep features, and then the classification results of the target object are obtained, including: first learning key information through one-dimensional convolution, then highlighting significant features through activation functions, and finally The function gets the prediction result, the expression is:

[0048] ;

[0049] ;

[0050] In the formula, is the deep feature, To predict the classification results, For full connection operation, and There are two activation functions.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] (1) The multi-source remote sensing data classification method provided by the present invention uses multi-scale convolution and channel-level fusion operations on the cubic blocks of each single-source remote sensing image to realize the extraction and fusion of shallow features; then the frequency feature decomposition module based on multi-head self-attention is used to capture the different frequency features of each single-source remote sensing image with different window shapes, and then the multi-source data is feature-fused using the same-frequency feature fusion module to realize the directional frequency feature decomposition and fusion of the multi-source data, and the multi-source fusion features extracted based on the multi-source remote sensing data are obtained as the basis for classifying the target object. Compared with the classification method based on single-source remote sensing data, the present invention provides more comprehensive information for the classification of the target object by fusing the multi-source remote sensing data, and can fully capture the complex ground features, making the classification results more accurate. In addition, the frequency modulation layer and the attention layer are combined to realize local and global feature learning through frequency modulation, obtain the features of different frequency components, and extract deep features based on this to complete the accurate classification of the target object;

[0053] (2) The multi-source remote sensing data classification method provided by the present invention uses a deep learning classification method with stronger representation and generalization capabilities to extract deeper image features, and learns the spatial and spectral features of remote sensing images under low training samples, thereby improving classification accuracy and obtaining more discriminant features to obtain good classification results;

[0054] (3) Introduction of the present invention The factor controls the number of FM layers and attention layers. By flexibly changing the number of FM layers and attention layers, it helps to accurately capture global and local features. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flow chart of the multi-source remote sensing data classification method provided in Example 1 of the present invention;

[0056] Figure 2 This is a schematic diagram of extracting shallow features of remote sensing data in Example 1 of the present invention;

[0057] Figure 3 It is a schematic diagram of obtaining frequency characteristics by using the frequency characteristic decomposition module in Embodiment 2 of the present invention;

[0058] Figure 4 It is a schematic diagram of obtaining same-frequency fusion features by using the same-frequency feature fusion module in Embodiment 1 of the present invention;

[0059] Figure 5 It is a schematic diagram of the frequency modulation layer provided in Embodiment 1 of the present invention;

[0060] Figure 6 A schematic diagram of the attention layer provided in Embodiment 1 of the present invention;

[0061] Figure 7 This is a diagram of the classification results of different methods in Example 3 of the present invention on the Houston 2013 dataset. DETAILED DESCRIPTION

[0062] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0063] It should be noted that remote sensing technology is a technology that detects and identifies targets by sensing electromagnetic waves, visible light, and infrared rays Q reflected or radiated by targets from a long distance. Remote sensing satellites are equipped with relevant remote sensing sensors, which use remote sensors to collect electromagnetic wave information radiated or reflected by targets on the earth or in the atmosphere, and record it. The information is sent back to the ground by signal and transmission equipment, and visible images are obtained through electromagnetic wave conversion and recognition, which is what we often call satellite images.

[0064] In the process of remote sensing digital image processing, the spatial characteristics of ground objects are mainly reflected by the changes in spectral characteristics. Multispectral technology refers to the spectrum detection technology that can simultaneously obtain multiple optical spectrum bands and expand to infrared and ultraviolet light based on visible light.

[0065] Example 1

[0066] The present invention provides a multi-source remote sensing data classification method, such as Figure 1 As shown, the method comprises the following steps:

[0067] Step 1: Obtain various types of remote sensing data of the target object.

[0068] In a specific embodiment, two or three remote sensing images of the target object are selectively acquired according to the actual needs of the image research.

[0069] It should be noted that after obtaining the hyperspectral image, PCA dimensionality reduction processing is required to extract the main features in the image and reduce redundant information.

[0070] Step 2: Extract shallow features of each remote sensing data, including:

[0071] Step 21: Divide the remote sensing data into image patches.

[0072] Specifically, patch partitioning of an image is to divide the image into smallest units.

[0073] In the embodiment of acquiring hyperspectral and LiDAR images, the hyperspectral image after dimensionality reduction is and LiDAR images The pixels of the image are taken one by one to obtain the hyperspectral image patch block. and LiDAR image patch blocks ,in, is the length and width of the patched pixel. For the The query tensor of the head , key tensor Sum value tensor Dimension.

[0074] Step 22: Perform multi-scale convolution operations on the divided patch blocks to obtain corresponding multi-scale features.

[0075] Specifically, the multi-scale convolution operation includes: performing multiple different convolution operations on the divided images, and performing batch normalization operations and ReLU operations in sequence after each convolution operation to obtain corresponding multi-scale features.

[0076] like Figure 2 As shown, in the embodiment of acquiring hyperspectral and LiDAR images, the hyperspectral image patch block and LiDAR image patch blocks 3×3 convolution + batch normalization + ReLU, 5×5 convolution + batch normalization + ReLU and 7×7 convolution + batch normalization + ReLU operations are performed respectively, and the calculation expression is:

[0077] ;

[0078] ;

[0079] in, is the convolution kernel size, set to 3, 5, 7..., For the Batch normalization function at scale, is a common activation function. and They are the hyperspectral image and LiDAR image. Output features at each scale.

[0080] Step 23: In any preset channel, first, the multiple scale features are superimposed along the channel dimension to obtain channel fusion features.

[0081] Specifically, the expression of channel fusion features is:

[0082] ;

[0083] in, For the channels, , , Respectively The output features of three different scales of channels, For the The channel fusion features of channels, This is a channel stacking operation.

[0084] In the embodiment of acquiring hyperspectral and LiDAR images, three different scale output features of the hyperspectral and LiDAR images are calculated respectively. Using the above expressions, multiple channel fusion features corresponding to the hyperspectral image and multiple channel fusion features corresponding to the LiDAR image are calculated respectively in each preset channel dimension.

[0085] Next, each channel fusion feature is subjected to element-wise summation, averaging, and maximum pooling operations along the channel dimension to extract channel-level attributes and obtain the corresponding sum feature map, average feature map, and maximum pooling feature map.

[0086] Specifically, the expression is:

[0087] ;

[0088] ;

[0089] ;

[0090] in, The feature map obtained by performing the element-wise sum operation, To perform the element-wise sum operation, The feature map obtained by performing the average operation, To perform the averaging operation, To perform the feature map obtained by the maximum pooling operation, To perform the maximum pooling operation.

[0091] Then, the three feature maps are concatenated and channel fusion is performed using 3×3 convolution to obtain low-dimensional channel features.

[0092] Specifically, the expression is:

[0093] ;

[0094] in, For the The low-dimensional channel features of channels.

[0095] In the embodiment of acquiring hyperspectral and LiDAR images, low-dimensional channel features of each channel are obtained for the hyperspectral image, and low-dimensional channel features of each channel are also obtained for the LiDAR image.

[0096] Step 24: Concatenate the low-dimensional channel features of all channels to obtain the shallow features of the remote sensing data.

[0097] Specifically, the expression is:

[0098] ;

[0099] in, is the shallow feature of remote sensing data, The first, second and The low-dimensional channel features of channels.

[0100] In the embodiment of obtaining hyperspectral and LiDAR images, the low-dimensional channel features of each channel of the hyperspectral image are spliced ​​by the above method to obtain the shallow features of the hyperspectral image At the same time, the low-dimensional channel features of each channel of the LiDAR image are spliced ​​to obtain the shallow features of the LiDAR image .

[0101] So far, each type of remote sensing data has performed the same steps and independently completed the extraction of shallow features. Since different remote sensing technologies have their own advantages and disadvantages, the shallow features of each type of remote sensing data also contain the data characteristics of its own technology. Next, the features of different types of remote sensing data need to be fused.

[0102] Step 3: Input the shallow features into the frequency feature decomposition module to obtain multiple preset frequency features for each remote sensing data.

[0103] It should be noted that the frequency feature decomposition module is built based on the frequency domain Transformer, which is used to extract the features of different frequencies of remote sensing data using different windows, so as to facilitate a more comprehensive analysis of remote sensing data.

[0104] Specifically, before frequency division, the shallow features of the input frequency feature decomposition module are first convolved and normalized to obtain processed shallow features.

[0105] In the embodiment of acquiring hyperspectral and LiDAR images, the expression is:

[0106] ;

[0107] ;

[0108] in, is the shallow feature of the processed hyperspectral image, is the shallow feature of the processed LiDAR image, Normalize the layer.

[0109] The process of obtaining multiple frequency features from the processed shallow features includes: dividing the processed shallow features into head, and Divide into Equal parts, of which, is the number of frequency features, so that each equal portion is used to calculate one frequency feature; is the default value.

[0110] Each frequency feature corresponds to a preset window shape. For any equal part, each head is first divided into non-overlapping windows evenly using the window shape corresponding to the frequency feature, and then the attention of each window is calculated; then the frequency feature of each head is obtained; finally, the frequency features of each head are spliced ​​to obtain the frequency feature corresponding to the equal part.

[0111] In the embodiments of acquiring hyperspectral and LiDAR images, the preset frequency features include: low-frequency features, high-frequency features, vertical features, and horizontal features. In these embodiments, the low-frequency features, high-frequency features, vertical features, and horizontal features of the hyperspectral image, and the low-frequency features, high-frequency features, vertical features, and horizontal features of the LiDAR image are obtained by the above method.

[0112] Step 4: Input the frequency characteristics of all remote sensing data into the same-frequency feature fusion module to obtain the corresponding multiple same-frequency fusion features.

[0113] The same-frequency feature fusion module performs feature fusion on the same frequency features, including:

[0114] Step 41: For any frequency component, add the frequency components from all remote sensing data element by element to obtain the same frequency component ;Will Perform global average pooling on the channel dimension, and then pass through the channel attention module to obtain the channel weight, which is expressed as:

[0115] ;

[0116] In the formula, is the channel weight, is a 1×1 convolutional layer, Output of global average pooling in the channel dimension.

[0117] Step 42: Global average pooling and global maximum pooling are performed in the spatial dimension, and then the spatial weight is obtained through the spatial attention module. , the expression is:

[0118] ;

[0119] In the formula, is the spatial weight, is a 7×7 convolutional layer, for The output of global average pooling in the spatial dimension, for Output of global max pooling in the spatial dimension.

[0120] Step 43: According to the broadcasting rules, the channel weights are added by addition. and spatial weight Fusion to get the rough weight ;Will and Each channel of is rearranged by a rearrangement operation to obtain a fine weight , the expression is:

[0121] ;

[0122] In the formula, is the fine weight, is the sigmoid function, For group convolution, the number of groups is set to the number of channels. For channel rearrangement operation, is the coarse weight.

[0123] Step 44: Based on the frequency components and detailed weights of all remote sensing data, combined with residual connection, a weighted summation method is adopted to obtain the same-frequency fusion features of the frequency features.

[0124] In an embodiment of acquiring hyperspectral and LiDAR images, such as Figure 4 As shown, the same-frequency feature 1 from the hyperspectral image and the same-frequency feature 2 from the LiDAR image are fused through the above method to obtain the same-frequency fusion feature, wherein the same-frequency fusion feature includes: high-frequency fusion feature, low-frequency fusion feature, vertical fusion feature and horizontal fusion feature.

[0125] Step 5: Concatenate and fuse multiple same-frequency fusion features to obtain multi-source fusion features.

[0126] In an embodiment of acquiring hyperspectral and LiDAR images, the obtained low-frequency fusion features, high-frequency fusion features, vertical fusion features, and horizontal fusion features are spliced ​​to obtain multi-source fusion features.

[0127] Step 6: Pass the multi-source fusion features through the superimposed frequency modulation layer and attention layer in sequence to obtain the fused global features and local features.

[0128] Specifically, the frequency modulation layer and the attention layer adopt a phased architecture layout, that is, multiple frequency modulation layers and multiple attention layers are first connected in series and stacked, and then the two parts are connected in series again.

[0129] Among them, the frequency modulation layer is used to capture local features, such as Figure 5 As shown, it includes: firstly applying a block-based fast Fourier transform Input features Transform to the frequency domain; then, introduce a learnable matrix to suppress or amplify all frequency components by multiplying the elements in the frequency domain to obtain the frequency modulation feature ; Then use the inverse Fourier transform And reconstruct to obtain refined output features , the expression is: ;

[0130] ;

[0131] ;

[0132] In the formula, is the input feature of the FM layer, is the feature obtained through the forward propagation network, is the frequency modulation feature, is the output feature of the FM layer, is a 1×1 convolutional layer, is the activation function, For block operation, is element-wise multiplication, is the learnable matrix, For block merging operations, is the fast Fourier transform, is the inverse fast Fourier transform, For multi-layer perceptron operations.

[0133] Attention layers are used to capture global attributes or semantic features, such as Figure 6 As shown, this part is a standard attention layer, including: First, perform layer normalization and multi-head attention operations in sequence, then perform layer normalization and multi-layer perceptron operations in sequence, and finally output. The expression is:

[0134] ;

[0135] ;

[0136] in, is the input feature of the attention layer, is the feature obtained after layer normalization and multi-head attention. is the output feature of the attention layer, For multi-head attention operation, Multilayer Perceptron operations for channel mixing in attention layers.

[0137] In some specific embodiments, by introducing The factor controls the number of FM layers and attention layers in the total number of layers, where is the proportion of FM layers in the total number of layers.

[0138] As is known, the frequency modulation layer has the disadvantage of not being able to accurately process global attributes or semantic features, while the attention layer has the disadvantage of not being able to accurately capture local features. Here, the two are combined and introduced The factor flexibly changes the number of FM layers and attention layers, which helps to accurately capture global and local features.

[0139] Step 7: Weight the fused global features and local features in the spectral dimension to obtain deep features, and then obtain the predicted classification results of the target object, including: first learning key information through one-dimensional convolution, then highlighting significant features through activation function, and finally The function gets the prediction result, the expression is:

[0140] ;

[0141] ;

[0142] In the formula, is the deep feature, To predict the classification results, For full connection operation, and There are two activation functions.

[0143] Example 2

[0144] Based on Example 1, this example provides a process of obtaining frequency characteristics of two types of remote sensing data by using a frequency characteristic decomposition module in the example of obtaining hyperspectral and LiDAR images.

[0145] In this embodiment, Figure 3 As shown, the frequency features include: low-frequency features, high-frequency features, vertical features and horizontal features.

[0146] Specifically, the window shape corresponding to the low-frequency feature is: ; The window shape corresponding to the high-frequency feature is: ; The window shape corresponding to the vertical feature is: ; The window shape corresponding to the horizontal feature is: ,in, is the window size index, Take a positive integer.

[0147] It should be noted that is the window size index, which represents the size of the window. The value determines the number of windows obtained by the division.

[0148] First, the shallow features of the processed hyperspectral image And the shallow features of the processed LiDAR image The average spectral dimension is head, and Divide the head into 4 equal parts.

[0149] The head in the first half ( ) is used to calculate low-frequency features, using The window shape divides each head in the first equal part into windows, each window contains tokens, where token refers to the smallest unit in the window.

[0150] The head in the second half ( ) is used to calculate high-frequency features, using The window shape divides each head in the second equal part into windows, each window contains tokens.

[0151] The head in the third section ( ) is used to calculate the vertical feature, using The window shape divides each head in the third equal part into windows, each window contains tokens.

[0152] The head in the fourth part ( ) is used as the calculation level feature, using The window shape divides each head in the first equal part into windows, each window contains tokens.

[0153] Below is the shallow features of the processed LiDAR image Taking the calculation of the vertical features of the third part of the head as an example, the specific calculation process is shown, including:

[0154] S1: The window shape will be The heads are evenly divided into non-overlapping Windows , ; ; The shallow features of the processed LiDAR image spectral dimension.

[0155] No. The dimensions of the query tensor, key tensor, and value tensor for each header are ;in, .

[0156] S2: Calculate the The attention of each window in the head; The attention calculation result of the window is:

[0157] ;

[0158] In the formula, For the The attention calculation results of the windows are: To calculate the attention operation, , , Respectively The query matrix, key matrix and value matrix of each header.

[0159] S3: According to The attention of each window in the head gets Vertical characteristics of the head, The vertical characteristics of the head are:

[0160] ;

[0161] In the formula, For the The vertical characteristics of the head, , , The first, second and The attention calculation results of the window;

[0162] S4: stitch the vertical features of all heads in the third equal part to obtain the vertical features of the shallow features of the processed LiDAR image; the expression is:

[0163] ;

[0164] In the formula, is the vertical feature of the shallow features of the processed LiDAR image, For splicing operation, , and Respectively , and The vertical characteristics of the head.

[0165] It should be noted that the process of calculating low-frequency features, high-frequency features and horizontal features is similar to the above method of calculating vertical features.

[0166] The frequency feature decomposition module in the present invention adopts the above method to extract different frequency features with different window postures, from the perspective of frequency domain decomposition, and through the multi-head attention mechanism, makes good use of multiple directional frequency features to enhance the accuracy of classification results.

[0167] Example 3

[0168] This embodiment provides an experimental process for verifying the classification effect of the method proposed in the present invention.

[0169] The hardware platform of this experiment is a high-performance computer with the following configurations: Intel Core i9-11900K, CPU operation speed of 3.50GHz, eight-core processor, 32G memory, graphics card of NVIDIA GTX3070Ti, and software platform of Python3.8 under Windows11 system environment. The method we proposed is implemented in the PyTorch framework.

[0170] 1. Experimental Data and Sample Division

[0171] In order to evaluate the classification effect of the proposed method, the Houston2013 dataset is selected to verify the performance of the proposed method. The Houston2013 dataset was collected by the National Center for Airborne Laser Mapping at the University of Houston campus, and the LiDAR data based on HSI and DSM in the area were obtained. The spatial resolution of HSI and LiDAR data is 2.5m, containing a total of 349×1905 pixels. Among them, HSI has 144 bands ranging from 0.38 nm to 1.05 nm.

[0172] The dataset is labeled with 15029 ground truth samples and samples selected from 15 categories. The size of each category sample and the division of the number of experimental training samples and test set samples are shown in Table 1.

[0173] Classification accuracy evaluation indicators of hyperspectral images: Three commonly used evaluation indicators, namely overall classification accuracy (OA), average classification accuracy (AA) and Kappa coefficient, are used to measure the classification accuracy.

[0174] Table 1: Number of samples in the training and test sets of the Houston2013 dataset

[0175] .

[0176] 2. Parameter settings

[0177] In this experiment, the learning rate, space size, and dropout rate are three parameters that have a significant impact on the experiment. Taking the Houston2013 dataset as an example, the experimental parameters are evaluated in detail.

[0178] 1) Learning rate: In the experiment, a higher learning rate may lead to rapid model convergence, but it may also cause training instability or even loss function divergence, while a lower learning rate usually makes convergence more stable, but the training process is slower. In addition, the learning rate also affects the model's ability to escape the local optimal solution. A learning rate that is too large may cause oscillation and failure to converge, while a learning rate that is too small may stagnate at the local optimum. Therefore, this experiment selects different learning rates to test the impact of the learning rate. The selected learning rates include: 0.01, 0.005, 0.001, 0.0007, 0.0005, 0.0003, 0.0001, 0.00007, 0.00005, 0.00003 and 0.00001. The experimental results show that the classification effect is best when the learning rate is 0.0001.

[0179] 2) Spatial size: Because the extraction of image spatial features is heavily dependent on the size of the spatial domain area. And larger spatial input will provide more opportunities to learn more spatial features. Smaller spatial size can capture more detailed local features, enhance the model's sensitivity to small objects and changes, and is suitable for identifying complex landform types. However, too small a spatial size may lead to information sparsity and increase noise interference, thereby affecting classification accuracy. On the contrary, a larger spatial size can extract richer contextual information and enhance the learning of global features. It is suitable for the classification of larger areas, but may ignore details and lead to category confusion. Therefore, choosing the right spatial size is very important for improving classification performance. When the number of spectral channels is fixed, the optimal learning rate, the batch size is 64, and the number of training iterations is 100, the classification accuracy results under different spatial sizes are shown in Table 2.

[0180] 3) Discard rate:

[0181] From Table 3, we can see that when the input data space size is 8 8. When the discard rate is 0.5, the classification effect is the best. In order to optimize the classification performance, this experiment selects the optimal discard rate.

[0182] Table 2: Classification accuracy at different spatial sizes

[0183] .

[0184] Table 3: Classification accuracy at different dropout rates

[0185] .

[0186] 3. Experimental Results

[0187] In order to ensure the accuracy of the experimental results, the experiment was repeated 10 times and the average value was taken.

[0188] In order to verify the effectiveness and superiority of the proposed method, the present invention conducts experimental comparisons with some traditional methods and mainstream deep learning methods.

[0189] The compared methods include: linear self-attention fusion algorithm LSAF, coupled CNN fusion algorithm CoupledCNN, classification algorithm based on coupled adversarial learning CALC, hierarchical CNN and Transformer algorithm HCT, and network method g2 for global and local feature fusion.

[0190] The experimental results of the classification performance comparison of different methods on the Houston2013 dataset are shown in Table 4.

[0191] From the results in Table 4, it can be seen that the OA value, AA value and Kappa coefficient value of the proposed method on the Houston2013 dataset are higher than those of other mainstream deep learning classification methods.

[0192] Among them, the OA value is 4.26% higher than LSAF, 2.07% higher than CoupledCNN, 2.02% higher than CALC, 1.89% higher than HCT, and 0.30% higher than the g2 classification method.

[0193] The AA value is 3.45% higher than LSAF, 1.48% higher than CoupledCNN, 1.62% higher than CALC, 1.63% higher than HCT, and 0.42% higher than g2.

[0194] The Kappa coefficient value is 4.61% higher than LSAF, 2.24% higher than CoupleCNN, 2.18% higher than CALC, 2.06% higher than HCT, and 0.37% higher than g2.

[0195] These three indicators all demonstrate that the classification performance of the method provided by the present invention is superior to other methods.

[0196] Table 4: Classification performance of different methods on the Houston2013 dataset

[0197] .

[0198] In addition, the classification diagrams of different methods on the Houston2013 dataset are shown in the figure below. Figure 7 As shown in the figure, it can be seen that compared with the ground truth (GT), the final classification results of the linear self-attention fusion algorithm (LSAF), the coupled CNN fusion algorithm (CoupledCNN), the classification algorithm based on coupled adversarial learning (CALC), the hierarchical CNN and Transformer algorithm (HCT), and the network method of global and local feature fusion (g2) all have a large number of messy spots, and some areas are misclassified. The g2 method has a good classification effect, but there are also fewer messy spots in the middle and lower positions. In the classification result map predicted by the method proposed in the present invention, the ground objects are basically completely correctly classified, almost no spots can be seen, and it is relatively smooth in homogeneous areas.

[0199] It can be seen from this that the method proposed in the present invention does not require a complex and large multi-level network. We approach from the perspective of frequency domain decomposition and modulation, and make good use of a variety of directional frequency features through a multi-head attention mechanism. We also perform multi-source co-frequency fusion through an adaptive method, and finally comprehensively extract local and global features. By further weighting the spectral information to extract deep feature information, we achieve a relatively ideal classification effect, effectively improve the accuracy of multi-source remote sensing data joint classification, and outperform several advanced classification methods.

[0200] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-source remote sensing data classification method, characterized in that: include: Obtain various types of remote sensing data of the target object; Extract shallow features of each type of remote sensing data respectively; Inputting the shallow features into a pre-built frequency feature decomposition module to obtain a plurality of preset frequency features for each remote sensing data; The frequency features of all remote sensing data are input into a pre-built same-frequency feature fusion module, and the same-frequency feature fusion module performs feature fusion on the same frequency features to obtain corresponding multiple same-frequency fusion features; The multiple same-frequency fusion features are concatenated and fused to obtain multi-source fusion features; Passing the multi-source fusion features through the superimposed frequency modulation layer and attention layer in sequence to obtain fused global features and local features; The fused global and local features are weighted in the spectral dimension to extract depth information and obtain the predicted classification results of the target object.

2. The multi-source remote sensing data classification method according to claim 1, characterized in that: The shallow features of each type of remote sensing data are extracted, including: Divide remote sensing data into image patches; Perform multi-scale convolution operations on the divided patch blocks to obtain corresponding multi-scale features; In any preset channel, Superimposing the multiple scale features along the channel dimension to obtain channel fusion features; Perform element summation, averaging and maximum pooling operations on the channel fusion features along the channel dimension to obtain corresponding sum feature maps, average feature maps and maximum pooling feature maps; The sum feature map, the average feature map and the maximum pooling feature map are concatenated, and convolution is used to further perform channel fusion to obtain low-dimensional channel features of the channel; The low-dimensional channel features of all channels are concatenated to obtain the shallow features of the remote sensing data.

3. The multi-source remote sensing data classification method according to claim 2, characterized in that: The multi-scale convolution operation includes: performing 3×3 convolution, 5×5 convolution and 7×7 convolution on the divided patch blocks respectively, and performing batch normalization operation and ReLU operation in sequence after each convolution operation to obtain corresponding three scale features.

4. The multi-source remote sensing data classification method according to claim 1, characterized in that: The frequency feature decomposition module is constructed based on the frequency domain Transformer; the multiple preset frequency features of each shallow feature are obtained, including: The input shallow features are passed through the convolution and normalization layers to obtain the processed shallow features ; The shallow features after processing Averaged over the spectral dimension In the head, Divide into Equal parts, of which, is the number of frequency features, so that each equal portion is used to calculate one frequency feature; is the default value; Each frequency feature corresponds to a preset window shape; For any equal part, first use the window shape of the corresponding frequency feature to evenly divide each head into non-overlapping windows, and then calculate the attention of each window; then obtain the frequency feature of each head; finally, splice the frequency features of each head to obtain the frequency feature corresponding to the equal part.

5. The multi-source remote sensing data classification method according to claim 4, characterized in that: The frequency characteristics include: low frequency characteristics, high frequency characteristics, vertical characteristics and horizontal characteristics; The window shape corresponding to the low-frequency feature is: , is the window size index, Take a positive integer; In the header used to calculate low-frequency features, each window contains token; the token refers to a minimum unit in the window; The window shape corresponding to the high-frequency feature is: ; In the header used to calculate high-frequency features, each window contains Tokens; The window shape corresponding to the vertical feature is: ; In the header used to calculate vertical features, each window contains Tokens; The window shape corresponding to the horizontal feature is: ; In the header used to calculate horizontal features, each window contains tokens.

6. The multi-source remote sensing data classification method according to claim 5, characterized in that: The process of obtaining the vertical feature includes: The first The heads are evenly divided into non-overlapping Windows , is the number of windows, , The shallow features after processing length or width; , The shallow features after processing The spectral dimension of The query tensor of the head , key tensor Sum value tensor The dimension is ,in, ; ; Calculate the The attention of each window in the head; The attention calculation result of the window is: ； In the formula, For the The attention calculation results of the windows are: To calculate the attention operation, , , Respectively The query matrix, key matrix and value matrix of each head; According to The attention of each window in the head gets The vertical characteristics of the head are expressed as: ； In the formula, For the The vertical characteristics of the head, , , The first, second and The attention calculation results of the window; The vertical features of all heads used to calculate the vertical features are concatenated to obtain the vertical features of the shallow features. The expression is: ； In the formula, is the vertical feature of the shallow feature, For splicing operation, , and Respectively , and The vertical characteristics of the head.

7. The multi-source remote sensing data classification method according to claim 1, characterized in that: The same-frequency component fusion module performs feature fusion on the same frequency components to obtain a corresponding plurality of same-frequency fusion features, including: For any frequency component, The frequency components from all remote sensing data are added element by element to obtain the same frequency components. ; Will Perform global average pooling on the channel dimension, and then pass through the channel attention module to obtain the channel weight, which is expressed as: ； In the formula, is the channel weight, is a 1×1 convolutional layer, To obtain the maximum value, The output of global average pooling in the channel dimension; Will Global average pooling and global maximum pooling are performed in the spatial dimension, and then the spatial weight is obtained through the spatial attention module. The expression is: ； In the formula, is the spatial weight, is a 7×7 convolutional layer, for The output of global average pooling in the spatial dimension, for The output of global maximum pooling in the spatial dimension; According to the broadcast rule, the channel weight and the spatial weight are fused through addition operation to obtain the coarse weight; The coarse weight and each channel of the same frequency component are rearranged through the rearrangement operation to obtain the fine weight, which is expressed as: ； In the formula, is the fine weight, is the sigmoid function, For group convolution, the number of groups is set to the number of channels. For channel rearrangement operation, is the coarse weight; According to the frequency components and detailed weights of all remote sensing data, combined with residual connection, a weighted summation method is adopted to obtain the same-frequency fusion features of this frequency feature.

8. The multi-source remote sensing data classification method according to claim 1, characterized in that: The frequency modulation layer and the attention layer adopt a phased architecture in which the frequency modulation layer is superimposed and the attention layer is superimposed and then connected in series, and introduces The factor controls the number of FM layers and attention layers in the total number of layers, where is the proportion of FM layers in the total number of layers.

9. The multi-source remote sensing data classification method according to claim 8, characterized in that: The frequency modulation layer is used to capture local features, including: first by applying a block-based fast Fourier transform Input features Transform to the frequency domain; then, introduce a learnable matrix to suppress or amplify all frequency components by multiplying the elements in the frequency domain to obtain the frequency modulation feature ; Then use the inverse Fourier transform And reconstruct to obtain refined output features , the expression is: ; ； ； In the formula, is the input feature, is the feature obtained through the forward propagation network, is the frequency modulation feature, is the refined output feature, is layer normalization, is a 1×1 convolutional layer, is the activation function, For block operation, is element-wise multiplication, is the learnable matrix, For block merging operations, is the fast Fourier transform, is the inverse fast Fourier transform, Operate for multi-layer perceptron; The attention layer is used to capture global attributes or semantic features, including: first performing layer normalization and multi-head attention operations on the features of the input attention layer, then performing layer normalization and multi-layer perceptron operations in sequence, and finally outputting; wherein the multi-layer perceptron operation is used for channel mixing in the attention layer.

10. The multi-source remote sensing data classification method according to claim 1, characterized in that: The method weights the fused global features and local features in the spectral dimension to obtain deep features, and then obtains the classification results of the target object, including: first learning key information through one-dimensional convolution, then highlighting significant features through activation functions, and finally The function gets the prediction result, the expression is: ； ； In the formula, is the deep feature, To predict the classification results, For full connection operation, and There are two activation functions.

Images