Multi-source remote sensing image fusion classification method

Abstract

The invention relates to a multi-source remote sensing image fusion classification method, which relates to the technical field of multi-source remote sensing image fusion, and comprises the following steps of: acquiring data of a hyperspectral image and a radar image, extracting multi-scale convolution features, extracting spectral features in the hyperspectral image through spectral four-way scanning, and classifying the spectral features in the hyperspectral image. Spatial information of a hyperspectral image and spatial information of a radar image are obtained through four-direction spatial feature scanning, and spectral information in the hyperspectral image and spatial information in the radar image as well as spatial information in the hyperspectral image and spatial information in the radar image are input into a spatial feature extraction module; the first spectral-spatial features and the second spatial features are obtained, the features are fused through tensor splicing, spatial information in the hyperspectral image and spatial information in the radar image are combined, and the accuracy of a hyperspectral image classification task is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to a multi-source remote sensing image fusion classification method, belonging to the multi-source remote sensing image fusion technical field. BACKGROUND

[0002] Hyperspectral images contain dozens or even hundreds of spectral bands, with high spectral resolution, containing rich spectral feature information of ground objects, ranging from visible spectrum to infrared spectrum, greatly improving the detection and identification ability of ground object classes. However, many materials of the same object have similar spectral responses, so it is difficult to distinguish them using only hyperspectral data. Radar images obtain the spatial characteristics of ground objects through microwave reflection, which can effectively identify the height of objects, has all-weather and cloud-penetrating capabilities, and provides additional spatial information, but for objects of the same height, radar cannot effectively identify them. Therefore, on the basis of hyperspectral data, the fusion of radar data can effectively utilize the complementary information of the two kinds of image data to improve the accuracy and robustness of downstream tasks, and has become one of the hot research directions in remote sensing image research.

[0003] Machine learning and pattern recognition are rapidly developing fields in recent years, which provide new ideas and methods for remote sensing images. Although this method improves the fusion effect to some extent, it cannot fully capture the rich information in the data when facing complex and variable remote sensing data. With the application of deep learning models of convolutional neural networks (CNN) in automatic feature extraction and fusion of images, the accuracy and efficiency of multi-source remote sensing image fusion have been greatly improved. Since most CNN methods use standard convolution kernels, they often ignore the information of adjacent bands, so there is a problem of insufficient utilization of spectral information of hyperspectral images and spatial information in radar images. SUMMARY

[0004] The purpose of the present application is to provide a multi-source remote sensing image fusion classification method, which obtains sufficient spectral information and spatial information by designing different scanning mechanisms, and effectively combines the spatial and spectral information in the fusion feature process, thereby improving the accuracy of multi-source remote sensing image fusion and classification.

[0005] In order to achieve the above-mentioned purpose / in order to solve the above-mentioned technical problems, the present application is realized by adopting the following technical scheme.

[0006] In one aspect, the present application provides a multi-source remote sensing image fusion classification method, comprising:

[0007] obtaining a hyperspectral image and a radar image of a region to be measured;

[0008] obtaining hyperspectral image feature information and radar image feature information under multi-scale convolution conditions;

[0009] Extracting spectral information and spatial information in hyperspectral image feature information, and spatial information in radar image feature information;

[0010] According to the spectral information in the hyperspectral image feature information and the spatial information in the radar image feature information, a first spatial-spectral feature is obtained by a splicing operation;

[0011] According to the spatial information in the hyperspectral image feature information and the spatial information in the radar image feature information, a second spatial feature is obtained by a splicing operation;

[0012] The first spatial-spectral feature and the second spatial feature are spliced to obtain a fusion feature of the hyperspectral image;

[0013] The fusion feature is subjected to a self-attention feature extraction operation, and a classification probability is calculated for the extracted self-attention feature through a Softmax function, so as to finally determine a classification result.

[0014] Further, the hyperspectral image feature information and the radar image feature information under the condition of multi-scale convolution are obtained, and specifically include:

[0015] The hyperspectral image and the radar image are subjected to a multi-scale convolution operation, and then normalized and activated to obtain the hyperspectral feature and the radar feature under the condition of multi-scale convolution The expression is:

[0016] ;

[0017] Wherein, is one of the hyperspectral feature and the radar feature under the condition of the convolution kernel scale s, and the convolution kernel scale is Take: , and ;

[0018] is one of the hyperspectral image and the radar image , is a convolution operation with a convolution kernel scale of , is a batch normalization operation, is an activation operation;

[0019] The hyperspectral feature and the radar feature under the condition of multi-scale convolution are spliced with the features of the same dimension, and the expression is:

[0020] ​ ;

[0021] in, It is one of the i-th dimension hyperspectral image stitching features and the i-th dimension radar image stitching features;

[0022] The scale of the i-th dimension convolution kernel is One of the hyperspectral features and radar features under convolution operations. The scale of the i-th dimension convolution kernel is One of the hyperspectral features and radar features under convolution operations. The scale of the i-th dimension convolution kernel is One of the hyperspectral features and radar features under convolution operations;

[0023] Extract the maximum, average, and summed features from the i-th dimension hyperspectral image stitching features and the i-th dimension radar image stitching features respectively, perform a stitching operation, and obtain hyperspectral image feature information under multi-scale convolution conditions. and radar image feature information The specific expression is:

[0024] ;

[0025] in, For hyperspectral image feature information and radar image feature information One of them;

[0026] For splicing operations, For maximum value operation, For averaging operations, For summation operations.

[0027] By employing multi-scale convolution kernels ( , and Image features are extracted and fused using three pooling strategies: max pooling, average pooling, and sum pooling. This enhances the model's ability to perceive ground features at different scales and effectively captures local details and global contextual information in the image, thereby improving the richness and discriminative power of feature representation.

[0028] Furthermore, the method for extracting spatial information from the radar image feature information is expressed as follows: ;

[0029] ;

[0030] ;

[0031] ;

[0032] ;

[0033] ;

[0034] wherein: is a layer normalization operation, is a linear layer operation, is a normalized feature of the radar image feature information, is an activation feature of , is a multiplication feature of , is a normalized feature of , is a multiplication feature of , is spatial information extracted from the radar image feature information; is a DW convolution operation with a convolution kernel size of , is a Sigmoid activation function activation processing, is a spatial scanning operation.

[0035] wherein the S6 state selection operation is specifically: obtaining a long distance feature from a vector flattened after the spatial scanning operation, and then obtaining a mean value of the long distance feature and restoring the square size.

[0036] Further, the spatial information extraction method in the hyperspectral image feature information is expressed as:

[0037] ;

[0038] ;

[0039] ;

[0040] ;

[0041] ;

[0042] ;

[0043] wherein: is a layer normalization operation, is a linear layer operation, is a normalized feature of the hyperspectral image feature information, is an activation feature of , is a multiplication feature of S6 state selection feature under spatial scanning, is a normalized feature, is a multiplication feature, is spatial information extracted from hyperspectral image feature information.

[0044] Further, the spectral information extraction method in the hyperspectral feature information is expressed as:

[0045]

[0046]

[0047]

[0048]

[0049]

[0050] wherein: is a layer normalization operation, is a linear layer operation, is a normalized feature of hyperspectral image feature information, is an activation feature, is S6 state selection feature under spectral scanning, is a normalized feature, is a multiplication feature, is spectral information extracted from hyperspectral image feature information, is a spectral scanning operation.

[0051] Further, based on , by , the following formula is obtained:

[0052]

[0053]

[0054] wherein, the feature size is , , the horizontal and vertical size of the feature , respectively, and C is the number of spectral channels; is​​​​​​​​​ The number of pixels spread along the spectral channels.

[0055] To flatten the surface from left to right, To flatten the surface from top to bottom, To flatten the surface from right to left, To flatten the surface from bottom to top;

[0056] This is the vector obtained after flattening from left to right in the spatial dimension. This is the vector obtained after flattening from top to bottom in the spatial dimension. This is the vector obtained after flattening from right to left in the spatial dimension. This is the vector obtained after flattening from bottom to top in the spatial dimension. This is to perform a state selection operation on the vector generated in the j-th dimension.

[0057] Furthermore, based on ,pass get Use the following formula:

[0058] ;

[0059] ;

[0060] in, Feature size is , , Features The horizontal and vertical dimensions, where C is the number of spectral channels; for The number of pixels spread along the spectral channels.

[0061] To flatten the surface from left to right, To flatten the surface from top to bottom, To flatten the surface from right to left, To flatten the surface from bottom to top;

[0062] This is the vector obtained after flattening from left to right in the spatial dimension. This is the vector obtained after flattening from top to bottom in the spatial dimension. This is the vector obtained after flattening from right to left in the spatial dimension. This is the vector obtained after flattening from bottom to top in the spatial dimension. This is to perform a state selection operation on the vector generated in the j-th dimension.

[0063] Through spectral four-way scanning and spatial four-way scanning mechanisms, the hyperspectral and radar images are respectively unfolded in four directions (left→right, up→down, right→left, down→up) and state selection (S6 operation) is performed;

[0064] The long-distance dependency can be captured, and the modeling capability for complex ground structure and spectral variation is enhanced.

[0065] Further, based on , by get , using the following formula:

[0066] ;

[0067] ;

[0068] wherein, indicates the number of pixels unfolded along the horizontal direction of the spatial channel,

[0069] is a left-to-right flattening operation, is an up-to-down flattening operation, is a right-to-left flattening operation, is a down-to-up flattening operation;

[0070] is a vector obtained after the left-to-right flattening operation in the spectral dimension, is a vector obtained after the up-to-down flattening operation in the spectral dimension, is a vector obtained after the right-to-left flattening operation in the spectral dimension, is a vector obtained after the down-to-up flattening operation in the spectral dimension, is a state selection operation on the vector generated in the mth dimension.

[0071] Further, the splicing operation according to the spectral information in the hyperspectral image feature information and the spatial information feature in the radar image feature information is performed to obtain a first spatial-spectral feature, and the expression is:

[0072] ;

[0073] ;

[0074] wherein: is the first spatial-spectral feature, is the first spectral information splicing feature, is the first spatial-spectral feature, is a spatial scanning Mamba operation; is a pooling operation.

[0075] Further, the second spatial feature is obtained by splicing the spatial information in the hyperspectral image feature information and the spatial information in the radar image feature information, and the expression is:

[0076] ;

[0077] ;

[0078] Wherein: is the second spatial feature, is the second spatial information splicing feature, is the Mamba operation of spatial scanning; is the pooling operation.

[0079] Compared with the prior art, the present application has the beneficial effects: the present application aims to fuse hyperspectral and radar images, by designing different scanning mechanisms, sufficient spectral information and spatial information are obtained, and then the spectral information of the hyperspectral image and the spatial information of the radar (first spatial-spectral feature) are fused through the first path; the spatial information of the hyperspectral and radar (second spatial feature) is fused through the second path; finally, efficient fusion of spatial-spectral information is realized, and the classification accuracy is significantly improved. During the fusion feature process, the spatial-spectral information is effectively combined, and the effectiveness of multi-source remote sensing image fusion and classification is improved. BRIEF DESCRIPTION OF DRAWINGS

[0080] Figure 1 is the flowchart of the present application;

[0081] Figure 2 is the framework diagram of the present application;

[0082] Figure 3 is the Houston2013 hyperspectral image of the embodiment of the present application;

[0083] Figure 4 is the Houston2013 radar image of the embodiment of the present application;

[0084] Figure 5 is the classification result diagram of the present application;

[0085] Figure 6 is the CALC classification result diagram;

[0086] Figure 7 is the HCT classification result diagram;

[0087] Figure 8 is the ExViT classification result diagram;

[0088] Figure 9 is the FDNet classification result diagram. Detailed Implementation

[0089] It should be noted that:

[0090] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0091] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0092] Example 1

[0093] like Figures 1-2 The embodiment shown provides a multi-source remote sensing image fusion classification method, including:

[0094] Step 1: Acquire hyperspectral and radar images of the area to be measured, specifically including:

[0095] Acquiring the Houston 2013 hyperspectral image and radar images of the same area PCA dimensionality reduction is used to remove redundant band information in hyperspectral images;

[0096] Where H and W represent the length and width of the hyperspectral and radar images, respectively, which are 149 and 1905, and B represents the number of bands in the hyperspectral image, which is 144.

[0097] Step 2: Obtain hyperspectral image feature information and radar image feature information under multi-scale convolution conditions, specifically including:

[0098] Step 2.1: Perform multi-scale convolution operations on the hyperspectral image and radar image, and then obtain the hyperspectral features under the multi-scale convolution conditions through normalization and activation operations. and radar characteristics The expression is:

[0099] ;

[0100] in, Hyperspectral features at kernel scale s and radar characteristics One of them, the kernel size is Pick: , and ;

[0101] is a hyperspectral image and a radar image , is a convolution operation with a convolution kernel size of , is a batch normalization operation, is an activation operation;

[0102] Step 2.2: Concatenate the hyperspectral features and radar features with the same dimension under the condition of multi-scale convolution, and the expression is:

[0103] ;

[0104] wherein, is one of the i-th dimensional hyperspectral image concatenated features and the i-th dimensional radar image concatenated features, wherein i takes values [0, 64];

[0105] is one of the i-th dimensional Houston2013 hyperspectral features and Houston2013 radar features under the condition of convolution operation with a convolution kernel size of , is one of the i-th dimensional Houston2013 hyperspectral features and Houston2013 radar features under the condition of convolution operation with a convolution kernel size of , is one of the i-th dimensional Houston2013 hyperspectral features and Houston2013 radar features under the condition of convolution operation with a convolution kernel size of ;

[0106] Step 2.3: Extract the maximum feature, the average feature and the sum feature in the i-th dimensional hyperspectral image concatenated features and the i-th dimensional radar image concatenated features respectively, perform a concatenation operation, and obtain the hyperspectral image feature information and radar image feature information under the condition of multi-scale convolution, and the specific expression is:

[0107] ;

[0108] wherein, is the hyperspectral image feature information and radar image feature information One of them;

[0109] For splicing operations, For maximum value operation, For averaging operations, For summation operations.

[0110] Step 3: Extract spectral and spatial information from hyperspectral image features, and spatial information from radar image features, specifically including:

[0111] Step 3.1: Extract spatial information from the hyperspectral image features using the spatial feature extraction module. The expression is:

[0112] ;

[0113] ;

[0114] ;

[0115] ;

[0116] ;

[0117] ;

[0118] in: For layer normalization operation, For linear layer operations, Normalized features for hyperspectral image feature information, for Activation features, for S6 state selection feature under spatial scanning for The normalization characteristics, for The multiplication characteristic, Spatial information extracted from hyperspectral image features. The kernel size is The DW convolution operation, Activation processing for the Sigmoid activation function. For spatial scanning operations;

[0119] Specifically, the S6 state selection operation is as follows: obtaining long-distance features from the flattened vector after the spatial scanning operation, then calculating the mean of the long-distance features and restoring the size of the cube.

[0120] Among them: based on , by obtaining , using the following formula:

[0121] ;

[0122] ;

[0123] wherein, characteristic size is , , respectively the horizontal and vertical dimension size of the feature , C is the number of spectral channels; is the number of pixels along the spectral channel,

[0124] is the left-to-right flattening operation, is the top-to-bottom flattening operation, is the right-to-left flattening operation, is the bottom-to-top flattening operation;

[0125] is the vector obtained after the left-to-right flattening operation in the spatial dimension, is the vector obtained after the top-to-bottom flattening operation in the spatial dimension, is the vector obtained after the right-to-left flattening operation in the spatial dimension, is the vector obtained after the bottom-to-top flattening operation in the spatial dimension, is the state selection operation on the vector generated in the jth dimension.

[0126] Step 3.2: Extract the spatial information in the radar image feature information through the spatial feature extraction module, and the expression is:

[0127] ;

[0128] ;

[0129] ;

[0130] ;

[0131] ;

[0132] ;

[0133] wherein: is the layer normalization operation, is the linear layer operation, a normalization feature of the radar image feature information, an activation feature of , a normalization feature of ,a multiplication feature of , a normalization feature of , a multiplication feature of , a multiplication feature of , space information extracted from the radar image feature information; wherein, based on

[0134] , by obtaining , the following formula is used:

[0135] ;

[0136] ;

[0137] wherein, the feature size is , , respectively, the horizontal and vertical dimension sizes of the feature , and C is the number of spectral channels; is the number of pixels along the spectral channel,

[0138] is a left-to-right flattening operation, is a top-to-bottom flattening operation, is a right-to-left flattening operation, is a bottom-to-top flattening operation;

[0139] is a vector obtained after a left-to-right flattening operation in the spatial dimension, is a vector obtained after a top-to-bottom flattening operation in the spatial dimension, is a vector obtained after a right-to-left flattening operation in the spatial dimension, is a vector obtained after a bottom-to-top flattening operation in the spatial dimension, is a state selection operation on the vector generated in the jth dimension.

[0140] Step 3.3: Extract the spectral information in the hyperspectral feature information through the spectral feature extraction module, and the expression is:

[0141] ;

[0142] ;

[0143] ;

[0144] ;

[0145] ;

[0146] ;

[0147] in: For layer normalization operation, For linear layer operations, Normalized features for hyperspectral image feature information, for Activation features, for S6 state selection characteristics under spectral scanning for The normalization characteristics, for The multiplication characteristic, Spectral information extracted from hyperspectral image features. This is a spectral scanning operation;

[0148] Among them: based on ,pass get Use the following formula:

[0149] ;

[0150] ;

[0151] in, express The number of pixels that expand horizontally along the spatial channel.

[0152] To flatten the surface from left to right, To flatten the surface from top to bottom, To flatten the surface from right to left, To flatten the surface from bottom to top;

[0153] This is the vector obtained after flattening from left to right along the spectral dimension. This is the vector obtained after flattening from top to bottom along the spectral dimension. This is the vector obtained after flattening from right to left along the spectral dimension. This is the vector obtained after flattening from bottom to top along the spectral dimension. This is for performing a state selection operation on the vector generated in the m-th dimension.

[0154] Step 4: performing a splicing operation according to the spectral information in the hyperspectral image feature information and the spatial information feature in the radar image feature information to obtain first spatial-spectral features, specifically including:

[0155]

[0156]

[0157] wherein: is the first spatial-spectral feature, is a first spatial-spectral information splicing feature, is the first spatial-spectral feature, is a spatial scanning Mamba operation; is a pooling operation.

[0158] Step 5: performing a splicing operation according to the spatial information in the hyperspectral image feature information and the spatial information in the radar image feature information to obtain second spatial features, specifically including:

[0159] The splicing operation according to the spatial information in the hyperspectral image feature information and the spatial information in the radar image feature information to obtain the second spatial features has an expression as follows:

[0160] ;

[0161] ;

[0162] wherein: is the second spatial feature, is a second spatial information splicing feature, is a spatial scanning Mamba operation; is a pooling operation.

[0163] Step 6: performing a splicing operation on the first spatial-spectral features and the second spatial features to obtain a fusion feature of the hyperspectral image, specifically including:

[0164] ;

[0165] wherein: is the fusion feature;

[0166] Step 7: performing a self-attention feature extraction operation on the fusion feature, and calculating a classification probability for the extracted self-attention feature through a Softmax function to finally determine a classification result, specifically including:

[0167] ;

[0168] wherein: is a probability distribution predicted by the model, For self-attention feature extraction operation, This is a Tensor product operation. for The convolution operation is where c is the number of classification categories and k is the number of pixels for the fused features. Let argmax be the final classification category of pixel k. A function to find the maximum value.

[0169] In this embodiment, the Houston 2013 hyperspectral image and radar images of the same area are used. The Houston 2013 hyperspectral image is as follows: Figure 3 As shown (using the 64th, 32nd, and 22nd bands as the RGB primary color values ​​respectively), the radar image is as follows: Figure 4 As shown, the spatial resolution of the hyperspectral and radar data is 2.5m, containing a total of 349×1905 pixels. The hyperspectral dataset has 144 bands, ranging from 0.38nm to 1.05nm. The dataset contains 15,029 labeled ground samples. The Houston 2013 hyperspectral image includes 15 land cover categories such as grassland, trees, and roads, covering the main land cover types in the urban environment. The existing CALC, HCT, ExViT, FDNet, and the hyperspectral fusion method of this application were used to perform multi-source fusion classification verification on the Houston 2013 hyperspectral image.

[0170] The results are shown in Table 1 below:

[0171] Table 1 Comparison of Accuracy of Hyperspectral and Radar Image Fusion

[0172] Multi-source fusion classification method CALC HCT ExViT FDNet The present invention Overall accuracy / % 93.52 93.64 92.95 93.02 94.94 Average accuracy / % 94.45 94.44 94.01 93.93 95.58 Kappa coefficient x 100 93.00 93.12 92.38 92.46 94.53

[0173] As shown in Table 1, the classification accuracy of the hyperspectral and radar image fusion method in this application reaches 94.94%, which is significantly better than other existing methods. The second-highest scoring method, HCT, is 1.30% higher.

[0174] The results of the hyperspectral fusion classification method, CALC, HCT, ExViT, and FDNet in this application are shown in the following figures: Figure 5 , Figure 6 , Figure 7 , Figure 8 , Figure 9 As can be seen from the classification results, the classification effect of hyperspectral and radar image fusion in this application has a clearer outline at the edges.

[0175] The above demonstrates that this application can effectively improve the accuracy of multi-source fusion and is feasible in hyperspectral multi-source fusion.

[0176] This invention first uses PCA to reduce the dimensionality of hyperspectral images, then performs multi-scale convolution on both hyperspectral and radar images, combining the maximum, average, and summation features from the multi-scale images for feature fusion to obtain multi-scale image information. In the hyperspectral image, a four-directional spectral scan is used to acquire spectral information; simultaneously, a four-directional spatial scan is used on both the hyperspectral and radar images to acquire spatial information. The spectral information from the hyperspectral image and the spatial information from the radar data are then fused to obtain the first spatial-spectral feature, and the spatial information from the hyperspectral image and the radar data is fused to obtain the second spatial feature. Finally, the first spatial-spectral feature and the second spatial feature are fused using tensor concatenation, and the classification probability is obtained through a normalized exponential function to determine the final classification result. Therefore, this invention's method, by designing different scanning mechanisms, acquires sufficient spectral and spatial information, effectively combining spatial and spectral information during feature fusion, thus improving the accuracy of multi-source remote sensing image fusion classification.

[0177] This invention achieves efficient fusion and accurate classification of hyperspectral and radar images through a series of technological innovations, including multi-scale convolution, four-way scanning, dual-path fusion, and self-attention mechanism. It demonstrates significant advantages in accuracy, robustness, and visualization effects, and is applicable to multiple application scenarios such as urban remote sensing, environmental monitoring, and agricultural assessment.

[0178] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0179] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0180] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0181] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0182] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A multi-source remote sensing image fusion and classification method, characterized in that, include: Acquire hyperspectral and radar images of the area to be measured; Acquire hyperspectral image feature information and radar image feature information under multi-scale convolution conditions; Extract spectral and spatial information from hyperspectral image features, and spatial information from radar image features; The first spatial-spectral feature is obtained by stitching together the spectral information in the hyperspectral image feature information and the spatial information feature information in the radar image feature information. A second spatial feature is obtained by stitching together the spatial information in the hyperspectral image feature information and the spatial information in the radar image feature information. The first spatial-spectral feature and the second spatial feature are stitched together to obtain the fused features of the hyperspectral image; The fused features are subjected to self-attention feature extraction, and the classification probability of the extracted self-attention features is calculated using the Softmax function to finally determine the classification result.

2. The multi-source remote sensing image fusion and classification method according to claim 1, characterized in that, The acquisition of hyperspectral image feature information and radar image feature information under multi-scale convolution conditions specifically includes: Multi-scale convolution operations are performed on hyperspectral and radar images, followed by normalization and activation operations to obtain hyperspectral features under multi-scale convolution conditions. and radar characteristics The expression is: ； in, Hyperspectral features at kernel scale s and radar characteristics One of them, the kernel size is Pick: , and ; For hyperspectral images and radar images One of them, The kernel size is Convolution operation, For batch normalization operations, For activation; Hyperspectral features under multi-scale convolution conditions and radar characteristics Features of the same dimension are concatenated, expressed as: ； in, It is one of the i-th dimension hyperspectral image stitching features and the i-th dimension radar image stitching features; The scale of the i-th dimension convolution kernel is One of the hyperspectral features and radar features under convolution operations. The scale of the i-th dimension convolution kernel is One of the hyperspectral features and radar features under convolution operations. The scale of the i-th dimension convolution kernel is One of the hyperspectral features and radar features under convolution operations; Extract the maximum, average, and summed features from the i-th dimension hyperspectral image stitching features and the i-th dimension radar image stitching features respectively, perform a stitching operation, and obtain hyperspectral image feature information under multi-scale convolution conditions. and radar image feature information The specific expression is: ； in, For hyperspectral image feature information and radar image feature information One of them; For splicing operations, For maximum value operation, For averaging operations, For summation operations.

3. The multi-source remote sensing image fusion method according to claim 2, characterized in that, The method for extracting spatial information from radar image feature information is expressed as follows: ； ； ； ； ； ； in: For layer normalization operation, For linear layer operations, These are normalized features of radar image feature information. for Activation features, for S6 state selection feature under spatial scanning for The normalization characteristics, for The multiplication characteristic, Spatial information extracted from radar image feature information; The kernel size is The DW convolution operation, Activation processing for the Sigmoid activation function. For spatial scanning operations; Specifically, the S6 state selection operation involves obtaining long-distance features from the flattened vector after the spatial scanning operation, then calculating the mean of the long-distance features and restoring the size of the cube.

4. The multi-source remote sensing image fusion and classification method according to claim 3, characterized in that, The method for extracting spatial information from hyperspectral image features is expressed as follows: ； ； ； ； ； ； in: For layer normalization operation, For linear layer operations, Normalized features for hyperspectral image feature information, for Activation features, for S6 state selection feature under spatial scanning for The normalization characteristics, for The multiplication characteristic, Spatial information extracted from hyperspectral image feature information.

5. The multi-source remote sensing image fusion and classification method according to claim 3, characterized in that, The method for extracting spectral information from the hyperspectral feature information is expressed as follows: ； ； ； ； ； ； in: For layer normalization operation, For linear layer operations, Normalized features for hyperspectral image feature information, for Activation features, for S6 state selection characteristics under spectral scanning for The normalization characteristics, for The multiplication characteristic, Spectral information extracted from hyperspectral image features. This is a spectral scanning operation.

6. The multi-source remote sensing image fusion and classification method according to claim 3, characterized in that, based on ,pass get Use the following formula: ； ； in, Feature size is , , Features The horizontal and vertical dimensions, where C is the number of spectral channels; for The number of pixels spread along the spectral channels. To flatten the surface from left to right, To flatten the surface from top to bottom, To flatten the surface from right to left, To flatten the surface from bottom to top; This is the vector obtained after flattening from left to right in the spatial dimension. This is the vector obtained after flattening from top to bottom in the spatial dimension. This is the vector obtained after flattening from right to left in the spatial dimension. This is the vector obtained after flattening from bottom to top in the spatial dimension. This is to perform a state selection operation on the vector generated in the j-th dimension.

7. The multi-source remote sensing image fusion and classification method according to claim 4, characterized in that, based on ,pass get Use the following formula: ； ； in, Feature size is , , Features The horizontal and vertical dimensions, where C is the number of spectral channels; for The number of pixels spread along the spectral channels. To flatten the surface from left to right, To flatten the surface from top to bottom, To flatten the surface from right to left, To flatten the surface from bottom to top; This is the vector obtained after flattening from left to right in the spatial dimension. This is the vector obtained after flattening from top to bottom in the spatial dimension. This is the vector obtained after flattening from right to left in the spatial dimension. This is the vector obtained after flattening from bottom to top in the spatial dimension. This is to perform a state selection operation on the vector generated in the j-th dimension.

8. The multi-source remote sensing image fusion and classification method according to claim 5, characterized in that, based on ,pass get Use the following formula: ； ； in, express The number of pixels that expand horizontally along the spatial channel. To flatten the surface from left to right, To flatten the surface from top to bottom, To flatten the surface from right to left, To flatten the surface from bottom to top; This is the vector obtained after flattening from left to right along the spectral dimension. This is the vector obtained after flattening from top to bottom along the spectral dimension. This is the vector obtained after flattening from right to left along the spectral dimension. This is the vector obtained after flattening from bottom to top along the spectral dimension. This is for performing a state selection operation on the vector generated in the m-th dimension.

9. The multi-source remote sensing image fusion and classification method according to claim 5, characterized in that, The first spatial-spectral feature is obtained by concatenating the spectral information from the hyperspectral image feature information and the spatial information from the radar image feature information, as expressed in: ； ； in: For the first spatial-spectral characteristics, This is the splicing feature of the first spatial spectrum information. For the first spatial-spectral characteristics, Mamba operations for spatial scanning; This is a pooling operation.

10. The multi-source remote sensing image fusion and classification method according to claim 4, characterized in that, The second spatial feature is obtained by stitching together the spatial information from the hyperspectral image features and the spatial information from the radar image features. The expression is as follows: ； ； in: As a second spatial feature, This is a feature for splicing second spatial information. Mamba operations for spatial scanning; This is a pooling operation.

Images