A remote sensing ground feature classification method based on dual-heterogeneous network for on-site security

Abstract

The embodiment of the application relates to the technical field of temporary security protection, and discloses a remote sensing ground feature classification method based on a dual-heterogeneous network for temporary security protection, which comprises the following steps: dimension reduction is performed on a hyperspectral image, sampling is performed around a pixel point to be classified to obtain an input image block; the input image block is input into a pre-trained classification model, spatial specificity features are obtained by using a spatial specificity branch of the classification model, and spectral specificity features are obtained by using a spectral specificity branch; the spectral specificity features are calibrated by using a spatial calibration module of the classification model, and the spatial specificity features are calibrated by using a spectral calibration module; according to the calibrated spatial specificity features, the calibrated spectral specificity features and a central spectral vector of the input image block, joint space-spectrum features are obtained, and based on the joint space-spectrum features, ground feature classification is performed by using a classifier to obtain a ground feature category corresponding to the pixel point to be classified, so that the accuracy of the ground feature classification is effectively improved.

Description

Technical Field

[0001] This application relates to the field of local security technology, and in particular to a remote sensing ground feature classification method based on dual heterogeneous networks for local security. Background Technology

[0002] Temporary security refers to a diversified, cross-domain, three-dimensional, collaborative, and intelligent technical system addressing the needs of defense, protection, production, safety, and rescue within space. Specific application scenarios include low-altitude security, underwater security, and cross-domain security. The temporary security technology system encompasses six major research directions: related light detection, stability detection, multimodal cognitive computing, swarm intelligence decision-making, waterborne optics, and cross-domain remote sensing. Obtaining accurate information on ground cover categories is crucial for the effective implementation of typical temporary security tasks such as low-altitude UAV detection, extreme disaster early warning and rescue, and target reconnaissance. Hyperspectral images contain fine-grained spectral information and rich spatial information, serving as an important information carrier in the cross-domain remote sensing research direction within the temporary security technology system. Analyzing hyperspectral remote sensing images to classify ground cover is key to the successful implementation of temporary security applications; therefore, ground cover classification in hyperspectral images has received widespread attention.

[0003] Due to the complexity and variability of ground features, their irregular regional distribution, numerous categories, and imbalanced sample sizes, coupled with the fact that the acquisition of remote sensing hyperspectral imagery is affected by meteorological conditions, sensor type, and shooting angle, and is accompanied by issues of "different spectra for the same object" and "different objects for the same spectrum," accurately identifying the categories of ground features contained in hyperspectral images is a challenging task. Therefore, constructing a hyperspectral ground feature classification method for local security tasks is a very important research area.

[0004] In hyperspectral images used for near-site security, the distribution of ground features is varied and irregular. This means that sampling based on neighboring image patches may include neighboring pixels of a different class than the central pixel, exhibiting spatial specificity. Most current methods use convolutional neural networks to model the features of hyperspectral images. However, the reusable convolutional kernel parameters across the spatial range are insufficient to fully characterize the spatial specificity of hyperspectral images. This results in classification models with inadequate discriminative power for hyperspectral image features, making accurate ground feature classification difficult for near-site security tasks. Summary of the Invention

[0005] The purpose of this application is to provide a remote sensing land cover classification method based on dual heterogeneous networks for local security, which can effectively characterize the spatial specificity of hyperspectral images, improve the ability to distinguish captured features, and improve the accuracy of land cover classification of hyperspectral images.

[0006] To address the aforementioned technical problems, embodiments of this application provide a remote sensing land cover classification method based on a dual heterogeneous network for near-site security, comprising the following steps: dimensionality reduction of a hyperspectral image using principal component analysis; sampling within the dimensionality-reduced hyperspectral image centered on the pixel to be classified to obtain an input image patch; inputting the input image patch into a pre-trained classification model; obtaining spatial-specific features using the spatial-specific branch of the classification model, and obtaining spectral-specific features using the spectral-specific branch of the classification model; wherein the spatial-specific branch extracts spatial-specific features based on a spatial-spectral joint involution operation, and the spatial-spectral joint involution operation is used to fuse central spectral features and neighborhood mean features; calibrating the spectral-specific features using the spatial calibration module of the classification model, and calibrating the spatial-specific features using the spectral calibration module of the classification model; obtaining spatial-spectral joint features based on the calibrated spatial-specific features, the calibrated spectral-specific features, and the central spectral vector of the input image patch; and classifying land cover using the classifier of the classification model based on the spatial-spectral joint features to obtain the land cover category corresponding to the pixel to be classified.

[0007] Embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned remote sensing ground feature classification method based on dual heterogeneous networks for local security.

[0008] This application provides a remote sensing ground cover classification method based on a dual heterogeneous network for near-site security. The classification model comprises a spatially specific branch and a spectrally specific branch forming a dual heterogeneous network. The spatially specific branch fuses central spectral features and neighborhood mean features based on a spatial-spectral joint involution operation. An involution kernel is generated based on the central spectral information of the receptive field and neighborhood spatial information. The kernel parameters of this involution kernel are not reused at different spatial locations, effectively characterizing the spatial specificity of the hyperspectral image. The inclusion of spatial and spectral calibration modules significantly enhances the information interaction between the two branches, thereby strengthening the representational ability of the classification model. The calibrated spatially specific features, calibrated spectral specific features, and the central spectral vector of the input image patch are fused to obtain spatially and spectrally complementary joint features. The introduction of the central spectral vector suppresses the influence of interfering pixels in the neighborhood that are different from the central pixel category, improving the discriminative ability of the captured features. Based on this classification model, the impact of climate conditions, sensor type and shooting angle, uneven distribution of ground objects in hyperspectral images, and problems such as "different objects with the same spectrum" and "different spectra with the same object" on the accuracy of ground object classification in hyperspectral images can be overcome. This significantly improves the accuracy of ground object classification in hyperspectral images and provides effective and accurate data support for typical on-site security tasks that require precise ground object category information, such as low-altitude UAV security, disaster early warning and rescue, and target reconnaissance.

[0009] In some optional embodiments, for any spatial-spectral joint involution kernel, the fusion center spectral feature and neighborhood mean feature are achieved by the following formula:

[0010] I i，j =W1σ[W0(F i，j +Avg i，j )]

[0011] Among them, F i，j Avg represents the central spectral feature of the input pixel (i, j) of the spatial-spectral joint involution operation. i，j This represents the mean feature of the neighborhood of pixel (i, j) with a receptive field size of K×K, where W0 and W1 represent convolution operations, σ(·) represents BN normalization and ReLU function, and I i，j This represents the generated spatial spectrum joint involution kernel.

[0012] In some optional embodiments, the spatially specific branch consists of three sequentially connected multi-scale involute blocks, with a spatial attention module positioned between adjacent multi-scale involute blocks. Each multi-scale involute block contains two connected 3×3 spatial-spectral joint involute kernels and one 5×5 spatial-spectral joint involute kernel. The step of inputting the input image block into a pre-trained classification model and obtaining spatially specific features using the spatially specific branch of the classification model includes: extracting preliminary features of the input image block using the pre-convolutional layer of the classification model, inputting the preliminary features into the first multi-scale involute block, and so on until the last multi-scale involute block outputs the spatially specific features. The output of any multi-scale involute block l is expressed by the formula:

[0013] F l =ReLU{F l-1 +σ[I5(F l-1 )]+σ{I3<σ[I3(F l-1 )]>}}

[0014] Among them, F l-1 For the input of the l-th multi-scale involute block, F l For the output of the l-th multi-scale involute block, l∈[1,3], F 0 For the aforementioned preliminary features, I3(·) represents the 3×3 spatial-spectral joint involution kernel, I3(·) represents the 5×5 spatial-spectral joint involution kernel ReLU(·), and RelU function represents the spatial-specific branch. The spatial-specific branch consists of multiple multi-scale involution blocks, with a spatial attention module positioned between adjacent multi-scale involution blocks. The introduction of the spatial attention module enhances useful spatial information and suppresses redundant spatial information. Each multi-scale involution block contains spatial-spectral joint involution kernels with different receptive fields. Performing spatial-spectral joint involution operations in different receptive fields better extracts spatial-specific features. The introduction of the ReLU function and BN normalization significantly improves the training performance of the classification model.

[0015] In some optional embodiments, the spectral specific branch consists of three sequentially connected multi-scale convolutional blocks, with a channel attention module positioned between adjacent multi-scale convolutional blocks. Each multi-scale convolutional block contains two connected 3×3 convolutional kernels and one 5×5 convolutional kernel. Obtaining spectral specific features using the spectral specific branch of the classification model includes: inputting the preliminary features into the first multi-scale convolutional block until the last multi-scale convolutional block outputs the spectral specific features. The output of any multi-scale convolutional block g is expressed by the formula:

[0016] F g =ReLU{F g-1 +σ[C5(Fg-1 )]+σ{C3<σ[C3(F g-1 )]>}}

[0017] Among them, F g-1 For the input of the g-th multi-scale convolutional block, F g For the output of the g-th multi-scale convolutional block, g∈[1,3], F 0 For the aforementioned preliminary features, C3(·) represents the 3×3 convolution kernel, and C5(·) represents the 5×5 convolution kernel. The spectral specificity branch consists of multiple conventional multi-scale convolution blocks, with a channel attention module positioned between adjacent multi-scale convolution blocks. The introduction of the channel attention module can assign different weights to different channels, thereby enhancing spectral specificity.

[0018] In some optional embodiments, the calibration of the spectral-specific features using the spatial calibration module of the classification model includes: performing average pooling and max pooling operations on the spatial-specific features to obtain a first intermediate feature and a second intermediate feature, and concatenating the first intermediate feature and the second intermediate feature to obtain a first tensor; inputting the first tensor into a 3×3 convolutional layer of the spatial calibration module for learning; obtaining spatial weights from the output of the 3×3 convolutional layer of the spatial calibration module using the Sigmoid function, and calibrating the spectral-specific features according to the spatial weights to obtain calibrated spectral-specific features. The spatial calibration module generates spatial weights using the spatial-specific features output from the spatial-specific branch, and uses the spatial weights to perform feature calibration on the spectral-specific features output from the spectral-specific branch in the spatial dimension, resulting in preliminary spatial complementarity of the calibrated spectral-specific features.

[0019] In some optional embodiments, the spatial weights are obtained by passing the output of the 3×3 convolutional layer of the spatial calibration module through the Sigmoid function using the following formula, and the spectral specific features are calibrated according to the spatial weights to obtain the calibrated spectral specific features:

[0020] F′ spe =S(C spa ([AvgPool(F spa MaxPool(F) spa )]))×F spe

[0021] Among them, F spe For the spectral specificity feature, F spa For the space-specific feature, AvgPool(·) represents the average pooling operation, MaxPool(·) represents the max pooling operation, and C spa(·) represents the 3×3 convolutional layer of the spatial calibration module, S(·) represents the Sigmoid function, and F′ spe The calibrated spectral specificity feature is represented by [·], where [·] indicates a connection operation.

[0022] In some optional embodiments, the spectral calibration module using the classification model calibrates the spatially specific features by: performing average pooling and max pooling operations on the spectrally specific features to obtain third and fourth intermediate features, and inputting the third and fourth intermediate features into a multilayer perceptron composed of two fully connected layers for learning; summing the outputs of the multilayer perceptron and obtaining spectral weights through the Sigmoid function; and calibrating the spatially specific features according to the spectral weights to obtain calibrated spatially specific features. The spectral calibration module generates spectral weights using the spectrally specific features output from the spectrally specific branch, and uses the spectral weights to perform feature calibration on the spatially specific features output from the spatially specific branch in the spectral dimension. The calibrated spatially specific features obtain preliminary spectral complementarity.

[0023] In some optional embodiments, the output of the 3×3 convolutional layer of the spectral calibration module is passed through the Sigmoid function to obtain spectral weights using the following formula. The spatial specific features are then calibrated based on these spectral weights to obtain the calibrated spatial specific features:

[0024] F′ spa =S{M <AvgPool(F spe ))+M <MaxPool(F spe )>}×F spa

[0025] Where M(·) represents the multilayer perceptron, F′ spa This refers to the calibrated spatial specific characteristics.

[0026] In some optional embodiments, obtaining the spatial-spectral joint feature based on the calibrated spatial-specific feature, the calibrated spectral-specific feature, and the center spectral vector of the input image patch includes: adding the calibrated spatial-specific feature and the calibrated spectral-specific feature pixel by pixel, and concatenating the pixel-by-pixel addition result with the center spectral vector of the input image patch to obtain the spatial-spectral joint feature, which is expressed by the formula:

[0027] F spe-spa =[(F′ spe +F′ spa );F center ]

[0028] Among them, F′ spa For the calibrated space-specific feature, F′ spe For the calibrated spectral specificity characteristics, F center F is the center spectral vector of the input image patch. spe-spa The spatial-spectral joint feature is denoted by [·], where [·] represents the concatenation operation. The fusion of spatially specific and spectrally specific features enhances the discriminative power of the features. Considering that the sampled input image patch may contain neighboring interfering pixels of a different category than the central pixel, concatenating the central spectral vector after the fused spatially and spectrally specific features effectively suppresses interference. The resulting spatial-spectral joint feature is highly expressive, less susceptible to interference, and well-suited for ground cover classification. Attached Figure Description

[0029] One or more embodiments are illustrated by way of example with reference to the accompanying drawings, and these illustrative descriptions do not constitute a limitation on the embodiments.

[0030] Figure 1 This is a flowchart of a remote sensing ground feature classification method based on a dual heterogeneous network for near-site security, provided in one embodiment of this application;

[0031] Figure 2 This is a schematic diagram illustrating the principle of the spatial-spectral joint involution operation provided in one embodiment of this application;

[0032] Figure 3 This is a schematic diagram of a DHNet network structure provided in one embodiment of this application;

[0033] Figure 4 This is a schematic diagram of an architecture for a multi-scale involute block provided in one embodiment of this application;

[0034] Figure 5 This is a schematic diagram of the architecture of a multi-scale convolutional block provided in one embodiment of this application;

[0035] Figure 6 This is a schematic diagram provided in one embodiment of the present application, illustrating the calibration of spectral specific features using a spatial calibration module of a classification model;

[0036] Figure 7 This is a schematic diagram provided by an embodiment of the present application, illustrating the calibration of spatially specific features using the spectral calibration module of a classification model;

[0037] Figure 8 This is a schematic diagram illustrating the impact of different neighborhood image patch sizes and preserved spectral dimensions on the classification results of DHNet on four datasets, provided by one embodiment of this application.

[0038] Figure 9 This is a schematic diagram illustrating the impact of different training sample rates on the classification results of DHNet on four datasets, provided by one embodiment of this application. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been provided in the various embodiments of this application to help readers better understand this application. However, the technical solutions claimed in this application can be implemented even without these technical details and various changes and modifications based on the following embodiments. The division of the various embodiments below is for the convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0040] One embodiment of this application relates to a remote sensing ground feature classification method based on a dual heterogeneous network for near-site security. The implementation details of this embodiment of the remote sensing ground feature classification method based on a dual heterogeneous network for near-site security are described in detail below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0041] The specific process of the remote sensing ground cover classification method based on dual heterogeneous networks for local security proposed in this embodiment can be described as follows: Figure 1 As shown, it includes:

[0042] Step 101: Dimensionality reduction of the hyperspectral image is performed using principal component analysis. Sampling is performed in the dimension-reduced hyperspectral image with the pixel to be classified as the center to obtain the input image block.

[0043] In practical implementation, to reduce the computation time of feature extraction, the acquired hyperspectral image cannot be directly input into the classification model for recognition. Instead, it needs to be dimensionality-reduced using Principal Component Analysis (PCA). For a given hyperspectral image... r×c represents the spatial dimensions of the hyperspectral image X, and b represents the spectral dimension of the hyperspectral image X. After PCA processing, only the first p principal components of the hyperspectral image X are retained. The dimension-reduced hyperspectral image can be represented as follows: p represents X p The spectral dimension is then determined. At this point, the pixel to be classified can be identified (e.g., it is assumed to be selected), and the pixel to be classified is used as the center in the dimensionality-reduced hyperspectral image X. pNeighborhood sampling is performed to obtain input image patches, which will be fed into the classification model and can be represented as follows: s×s is the window size for neighborhood sampling, and its size can be set by those skilled in the art according to actual needs.

[0044] Step 102: Input the input image patch into the pre-trained classification model, obtain spatial specific features using the spatial specific branch of the classification model, and obtain spectral specific features using the spectral specific branch of the classification model. The spatial specific branch extracts spatial specific features based on the spatial-spectral joint involution kernel. The spatial-spectral joint involution kernel is used to fuse the central spectral features and neighborhood mean features of the input spatial-spectral joint involution kernel.

[0045] In the specific implementation, after sampling the input image patch, the input image patch is fed into a pre-trained classification model (also known as DHNet). The classification model has two branches: a spatially specific branch and a spectrally specific branch. Based on the input image patch, the classification model uses the spatially specific branch to obtain spatially specific features, and uses the spectrally specific branch to obtain spectrally specific features. The spatially specific branch extracts spatially specific features based on a joint spatial-spectral involution operation, which is used to fuse the central spectral features and the neighborhood mean features.

[0046] It is worth noting that the spatial-spectral joint involution operation is designed to fully characterize the spatial specificity of hyperspectral images. During the generation of the involution kernel, not only the central spectrum of the receptive field is considered, but also neighborhood spatial information, ensuring that the involution kernel parameters are not reused at different spatial locations. For any spatial-spectral joint involution operation, assuming its input features are... Where H×W is the spatial dimension of the feature map, and C is the number of channels in the feature map, the spatial-spectral joint involution operation fuses the central spectral features and the neighborhood mean features, which can be achieved by the following formula:

[0047] I i,j =W1σ[W0(F i，j +Avg i,j )]

[0048] In the formula, F i，j Avg represents the central spectral feature of pixel (i, j) in F. i,j This represents the mean feature of the neighborhood of pixel (i, j) with a receptive field size of K×K, where W0 and W1 represent convolution operations, σ(·) represents BN normalization and ReLU function, and I i，j This represents the generated joint involution kernel of the spatial spectrum. After the joint involution operation of the spatial spectrum, F and I need to be summed to obtain the output feature map Y. The principle of the spatial-spectral joint involution operation can be described as follows: Figure 2 As shown.

[0049] In one example, the DHNet network structure is as follows: Figure 3 As shown, DHNet mainly consists of a pre-convolutional layer, a spatially specific branch, a spectrally specific branch, a spatial calibration module, a spectral calibration module, and a classifier. The spatially specific branch consists of three sequentially connected multi-scale inconvolutional blocks, with a spatial attention module (SAM) placed between adjacent multi-scale inconvolutional blocks. The architecture of the multi-scale inconvolutional blocks can be as follows: Figure 4 As shown, each multi-scale inconvolution block contains two contiguous 3×3 spatial-spectral joint inconvolution kernels and one 5×5 spatial-spectral joint inconvolution kernel. The multi-scale inconvolution block also embeds a ReLU function and BN normalization. The input image block is fed into the classification model, enters the pre-convolutional layer for feature extraction, and obtains preliminary features. These preliminary features are then fed into the first multi-scale inconvolution block of the spatial specificity branch, until the last multi-scale inconvolution block outputs the spatial specific features. The output of any multi-scale inconvolution block l can be expressed by the formula:

[0050] F l =ReLU{F l-1 +σ[I5(F l-1 )]+σ{I3<σ[I3(F l-1 )]>}}

[0051] In the formula, F l-1 For the input of the l-th multi-scale involute block, F l For the output of the l-th multi-scale involute block, l∈[1,3], F 0 For the initial features extracted by the pre-convolutional layer, I3(·) represents a 3×3 spatial-spectral joint involution kernel, I3(·) represents a 5×5 spatial-spectral joint involution kernel, and ReLU(·) represents the RelU function. The spatial-specific branch consists of multiple multi-scale involution blocks, with a spatial attention module placed between adjacent multi-scale involution blocks. The introduction of the spatial attention module can enhance useful spatial information and suppress redundant spatial information. Each multi-scale involution block contains a spatial-spectral joint involution kernel with a different receptive field. Performing spatial-spectral joint involution operations in different receptive fields can better extract spatial-specific features. The introduction of the ReLU function and BN normalization significantly improves the training performance of the classification model.

[0052] In one example, different spectral bands contain inconsistent information. To capture complementary spatial-spectral joint features, it is necessary to construct spectral-specific branches to extract spectral-specific features. For example... Figure 3As shown, the spectral specificity branch consists of three sequentially connected multi-scale convolutional blocks, with a channel attention module (CAM) positioned between adjacent multi-scale convolutional blocks. The architecture of the multi-scale convolutional blocks can be as follows: Figure 5 As shown, each multi-scale convolutional block contains two connected 3×3 convolutional kernels and one 5×5 convolutional kernel. The multi-scale convolutional block also embeds the ReLU function and BN normalization. The input image block is fed into the classification model, enters the pre-convolutional layer for feature extraction, and obtains preliminary features of the input image block. These preliminary features then enter the first multi-scale convolutional block of the spectral-specific branch, until the last multi-scale convolutional block outputs the spectral-specific features. The output of any multi-scale convolutional block g can be expressed by the formula:

[0053] F g =ReLU{F g-1 +σ[C5(F g-1 )]+σ{C3<σ[C3(F g-1 )]>}}

[0054] In the formula, F g For the input of the g-th multi-scale convolutional block, F g+1 For the output of the g-th multi-scale convolutional block, g∈[1,3], F 0 The initial features extracted from the preceding convolutional layer are represented by C3(·), which indicates a 3×3 convolutional kernel, and C5(·), which indicates a 5×5 convolutional kernel. The spectral-specific branch consists of multiple conventional multi-scale convolutional blocks, with channel attention modules placed between adjacent multi-scale convolutional blocks. The introduction of channel attention modules can assign different weights to different channels, thereby enhancing spectral specificity.

[0055] In one example, when training the classification model DHNet, the loss function used includes a cross-entropy loss term and a center loss term.

[0056] Step 103: Use the spatial calibration module of the classification model to calibrate the spectral specific features, and use the spectral calibration module of the classification model to calibrate the spatial specific features.

[0057] In its implementation, the spatial calibration module uses the feature map output by the spatially specific branch to generate spatial weights, and performs feature calibration on the feature map output by the spectrally specific branch in the spatial dimension. The spatially specific features output by the spatially specific branch can be represented as... The spectral specificity feature of the spectral specificity branch output can be expressed as F spe , H×W represents the spatial dimensions, and C represents the number of channels. The spatial calibration module of the classification model can be used to calibrate spectral specific features as follows: Figure 6As shown, the spatial calibration module performs average pooling and max pooling operations on the spatially specific features output from the spatially specific branch to achieve channel dimension compression, obtaining a first intermediate feature and a second intermediate feature of size H×W×1. The first and second intermediate features are then concatenated to form a first tensor of size H×W×2. This first tensor is then fed into a 3×3 convolutional layer of the spatial calibration module for learning. Finally, the output of the 3×3 convolutional layer of the spatial calibration module is processed by the Sigmoid function to obtain spatial weights. These spatial weights are then used to calibrate the spectrally specific features, resulting in the calibrated spectrally specific feature F′. spe The spatial calibration module generates spatial weights using the spatially specific features output by the spatially specific branch. These spatial weights are then used to perform feature calibration on the spectrally specific features output by the spectrally specific branch in the spatial dimension. The calibrated spectrally specific features achieve preliminary spatial complementarity.

[0058] In one example, the output of the 3×3 convolutional layer of the spatial calibration module is used to obtain spatial weights through the Sigmoid function. The spectral specific features are then calibrated based on these spatial weights to obtain the calibrated spectral specific features F′. spe This can be achieved using the following formula:

[0059] F′ spe =S(C spa ([AvgPool(F spa MaxPool(F) spa )]))×F spe

[0060] In the formula, F spe For spectral specificity, F spa To highlight the space-specific characteristics, AvgPool(·) represents the average pooling operation, MaxPool(·) represents the max pooling operation, and C spa (·) represents a 3×3 convolutional layer of the spatial calibration module, S(·) represents the Sigmoid function, and F′ spe For the calibrated spectral specificity features, [·] indicates a connection operation.

[0061] In its implementation, the spectral calibration module generates spectral weights using the spectral-specific features output from the spectral-specific branch, and then calibrates the spatial-specific features output from the spatial-specific branch along the spectral dimension. The spectral calibration module of the classification model can be used to calibrate spatial-specific features as follows: Figure 7As shown, the spectral calibration module performs average pooling and max pooling operations on the spectral-specific features output from the spectral-specific branch to obtain third and fourth intermediate features of size H×W×1. These third and fourth intermediate features are then concatenated to form a second tensor of size H×W×2. This second tensor is then fed into a 3×3 convolutional layer of the spectral calibration module for learning. Finally, the output of the 3×3 convolutional layer is processed using the Sigmoid function to obtain spectral weights. These weights are then used to calibrate the spatial-specific features output from the spatial-specific branch, resulting in the calibrated spatial-specific feature F′. spa The spectral calibration module generates spectral weights using the spectral-specific features output from the spectral-specific branch. These spectral weights are then used to calibrate the spatial-specific features output from the spatial-specific branch along the spectral dimension. The calibrated spatial-specific features then achieve preliminary spectral complementarity.

[0062] In one example, the output of the 3×3 convolutional layer of the spectral calibration module is used to obtain spectral weights through the Sigmoid function. The spatial specific features are then calibrated based on these spectral weights to obtain the calibrated spatial specific features F′. spa This can be achieved using the following formula:

[0063] F′ spa =S{M <AvgPool(F spe )>+M <MaxPool(F spe )>}×F spa

[0064] In the formula, M(·) represents a multilayer perceptron, and F′ spa These are the calibrated space-specific features.

[0065] Step 104: Based on the calibrated spatial specific features, the calibrated spectral specific features, and the center spectral vector of the input image patch, obtain the spatial-spectral joint features, and based on the spatial-spectral joint features, use the classifier of the classification model to classify land features and obtain the land feature category corresponding to the pixel to be classified.

[0066] In the specific implementation, the classification model obtains the calibrated spatial specific features F′ spa and the calibrated spectral specificity feature F′ spe Then, the calibrated space-specific feature F′ can be... spa and the calibrated spectral specificity feature F′ spe Perform pixel-by-pixel summation, and then compare the summation result with the center spectral vector F of the input image patch. center By connecting the spatial and spectral joint features F, we obtain the joint feature F. spe-spaThe joint spatial-spectral characteristics can be expressed by the formula:

[0067] F spe-spa =[(F′ spe +F′ spa );F center ]

[0068] In the formula, F′ spa For the calibrated space-specific features, F′ spe For the calibrated spectral specificity characteristics, F center Let F be the center spectral vector of the input image patch. spe-spa The spatial-spectral joint feature is represented by [·], which indicates a concatenation operation. The fusion of spatially specific and spectrally specific features enhances the discriminative power of the features. Considering that the sampled input image patch may contain neighboring interfering pixels of a different category than the central pixel, concatenating the central spectral vector after the fused spatially and spectrally specific features effectively suppresses interference. The resulting spatial-spectral joint feature is highly expressive, less susceptible to interference, and well-suited for ground cover classification.

[0069] In this embodiment, the classification model employs a dual heterogeneous network composed of a spatially specific branch and a spectrally specific branch. The spatially specific branch fuses central spectral features and neighborhood mean features based on a joint spatial-spectral involution operation. It generates an involution kernel based on the central spectral information of the receptive field and neighborhood spatial information. The kernel parameters of this involution kernel are not reused at different spatial locations, effectively characterizing the spatial specificity of the hyperspectral image. The spatial calibration module and the spectral calibration module significantly enhance the information interaction between the two branches, thereby strengthening the representational ability of the classification model. The calibrated spatially specific features, calibrated spectral specific features, and the central spectral vector of the input image patch are fused to obtain spatially and spectrally complementary joint features. The introduction of the central spectral vector suppresses the influence of interfering pixels in the neighborhood that belong to a different category than the central pixel, improving the discriminative ability of the captured features. Based on this classification model, the impact of climate conditions, sensor type and shooting angle, uneven distribution of ground objects in hyperspectral images, and problems such as "different objects with the same spectrum" and "different spectra with the same object" on the accuracy of ground object classification in hyperspectral images can be effectively overcome. This significantly improves the accuracy of ground object classification in hyperspectral images and provides effective and accurate data support for typical on-site security tasks that require precise ground object category information, such as low-altitude UAV security, disaster early warning and rescue, and target reconnaissance.

[0070] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0071] In one embodiment, to verify the effectiveness of DHNet, classification experiments were conducted on four publicly available hyperspectral image datasets: Indian Pines, Pavia University, Salinas, and Botswana. First, spectral dimensionality reduction was performed on all four datasets using the PCA algorithm, retaining the first 16 principal components, resulting in a spectral dimension of 16. Then, neighborhood sampling was performed on the pixel to be classified, with the category of the center pixel used as the category of the image patch. On the Indian Pines, Pavia University, and Botswana datasets, the neighborhood size for image patch sampling was 15×15, while on the Salinas dataset, the neighborhood size was 19×19. Specific experiments are as follows:

[0072] (1) Hyperspectral image dataset

[0073] The Indian Pines dataset was collected by the AVIRIS airborne visible / infrared imaging spectrometer in a farmland area in northern Indiana. The dataset has dimensions of 145×145×200, with a spatial dimension of 145×145 and 200 spectral bands. It contains 16 different land cover categories, including various materials found in real-world scenes such as coniferous forests and farmland, including corn, soybeans, and pine trees. 5% of the samples from each category were randomly selected as the training set, and the remaining 95% were used as the test set.

[0074] The Pavia University dataset consists of hyperspectral urban remote sensing images acquired near the University of Pavia in Italy by the ROSIS-3HS imaging spectrometer sensor. The dataset has dimensions of 610×340×103, with a spatial dimension of 610×340 and 103 spectral bands. The dataset contains 9 land cover categories, mainly including urban land cover categories such as asphalt, grass, and gravel. Ten samples are randomly selected from each category as the training set, and the remaining samples are used as the test set.

[0075] The Salinas dataset is a rural hyperspectral remote sensing dataset captured by the AVIRIS airborne visible / infrared imaging spectrometer sensor in the Salinas Valley, California. The dataset has dimensions of 512×217×204, with a spatial dimension of 512×217 and 204 spectral bands. The dataset contains 16 land cover categories, including crop land cover categories such as corn, soybeans, cucumbers, and tomatoes. Ten samples from each category were randomly selected as the training set, and the remaining samples were used as the test set.

[0076] The Botswana dataset was captured by the Hyperion sensor aboard NASA's EO-1 satellite over the Okavango Delta in Botswana. The dataset has dimensions of 1476×256×145, with a spatial dimension of 1476×256 and 145 spectral bands. It contains 14 land cover categories, including hippopotamus grass, floodplain grass, reeds, revetment engineering, and woodland. Ten samples were randomly selected from each category as the training set, and the remaining samples were used as the test set.

[0077] (2) Experimental setup

[0078] In the experiment, the training optimizer was set to Adam, the initial learning rate was set to 0.001, and the learning rate decreased by 0.1 every 50 epochs. The model was trained for a total of 200 epochs, and the batch size was set to 32.

[0079] (3) Evaluation indicators

[0080] Overall Accuracy (OA) is the ratio of the number of correctly classified pixels to the total number of sample pixels.

[0081] Average Accuracy (AA) is the average classification accuracy, representing the average classification accuracy across different categories.

[0082] The Kappa coefficient is used to evaluate the consistency between the classification results and the true ground values.

[0083] (4) Comparative Experiment

[0084] Experiments were conducted to compare DHNet with eight representative hyperspectral image classification methods on the four publicly available hyperspectral image datasets mentioned above. The experimental results on the Indian Pines dataset are shown in Table 1, the Pavia University dataset in Table 2, the Salinas dataset in Table 3, and the Botswana dataset in Table 4.

[0085] Table 1: Results of comparative experiments using 8 representative methods and DHNet on the Indian Pines dataset.

[0086] OA 80.20 91.57 94.96 96.20 92.38 AA 68.03 83.61 89.74 78.24 83.38 Kappa 77.31 90.33 94.25 95.67 91.29 Method HybridSN RSSAN A2S2KResNet DHNet OA 93.89 93.31 96.28 97.51 AA 83.02 85.69 90.51 91.23 Kappa 93.02 92.36 95.75 97.16

[0087] Table 2: Results of comparative experiments using 8 representative methods and DHNet on the Pavia University dataset.

[0088]

[0089]

[0090] Table 3: Results of comparative experiments using 8 representative methods and DHNet on the Salinas dataset.

[0091] OA 86.05 85.06 93.67 92.35 91.43 AA 90.84 92.60 96.98 96.43 95.41 Kappa 84.48 83.40 92.48 91.49 90.49 Method HybridSN RSSAN A2S2KResNet DHNet OA 93.07 85.75 92.80 95.32 AA 96.60 91.83 96.72 97.09 Kappa 92.29 84.18 91.98 94.80

[0092] Table 4: Results of comparative experiments using 8 representative methods with DHNet on the Botswana dataset.

[0093] OA 91.31 94.27 97.41 96.24 91.78 AA 92.31 94.43 97.44 96.46 92.35 Kappa 90.85 93.79 97.19 95.92 91.10 Method HybridSN RSSAN A2S2KResNet DHNet OA 92.66 93.52 97.71 99.48 AA 92.92 94.18 98.00 99.48 Kappa 92.04 92.97 97.52 99.43

[0094] As can be seen from Tables 1 to 4, DHNet has higher classification accuracy.

[0095] In another embodiment, based on four datasets—Indian Pines, Pavia University, Salinas, and Botswana—the impact of different neighborhood image patch sizes and preserved spectral dimensions on classification results was also investigated (e.g., Figure 8 As shown), and the impact of different training sample rates on classification results (e.g. Figure 9 (As shown). When the neighborhood image patch size is 15×15, the first 16 spectral dimensions are retained, and the training sample rate is 50%, DHNet achieves good classification accuracy.

[0096] Another embodiment of this application relates to a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the method embodiments described above.

[0097] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0098] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A remote sensing land cover classification method based on dual heterogeneous networks for near-site security, characterized in that, include: The hyperspectral image is reduced in dimensionality using principal component analysis. The input image patch is obtained by sampling the pixel to be classified in the dimensionality-reduced hyperspectral image. The input image patch is input into a pre-trained classification model. Spatial-specific features are obtained using the spatial-specific branch of the classification model, and spectral-specific features are obtained using the spectral-specific branch of the classification model. The spatial-specific branch extracts spatial-specific features based on a spatial-spectral joint involution operation, which is used to fuse central spectral features and neighborhood mean features. The spatial calibration module of the classification model is used to calibrate the spectral specific features, and the spectral calibration module of the classification model is used to calibrate the spatial specific features. Based on the calibrated spatial specific features, the calibrated spectral specific features, and the center spectral vector of the input image patch, the spatial-spectral joint features are obtained. Based on the spatial-spectral joint features, the classifier of the classification model is used to classify land features and obtain the land feature category corresponding to the pixel to be classified. The spatially specific branch consists of three sequentially connected multi-scale involute blocks, with a spatial attention module positioned between adjacent multi-scale involute blocks. Each multi-scale involute block contains two connected 3×3 spatial-spectral joint involute kernels and one 5×5 spatial-spectral joint involute kernel. The step of inputting the input image patch into a pre-trained classification model and using the spatially specific branch of the classification model to obtain spatially specific features includes: The preliminary features of the input image patch are extracted using the pre-convolutional layer of the classification model. These preliminary features are then input into the first multi-scale inconvolutional block, and so on, until the spatially specific features are output from the last multi-scale inconvolutional block. Any one of these multi-scale inconvolutional blocks... l The output is expressed by the formula: ； in, For the first Input to multi-scale involute blocks, For the first The output of multi-scale involute blocks , For the aforementioned preliminary features, This represents the 3×3 spatial-spectral joint involution kernel. This refers to the 5×5 spatial-spectral joint involution kernel. Represents the RelU function; The spectral-specific branch consists of three sequentially connected multi-scale convolutional blocks, with a channel attention module positioned between adjacent multi-scale convolutional blocks. Each multi-scale convolutional block contains two connected 3×3 convolutional kernels and one 5×5 convolutional kernel. Obtaining spectral-specific features using the spectral-specific branch of the classification model includes: The preliminary features are input into the first multi-scale convolutional block, and so on until the last multi-scale convolutional block outputs the spectral specific features. Any one of these multi-scale convolutional blocks... The output is expressed by the formula: ； in, For the first The input of a multi-scale convolutional block, For the first The output of a multi-scale convolutional block , For the aforementioned preliminary features, This refers to the 3×3 convolution kernel. This refers to the 5×5 convolution kernel.

2. The remote sensing land cover classification method based on dual heterogeneous networks for near-site security as described in claim 1, characterized in that, The spectral features of the fusion center and the mean features of the neighborhood are achieved through the following formula: ； in, The pixel representing the input of the spatial-spectral joint involution operation. The central spectral characteristics, This indicates that the center is located at the pixel. The size of the receptive field is × The neighborhood mean characteristic, and This represents the convolution operation. This represents BN normalization and the ReLU function. This represents the generated spatial spectrum joint involution kernel.

3. A remote sensing land cover classification method based on a dual heterogeneous network for near-site security according to any one of claims 1 to 2, characterized in that, The spatial calibration module of the classification model is used to calibrate the spectral specific features, including: Average pooling and max pooling operations are performed on the spatial specific features respectively to obtain a first intermediate feature and a second intermediate feature, and the first intermediate feature and the second intermediate feature are concatenated to obtain a first tensor; The first tensor is input into the 3×3 convolutional layer of the spatial calibration module for learning; The spatial weights are obtained by passing the output of the 3×3 convolutional layer of the spatial calibration module through the Sigmod function. The spectral specific features are then calibrated based on the spatial weights to obtain the calibrated spectral specific features.

4. The remote sensing ground feature classification method based on dual heterogeneous networks for near-site security as described in claim 3, characterized in that, The spatial weights are obtained by passing the output of the 3×3 convolutional layer of the spatial calibration module through the Sigmoid function using the following formula. The spectral specific features are then calibrated based on these spatial weights to obtain the calibrated spectral specific features: ； in, For the spectral specificity characteristics, For the aforementioned space-specific features, This refers to the average pooling operation. This refers to the max pooling operation. This represents the 3×3 convolutional layer of the spatial calibration module. This refers to the Sigmod function. The calibrated spectral specificity characteristics, This indicates a join operation.

5. A remote sensing land cover classification method based on a dual heterogeneous network for near-site security, as described in claim 4, is characterized in that... The calibration of the spatial specific features using the spectral calibration module of the classification model includes: The spectral specific features are subjected to average pooling and max pooling operations respectively to obtain the third intermediate feature and the fourth intermediate feature. The third intermediate feature and the fourth intermediate feature are then input into a multilayer perceptron composed of two fully connected layers for learning. The outputs of the multilayer perceptron are summed and then the spectral weights are obtained through the Sigmod function. The spatial specific features are then calibrated based on the spectral weights to obtain the calibrated spatial specific features.

6. A remote sensing land cover classification method based on a dual heterogeneous network for near-site security, as described in claim 5, is characterized in that... The output of the 3×3 convolutional layer of the spectral calibration module is used to obtain spectral weights through the Sigmoid function using the following formula. The spatial specific features are then calibrated based on these spectral weights to obtain the calibrated spatial specific features: ； in, This refers to the multilayer sensor. This refers to the calibrated spatial specific characteristics.

7. A remote sensing land cover classification method based on a dual heterogeneous network for near-site security according to any one of claims 1 to 2, characterized in that, The step of obtaining joint spatial-spectral features based on calibrated spatial-specific features, calibrated spectral-specific features, and the center spectral vector of the input image patch includes: The calibrated spatial-specific features and the calibrated spectral-specific features are added pixel by pixel, and the pixel-by-pixel sum is concatenated with the center spectral vector of the input image patch to obtain the spatial-spectral joint feature, which is expressed by the formula: ； in, The calibrated spatial specific features, The calibrated spectral specificity characteristics, The center spectral vector of the input image patch. For the joint spatial-spectral features, This indicates a join operation.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements a remote sensing ground feature classification method based on a dual heterogeneous network for near-site security, as described in any one of claims 1 to 7.

Images