Remote sensing image classification method based on three-branch spatial spectrum convolutional neural network

Abstract

The invention belongs to the technical field of remote sensing image processing, and particularly discloses a remote sensing image classification method based on a three-branch spatial spectrum convolutional neural network, and the method comprises the following steps: constructing a hyperspectral remote sensing image classification data set; a classification model of a three-branch spatial-spectral convolutional neural network based on multi-attention assistance is constructed, and the classification model comprises a spectral feature branch, a spatial feature branch and a spatial-spectral joint feature branch which are used for extracting spectral features, spatial features and spatial-spectral joint features in the image respectively; training the classification model by using the data set to obtain a trained classification network model; and inputting a to-be-classified hyperspectral remote sensing image into the trained classification network model to obtain an image classification result. The hyperspectral remote sensing image classification model generation method can solve the problems of weak global information modeling capability, high calculation complexity, insufficient spectral feature mining and insufficient spatial-spectral cooperative characterization in a current hyperspectral classification method, and realizes hyperspectral remote sensing image classification model generation with high recognition accuracy and efficiency.

Description

Technical Field

[0001] This application belongs to the field of remote sensing image processing technology, and more specifically, relates to a remote sensing image classification method based on a three-branch spatial spectrum convolutional neural network. Background Technology

[0002] Hyperspectral remote sensing imagery integrates the spectral and spatial information of ground features, enabling pixel-level fine-grained identification and classification of ground features thanks to its "image-spectrum integration" characteristic. This has significant application value in many fields such as precision agriculture, environmental monitoring, national defense, and soil assessment. However, as a typical high-dimensional data source, hyperspectral data is typically large in volume and complex in structure. Furthermore, the variability of spectra leads to severe phenomena of "different spectra for the same object" and "same spectra for different objects," and the non-linear separability of ground features poses a significant challenge to ground feature classification.

[0003] Traditional hyperspectral remote sensing classification methods are mostly based on handcrafted features and shallow machine learning models, often suffering from poor discriminative power and weak generalization ability, which limits classification accuracy and reduces practicality. In recent years, deep learning, with its powerful automatic feature learning capabilities, has been widely applied to hyperspectral remote sensing image classification tasks, resulting in a large number of deep learning classification methods. Compared to traditional shallow models based on handcrafted features, deep learning can deeply mine the essential attribute features of ground objects through hierarchical feature extraction and nonlinear transformation, improving the accuracy of ground object classification. From a computational efficiency perspective, this automated feature learning paradigm reduces reliance on expert knowledge, significantly lowers labor costs, and improves classification efficiency. Among them, convolutional neural networks (CNNs), as a typical paradigm of deep learning frameworks, have powerful local detail description capabilities. Through a hierarchical learning mechanism that combines convolution and pooling, they achieve progressive feature expression from local details to high-level semantics, demonstrating great potential in hyperspectral remote sensing classification. Compared to other network models, CNNs are more flexible and adaptable, able to adjust their structure to suit the different dimensional characteristics of hyperspectral remote sensing images, making them highly applicable. In recent years, attention mechanisms have been introduced into hyperspectral classification tasks, further driving performance improvements. Through an end-to-end network structure, attention mechanisms adaptively learn the importance weights of feature channels and spatial locations, achieving feature recalibration in both the spectral and spatial domains. This enhances the discriminative feature responses of target objects and suppresses background noise interference through weight attenuation, thereby optimizing the discriminative power of feature representations.

[0004] While deep learning has significantly improved classification accuracy and driven the rapid development of hyperspectral remote sensing classification, existing convolutional neural networks struggle to effectively extract global correlation information due to the locality of inertia inherent in convolutional computation. Although self-attention methods can incorporate global information, they greatly increase model complexity and computational load, reducing the network's practicality and adaptability to small sample sizes. Furthermore, existing multi-branch networks focus excessively on spatial information mining while neglecting spectral information mining, overlooking the importance of spatial-spectral co-operation. This leads to overly smoothed classification results, degrading the model's classification performance.

[0005] Therefore, how to solve the problems of weak global information modeling ability, high computational complexity, insufficient spectral feature mining, and inadequate spatial-spectral co-representation in existing hyperspectral image classification methods based on convolutional neural networks and attention mechanisms is a pressing research challenge. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this application aims to provide a remote sensing image classification method based on a three-branch spatial-spectral convolutional neural network. This method can solve the problems of weak global information modeling ability, high computational complexity, insufficient spectral feature mining, and inadequate spatial-spectral co-representation in current hyperspectral classification methods, thereby achieving high recognition accuracy and efficiency in generating hyperspectral remote sensing image classification models.

[0007] To achieve the above objectives, in a first aspect, this application provides a remote sensing image classification method based on a three-branch spatial-spectral convolutional neural network, comprising the following steps: S10: Acquire and preprocess hyperspectral remote sensing satellite images, extract the three-dimensional neighborhood cube of each pixel, and construct a hyperspectral remote sensing image classification dataset. S20, construct a classification model based on a multi-attention-assisted three-branch spatial-spectral convolutional neural network. The classification model includes a spectral feature branch, a spatial feature branch, and a spatial-spectral joint feature branch. The spectral feature branch is used to extract deep implicit information in the spectrum; the spatial feature branch is used to extract spatial discriminative structural information and mine non-local sensitive information; the spatial-spectral joint feature branch is used to extract spatial-spectral coupling features and mine non-local sensitive information. S30, The classification model is trained using the hyperspectral remote sensing image classification dataset to obtain the trained classification network model; S40, input the hyperspectral remote sensing image to be classified into the trained classification network model to obtain the image classification result.

[0008] The remote sensing image classification method based on a three-branch spatial-spectral convolutional neural network provided in this application has the following advantages: By constructing a dedicated classification model containing three parallel feature branches—spectral, spatial, and spatial-spectral joint—it extracts spectral features, spatial features, and spatial-spectral joint features from the image, thereby improving the model's discriminative ability. Specifically, the spectral feature branch is used to achieve high-intensity mining of deep, implicit spectral information; the spatial feature branch is used to mine effective non-local attention-sensitive information; and the spatial-spectral joint feature branch utilizes a cross-joint attention module to mine non-local sensitive information, effectively improving the model's spatial-spectral collaborative performance. Through the construction of the classification network model, it effectively solves the problems of weak global information modeling ability, high computational complexity, insufficient spectral feature mining, and inadequate spatial-spectral collaborative representation in current hyperspectral classification methods, achieving high recognition accuracy and efficiency in generating hyperspectral remote sensing image classification models.

[0009] As a further preferred option, step S10 specifically includes: Acquire hyperspectral remote sensing satellite images of different land cover categories, different spatial resolutions, and covering multiple combinations of land cover; Hyperspectral remote sensing satellite imagery Principal component analysis was performed along the spectral dimension to reduce the dimensionality of the original B-band hyperspectral data to b-bands, resulting in a dimensionality-reduced hyperspectral image. , where H, W, B and b represent height, width, number of original spectral bands and number of spectral bands after dimensionality reduction, respectively; Zero-filling and edge expansion are performed on the dimension-reduced hyperspectral image. The edge expansion size is [value missing]. , The size of the neighborhood window; Extracting from each effective pixel of the hyperspectral image The three-dimensional neighborhood cube is used to generate N three-dimensional data blocks corresponding to spatial locations, where N = H × W, and N represents the number of spatial pixels in the hyperspectral image; A hyperspectral remote sensing image classification dataset is constructed using three-dimensional data blocks.

[0010] As a further preferred embodiment, in step S20, the classification model includes an input layer, a spectral feature branch, a spatial feature branch, a spatial-spectral joint feature branch, a global average pooling unit, a feature fusion unit, and a fully connected layer; the spectral feature branch, spatial feature branch, and spatial-spectral joint feature branch are all connected to the input layer; the spectral feature branch, spatial feature branch, and spatial-spectral joint feature branch are all connected to the global average pooling unit; the global average pooling unit, feature fusion unit, and fully connected layer are connected sequentially. The input layer is used to receive raw hyperspectral image data, which is then processed by principal component analysis for dimensionality reduction and fed into three subsequent parallel feature extraction branches. The spectral feature branch is used to extract deep implicit information in the spectrum; the spatial feature branch is used to extract spatial discriminative structural information and mine non-local sensitive information; the spatial-spectral joint feature branch is used to extract spatial-spectral coupling features and mine non-local sensitive information. The global average pooling unit is used to perform global average pooling on spatial information, spectral information and spatial-spectral information to compress spatial dimensions. The feature fusion unit is used to integrate the pooled spectral information, spatial information, and spatial-spectral information through element-wise addition to obtain a combined feature map. The fully connected layer is used to dynamically adjust the output weights of the spectral feature branch, spatial feature branch, and combined spatial-spectral feature branch in the combined feature map to obtain the final fused map.

[0011] As a further preferred embodiment, in step S20, the spectral feature branch is composed of three consecutive attention-assisted densely connected convolutional modules in series; the spatial feature branch includes a cross-joint attention module and a first convolutional module connected in sequence; and the spatial-spectral joint feature branch includes a cross-joint attention module and a second convolutional module connected in sequence.

[0012] As a further preferred embodiment, the processing steps of the attention-assisted densely connected convolutional module are as follows: Let the spectral features of the hyperspectral images in the hyperspectral remote sensing image classification dataset be... Where C is the number of spectral channels; The spectral feature X first passes through an improved, efficient channel attention layer (IECA). The IECA performs nonlinear modeling of the channel dimension through two parallel one-dimensional convolutions of different scales, generating the channel weight vector W. R C Multiply W channel-by-channel with the input feature X to obtain the channel attention-weighted feature X. att ; X att A locally densely connected structure consisting of three consecutive one-dimensional dilated convolutional layers is fed in. Each layer uses 32 convolutional kernels of size 5. The third convolutional layer is set with a dilation rate of 2 to expand the receptive field. All convolutional layers maintain the same size as the original features and are activated by the Mish function. Intermediate spectral feature maps F1, F2 and F3 are generated in sequence. During the feature extraction process, the module adopts a local dense connection mechanism. Specifically, the input of the current layer is formed by concatenating the initial input X of the module with the output features of all previous convolutional layers in the channel dimension, so that each layer fully integrates the effective information of all previous layers and finally forms the fused spectral features F. The feature F output by the densely connected path is combined with the feature X after channel attention weighting.att The final output of the densely connected convolutional module is obtained by adding elements one by one through global residual connections.

[0013] As a further preferred embodiment, the first convolutional module is composed of three consecutive 3D convolutional layers with a kernel size of 7x7x1 combined with a BN layer; the second convolutional module is composed of three consecutive 3D convolutional layers with a kernel size of 3x3x3 combined with a BN layer.

[0014] As a further preferred embodiment, the processing steps of the cross-joint attention module are as follows: Let the input features of the hyperspectral remote sensing image be... ; Inputting feature X, we first perform global average pooling along the spectral dimension to compress the original data and fuse the effective information from each spectral band, resulting in the initial compressed feature F0. ; Global average pooling is performed on F0 along both the horizontal and vertical directions to further compress the data and obtain long-range sensitive information H in both directions. avg W avg Based on this, H avg W avg Convolution operations are performed separately to restore the original size, and then weighted by dot product with the initial compressed feature F0 to obtain the horizontal and vertical intermediate attention W_Attention. and H_Attention ; At the same time, for H avg With W avg Vector multiplication is performed to obtain cross-sensitive features that fuse bidirectional discriminative information. These features are then restored to their original size via convolution and weighted with the original compressed features F0 to obtain the intermediate attention HW_Attention. ; The three intermediate attentions are added element-wise to obtain the final cross-joint attention. This cross-joint attention is then combined with the original input feature X through element-wise multiplication to obtain the attention-enhanced output feature X. out .

[0015] As a further preferred option, step S30 also includes: The initial hyperspectral classification model is trained using the classification cross-entropy loss function, calculated as follows:

[0016] Where m represents the total number of training samples, The predicted label for the i-th sample. This is the real label. This represents the loss value between the predicted label and the actual label.

[0017] Secondly, this application provides an electronic device, including a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the remote sensing image classification method based on a three-branch spatial spectrum convolutional neural network as described in any of the above.

[0018] Thirdly, this application provides a computer-readable storage medium storing instructions that, when executed by a computer, perform the remote sensing image classification method based on a three-branch spatial spectrum convolutional neural network as described in any of the above.

[0019] It is understood that the beneficial effects of the second and third aspects mentioned above can be found in the relevant descriptions in the first aspect above, and will not be repeated here. Attached Figure Description

[0020] Figure 1 This is a flowchart of the remote sensing image classification method using a multi-attention-assisted three-branch spatial-spectral convolutional neural network provided in this application. Figure 2 This is a flowchart of the remote sensing image classification method using a multi-attention-assisted three-branch spatial spectrum convolutional neural network provided in the embodiments of this application; Figure 3 This is a network structure framework diagram provided in the embodiments of this application; Figure 4 This is a schematic diagram of the attention-assisted densely connected convolutional module ARD Block structure provided in an embodiment of this application; Figure 5 This is a schematic diagram of the improved high-efficiency channel attention layer IECA structure within the attention-assisted densely connected convolutional module ARD Block structure provided in this application embodiment; Figure 6 This is a schematic diagram of the Cross-Joint Attention Module (CJA) structure provided in an embodiment of this application; Figure 7 This is a schematic diagram of the electronic device structure provided in the embodiments of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0022] like Figure 1 As shown, this application provides a remote sensing image classification method based on a three-branch spatial spectrum convolutional neural network, including steps S10 to S40, which are detailed below: Step S10: Acquire and preprocess the hyperspectral remote sensing satellite imagery, extract the three-dimensional neighborhood cube of each pixel, and construct a hyperspectral remote sensing image classification dataset.

[0023] In step S10, the acquired hyperspectral remote sensing satellite images can be processed by principal component analysis for dimensionality reduction, zero-filling, and edge expansion, and a fixed-size three-dimensional neighborhood cube can be extracted with each pixel as the center, thereby constructing a classification dataset suitable for training the network model.

[0024] Step S20: Construct a classification model based on a three-branch spatial-spectral convolutional neural network with multi-attention assistance. The classification model includes a spectral feature branch, a spatial feature branch, and a spatial-spectral joint feature branch.

[0025] Among them, the spectral feature branch is used to extract deep hidden information in the spectrum; the spatial feature branch is used to extract spatial discriminative structural information and mine non-local sensitive information; and the spatial-spectral joint feature branch is used to extract spatial-spectral coupling features and mine non-local sensitive information.

[0026] In step S20, the constructed classification model performs deep feature mining on different characteristics of hyperspectral images through parallel spectral feature branches, spatial feature branches, and spatial-spectral joint feature branches. The spectral feature branch focuses on extracting deep, implicit discriminative information from the continuous spectrum; the spatial feature branch aims to capture the spatial structural context in the image and uses an attention mechanism to mine long-distance dependencies to obtain non-local sensitive information; the spatial-spectral joint feature branch simultaneously processes the coupling information of spatial and spectral dimensions, further enhancing the ability to extract non-local sensitive features.

[0027] Step S30: Use the hyperspectral remote sensing image classification dataset to train the classification model and obtain the trained classification network model.

[0028] Step S40: Input the hyperspectral remote sensing image to be classified into the trained classification network model to obtain the image classification result.

[0029] The remote sensing image classification method based on a three-branch spatial-spectral convolutional neural network provided in this application has the following advantages: By constructing a dedicated classification model containing three parallel feature branches—spectral, spatial, and spatial-spectral joint—it extracts spectral features, spatial features, and spatial-spectral joint features from the image, thereby improving the model's discriminative ability. Specifically, the spectral feature branch is used to achieve high-intensity mining of deep, implicit spectral information; the spatial feature branch is used to mine effective non-local attention-sensitive information; and the spatial-spectral joint feature branch utilizes a cross-joint attention module to mine non-local sensitive information, effectively improving the model's spatial-spectral collaborative performance. Through the construction of the classification network model, it effectively solves the problems of weak global information modeling ability, high computational complexity, insufficient spectral feature mining, and inadequate spatial-spectral collaborative representation in current hyperspectral classification methods, achieving high recognition accuracy and efficiency in generating hyperspectral remote sensing image classification models.

[0030] In one embodiment, the technical solution for achieving the above objective can be specifically as follows: Figure 2 As shown, the flowchart of the remote sensing image classification method based on a multi-attention-assisted three-branch spatial-spectral convolutional neural network provided in this embodiment includes: S1: Acquire hyperspectral remote sensing satellite images and perform data preprocessing to construct a hyperspectral remote sensing image classification dataset.

[0031] In this embodiment, the hyperspectral classification dataset can be divided into a training set and a test set. Both the training set and the test set contain different land cover categories and different spatial resolutions to increase the representativeness of the dataset.

[0032] S2: Construct a classification model ATSSN based on a three-branch spatial spectrum convolutional neural network with multi-attention assistance and use it as the initial hyperspectral classification model.

[0033] S3: Train the initial hyperspectral classification model using a hyperspectral remote sensing image classification dataset to obtain the trained classification network model.

[0034] S4: Obtain the hyperspectral remote sensing image to be classified, input the hyperspectral remote sensing image to be classified into the trained classification network model, and obtain the image classification result.

[0035] In this embodiment, step S1 specifically includes: S11: Acquire hyperspectral remote sensing satellite images of different land cover categories, different spatial resolutions, and covering multiple combinations of land cover; S12: Hyperspectral remote sensing satellite imagery Principal component analysis is performed along the spectral dimension to reduce the original B-band hyperspectral data to b-bands, resulting in a dimension-reduced hyperspectral image. Here, H, W, B, and b represent the height, width, number of original spectral bands, and number of dimension-reduced spectral bands, respectively. S13: Perform zero-padding and edge expansion on the dimension-reduced hyperspectral image. The edge expansion size is... , The size of the neighborhood window; S14: Extract from each effective pixel of the hyperspectral image. The three-dimensional neighborhood cube is used to generate N three-dimensional data blocks corresponding to spatial locations, where N = H × W, and N represents the number of spatial pixels in the hyperspectral image; S15: Construct a hyperspectral remote sensing image classification dataset using three-dimensional data blocks.

[0036] In this embodiment, the acquired hyperspectral remote sensing images are subjected to PCA dimensionality reduction and spatial block processing to obtain the final hyperspectral classification dataset.

[0037] In this embodiment, step S2 specifically includes: like Figure 3 As shown, the initial hyperspectral classification model includes: an input layer, a spectral feature branch, a spatial feature branch, a joint spatial-spectral feature branch, a global average pooling unit, a feature fusion unit, and a fully connected layer; The spectral feature branch, spatial feature branch, and joint spatial-spectral feature branch are all connected to the input layer; the spectral feature branch, spatial feature branch, and joint spatial-spectral feature branch are all connected to the global average pooling unit; the global average pooling unit, feature fusion unit, and fully connected layer are connected sequentially. The input layer receives the raw hyperspectral image data, which is then processed by principal component analysis for dimensionality reduction and fed into three parallel feature extraction branches. The spectral feature branch is composed of three consecutive attention-assisted densely connected convolutional modules (ARD blocks) in series.

[0038] High-intensity mining of deep, hidden information in the spectrum is achieved through spectral feature branching; The spatial feature branch includes: a cross-joint attention module (CJA) and a convolutional module; the cross-joint attention module (CJA) and the convolutional module are connected sequentially. Spatial feature branches are used to extract spatial discriminative structural information, thereby uncovering effective non-local attention-sensitive information. The spatial-spectral joint feature branch includes: a cross-joint attention module (CJA) and a convolutional module; the cross-joint attention module (CJA) and the convolutional module are connected sequentially. The spatial-spectral coupling features are fully extracted and non-local sensitive information is mined by using spatial-spectral joint feature branches; The global average pooling unit is used to perform global average pooling on spatial information, spectral information, and spatial-spectral information to compress spatial dimensions. The feature fusion unit is used to integrate the pooled spectral information, spatial information, and spatial-spectral information through element-wise addition to obtain a combined feature map. A fully connected layer is used to dynamically adjust the output weights of the spectral feature branch, spatial feature branch, and combined spatial-spectral feature branch in the combined feature map, resulting in the final fused map.

[0039] In this embodiment, the initial hyperspectral classification model specifically includes: a spectral feature branch consisting of three consecutive ARD blocks in series, enabling high-intensity mining of deep, implicit spectral information; a spatial feature branch equipped with a cross-joint attention module to mine effective non-local attention-sensitive information, and consisting of three consecutive 3D convolutional layers with a kernel size of 7x7x1 combined with a BN layer; and a spatial-spectral joint feature branch utilizing the cross-joint attention module to mine non-local sensitive information, and consisting of three consecutive 3D convolutional layers with a kernel size of 3x3x3 combined with a BN layer, effectively improving the spatial-spectral collaborative performance of the model.

[0040] In this embodiment, the design of the three-branch spatial-spectral feature extraction is mainly based on the structure and limitations of convolutional neural networks and self-attention mechanisms. It combines attention-assisted densely connected convolutional modules and cross-joint attention modules to adaptively extract non-local information with a lightweight structure, fully explore the local and global contextual information and spectral discriminative information of hyperspectral images, reduce complexity while improving the spatial-spectral co-operation performance of the model, thereby effectively improving the classification performance of the model.

[0041] In this embodiment, as Figure 4 As shown, the processing steps of the attention-assisted densely connected convolutional module ARD Block are as follows: Let the spectral features of the hyperspectral images in the hyperspectral remote sensing image classification dataset be... , where C is the number of spectral channels.

[0042] Spectral characteristics X, such as Figure 5 As shown, the channel weight vector W is first passed through an improved, efficient channel attention layer, IECA. IECA performs nonlinear modeling of the channel dimension through two parallel one-dimensional convolutions of different scales. R C Multiply W channel-by-channel with the input feature X to obtain the channel attention-weighted feature X. att ; Then X attA locally densely connected structure consisting of three consecutive one-dimensional dilated convolutional layers is fed in. Each layer uses 32 convolutional kernels of size 5. The third convolutional layer is set with a dilation rate of 2 to expand the receptive field. All convolutional layers maintain the same size as the original features and are activated by the Mish function. Intermediate spectral feature maps F1, F2 and F3 are generated in sequence. During feature extraction, the module employs a locally dense connection mechanism. Specifically, the input of the current layer is formed by concatenating the initial input X of the module with the output features of all preceding convolutional layers along the channel dimension, allowing each layer to fully integrate the effective information from all previous layers, ultimately forming the fused spectral features F. Finally, the features F output by the densely connected path are combined with the features X after channel attention weighting. att The final output of the ARD Block is obtained by adding elements one by one through global residual connections.

[0043] In one embodiment of this application, in order to fully explore the rich spectral diagnostic information contained in the continuous spectrum of hyperspectral images, this embodiment adopts a cascaded combination structure of three consecutive ARD blocks.

[0044] The convolutional module of the spatial feature branch consists of three consecutive 3D convolutional layers with a kernel size of 7x7x1 combined with a BN layer; the convolutional module of the spatial-spectral joint feature branch consists of three consecutive 3D convolutional layers with a kernel size of 3x3x3 combined with a BN layer.

[0045] As one example, a hyperspectral classification method based on a three-branch spatial-spectral convolutional neural network with multi-attention assistance is constructed. It introduces a cross-joint attention module and combines three consecutive 3D convolutional layers to adaptively extract non-local information with a lightweight structure, thereby fully mining the local and global contextual information of hyperspectral images.

[0046] As one example, such as Figure 6 As shown, in the spatial feature branch, the cross-joint attention module (CJA) first compensates for the inherent locality defects of convolution, mining effective non-local attention-sensitive information. Then, it passes through three consecutive 3D convolutional layers with 64 kernels and a size of 7x7x1. The CJA module first processes the input feature map... Global average pooling is performed to compress the original data. Then, global average pooling is performed horizontally and vertically to further compress the data and obtain long-range sensitive information in both directions. Subsequently, based on the obtained long-range sensitive information in both directions, convolution is performed to restore the original size, and then the compressed features are multiplied and weighted to obtain the horizontal and vertical intermediate attention W_Attention and H_Attention.

[0047] Simultaneously, cross-sensitive information containing discriminative information from both directions is obtained through vector multiplication of long-distance sensitive information in the horizontal and vertical directions. This cross-sensitive information is then multiplied and weighted with the original compressed features to obtain the intermediate attention HW_Attention. Finally, the three intermediate attentions are added together to obtain the final cross-joint attention, and the resulting CJA attention is combined with the original feature map through element-wise multiplication, thereby enhancing the network's ability to discriminate key information.

[0048] As one example, the spatial-spectral joint feature branch uses three consecutive 3D convolutional layers with 64 kernels and a size of 3x3x3, and the rest is the same as the spatial feature branch processing.

[0049] Step S3 also includes: training the initial hyperspectral classification model using the classification cross-entropy loss function, as expressed below:

[0050] Where m represents the total number of training samples, The predicted label for the i-th sample. This is the real label. This represents the loss value between the predicted label and the actual label.

[0051] As one example, an initial learning rate is set for the network, and a learning rate decay strategy is selected. The training parameters and optimizer are adjusted, and accuracy verification is performed simultaneously during training. The accuracy of the model is evaluated for each training session using a validation sample dataset. The network model parameters are adjusted based on the evaluation accuracy and loss value of the validation set, and the accuracy and loss value of the network model are recorded for each training session to obtain the optimal network parameter model.

[0052] The following is a specific implementation example of this application: 1. Experimental conditions: Three hyperspectral image datasets were used, including: The first hyperspectral dataset is the Pavia University dataset obtained using ROSIS, which covers 610×340 pixels and contains 9 land cover classes.

[0053] The second dataset is the Salinas dataset, acquired using an airborne visible / infrared imaging spectrometer. This dataset covers 512×217 pixels and includes 16 land cover classes.

[0054] The third dataset is the Whu-Hi-LongKou dataset, which was acquired in July 2018 by an unmanned aerial vehicle-borne hyperspectral imaging system. The dataset is 550×400 pixels in size and contains 9 types of ground features.

[0055] 2. Experimental content and results analysis: This experiment uses three quantitative evaluation indicators: overall accuracy (OA), average accuracy (AA), and Kappa coefficient (Kappa).

[0056] 2.1 This embodiment and nine existing methods (SSUN, ASSMN, SSRN, SSFCN, MSDN, MCNN-CP, TPNet, TriCNN, and 2SRS) were used to classify the Pavia University dataset, and three quantitative evaluation metrics were recorded. The results are shown in Table 1.

[0057] Table 1

[0058] 2.2 This embodiment and nine existing methods (SSUN, ASSMN, SSRN, SSFCN, MSDN, MCNN-CP, TPNet, TriCNN, and 2SRS) were used to classify the Salinas dataset, and three quantitative evaluation metrics were recorded. The results are shown in Table 2.

[0059] Table 2

[0060] 2.3 This embodiment and nine existing methods (SSUN, ASSMN, SSRN, SSFCN, MSDN, MCNN-CP, TPNet, TriCNN, and 2SRS) were used to classify the Whu-Hi-LongKou dataset, and three quantitative evaluation metrics were recorded. The results are shown in Table 3.

[0061] Table 3

[0062] As shown in Tables 1-3, on the three datasets, the overall classification accuracy (OA) and Kappa coefficient of the method in this embodiment are higher than those of the nine existing methods. The average classification accuracy (AA) and Kappa coefficient of the Pavia University dataset, Salinas dataset, and Whu-Hi-LongKou dataset are also higher than those of the nine existing methods. This demonstrates the effectiveness and superiority of the method proposed in this embodiment.

[0063] The technical solution of this embodiment involves collecting hyperspectral remote sensing images and preprocessing the images to obtain a hyperspectral remote sensing image classification dataset. This dataset is then divided into a training set and a test set. A classification model based on a three-branch convolutional neural network with multi-attention assistance is constructed. Training samples are input into the classification model in batches to train and optimize the network parameters. A 1% validation set is used for verification. After multiple training iterations, the network finally converges, resulting in a well-trained model. The test set is then input into the trained model, and the class prediction probability image is obtained after testing.

[0064] This application also discloses an electronic device. (See reference...) Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. The electronic device 500 may include: at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.

[0065] The communication bus 502 is used to enable communication between these components.

[0066] The user interface 503 may include a display screen, and optionally, the user interface 503 may also include a standard wired interface or a wireless interface.

[0067] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0068] This application also discloses a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute the aforementioned remote sensing image classification method based on a multi-attention-assisted three-branch convolutional neural network.

[0069] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A remote sensing image classification method based on a three-branch spatial-spectral convolutional neural network, characterized in that, Includes the following steps: S10: Acquire and preprocess hyperspectral remote sensing satellite images, extract the three-dimensional neighborhood cube of each pixel, and construct a hyperspectral remote sensing image classification dataset. S20, construct a classification model based on a multi-attention-assisted three-branch spatial-spectral convolutional neural network. The classification model includes a spectral feature branch, a spatial feature branch, and a spatial-spectral joint feature branch. The spectral feature branch is used to extract deep implicit information in the spectrum; the spatial feature branch is used to extract spatial discriminative structural information and mine non-local sensitive information; the spatial-spectral joint feature branch is used to extract spatial-spectral coupling features and mine non-local sensitive information. S30, The classification model is trained using the hyperspectral remote sensing image classification dataset to obtain the trained classification network model; S40, input the hyperspectral remote sensing image to be classified into the trained classification network model to obtain the image classification result.

2. The remote sensing image classification method based on a three-branch spatial spectrum convolutional neural network as described in claim 1, characterized in that, Step S10 is as follows: Acquire hyperspectral remote sensing satellite images of different land cover categories, different spatial resolutions, and covering multiple combinations of land cover; Hyperspectral remote sensing satellite imagery Principal component analysis was performed along the spectral dimension to reduce the dimensionality of the original B-band hyperspectral data to b-bands, resulting in a dimensionality-reduced hyperspectral image. , where H, W, B and b represent height, width, number of original spectral bands and number of spectral bands after dimensionality reduction, respectively; Zero-filling and edge expansion are performed on the dimension-reduced hyperspectral image. The edge expansion size is [value missing]. , The size of the neighborhood window; Extracting from each effective pixel of the hyperspectral image The three-dimensional neighborhood cube is used to generate N three-dimensional data blocks corresponding to spatial locations, where N = H × W, and N represents the number of spatial pixels in the hyperspectral image; A hyperspectral remote sensing image classification dataset is constructed using three-dimensional data blocks.

3. The remote sensing image classification method based on a three-branch spatial spectrum convolutional neural network as described in claim 1, characterized in that, In step S20, the classification model includes an input layer, a spectral feature branch, a spatial feature branch, a spatial-spectral joint feature branch, a global average pooling unit, a feature fusion unit, and a fully connected layer; the spectral feature branch, spatial feature branch, and spatial-spectral joint feature branch are all connected to the input layer; the spectral feature branch, spatial feature branch, and spatial-spectral joint feature branch are all connected to the global average pooling unit; the global average pooling unit, feature fusion unit, and fully connected layer are connected sequentially. The input layer is used to receive raw hyperspectral image data, which is then processed by principal component analysis for dimensionality reduction and fed into three subsequent parallel feature extraction branches. The spectral feature branch is used to extract deep implicit information in the spectrum; the spatial feature branch is used to extract spatial discriminative structural information and mine non-local sensitive information; the spatial-spectral joint feature branch is used to extract spatial-spectral coupling features and mine non-local sensitive information. The global average pooling unit is used to perform global average pooling on spatial information, spectral information and spatial-spectral information to compress spatial dimensions. The feature fusion unit is used to integrate the pooled spectral information, spatial information, and spatial-spectral information through element-wise addition to obtain a combined feature map. The fully connected layer is used to dynamically adjust the output weights of the spectral feature branch, spatial feature branch, and combined spatial-spectral feature branch in the combined feature map to obtain the final fused map.

4. The remote sensing image classification method based on a three-branch spatial spectrum convolutional neural network as described in claim 1, characterized in that, In step S20, the spectral feature branch is composed of three consecutive attention-assisted densely connected convolutional modules in series; the spatial feature branch includes a cross-joint attention module and a first convolutional module connected in sequence; the spatial-spectral joint feature branch includes a cross-joint attention module and a second convolutional module connected in sequence.

5. The remote sensing image classification method based on a three-branch spatial spectrum convolutional neural network as described in claim 4, characterized in that, The processing steps of the attention-assisted densely connected convolutional module are as follows: Let the spectral features of the hyperspectral images in the hyperspectral remote sensing image classification dataset be... Where C is the number of spectral channels; The spectral feature X first passes through an improved, efficient channel attention layer (IECA). The IECA performs nonlinear modeling of the channel dimension through two parallel one-dimensional convolutions of different scales, generating the channel weight vector W. R C Multiply W channel-by-channel with the input feature X to obtain the channel attention-weighted feature X. att ; X att A locally densely connected structure consisting of three consecutive one-dimensional dilated convolutional layers is fed in. Each layer uses 32 convolutional kernels of size 5. The third convolutional layer is set with a dilation rate of 2 to expand the receptive field. All convolutional layers maintain the same size as the original features and are activated by the Mish function. Intermediate spectral feature maps F1, F2 and F3 are generated in sequence. During the feature extraction process, the module adopts a local dense connection mechanism. Specifically, the input of the current layer is formed by concatenating the initial input X of the module with the output features of all previous convolutional layers in the channel dimension, so that each layer fully integrates the effective information of all previous layers and finally forms the fused spectral features F. The feature F output by the densely connected path is combined with the feature X after channel attention weighting. att The final output of the densely connected convolutional module is obtained by adding elements one by one through global residual connections.

6. The remote sensing image classification method based on a three-branch spatial spectrum convolutional neural network as described in claim 4, characterized in that, The first convolutional module consists of three consecutive 3D convolutional layers with a kernel size of 7x7x1 combined with a BN layer; the second convolutional module consists of three consecutive 3D convolutional layers with a kernel size of 3x3x3 combined with a BN layer.

7. The remote sensing image classification method based on a three-branch spatial spectrum convolutional neural network as described in claim 4, characterized in that, The processing steps of the cross-joint attention module are as follows: Let the input features of the hyperspectral remote sensing image be... ; Inputting feature X, we first perform global average pooling along the spectral dimension to compress the original data and fuse the effective information from each spectral band, resulting in the initial compressed feature F0. ; Global average pooling is performed on F0 along both the horizontal and vertical directions to further compress the data and obtain long-range sensitive information H in both directions. avg W avg Based on this, H avg W avg Convolution operations are performed separately to restore the original size, and then weighted by dot product with the initial compressed feature F0 to obtain the horizontal and vertical intermediate attention W_Attention. and H_Attention ; At the same time, for H avg With W avg Vector multiplication is performed to obtain cross-sensitive features that fuse bidirectional discriminative information. These features are then restored to their original size via convolution and weighted with the original compressed features F0 to obtain the intermediate attention HW_Attention. ; The three intermediate attentions are added element-wise to obtain the final cross-joint attention. This cross-joint attention is then combined with the original input feature X through element-wise multiplication to obtain the attention-enhanced output feature X. out .

8. The remote sensing image classification method based on a three-branch spatial spectrum convolutional neural network as described in claim 1, characterized in that, Step S30 also includes: The initial hyperspectral classification model is trained using the classification cross-entropy loss function, calculated as follows: Where m represents the total number of training samples, The predicted label for the i-th sample. This is the real label. This represents the loss value between the predicted label and the actual label.

9. An electronic device, characterized in that, The device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to enable the electronic device to perform the remote sensing image classification method based on a three-branch spatial spectrum convolutional neural network as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a computer, perform the remote sensing image classification method based on a three-branch spatial spectrum convolutional neural network as described in any one of claims 1 to 8.

Images