Hyperspectral classification method based on local enhanced selective state space model
By adopting locally enhanced selective state space model and multi-directional local scanning mechanism in hyperspectral image classification, combined with bidirectional spectral scanning technology, the problems of local receptive field limitation and computational overhead in hyperspectral image classification are solved, and the problem of large calculation overhead in hyperspectral image classification is achieved. Improved efficient classification performance in small sample cases.
Patent Information
- Application Number
- CN202410993097.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-23
AI Technical Summary
The existing hyperspectral image classification model has problems such as local receptive field limitation and high computational overhead when dealing with hyperspectral dimensions and small samples. The application of selective state space model in hyperspectral image classification has not been effectively improved.
Using a selective state space model based on local enhancement, combined with multi-directional local scanning mechanism and bidirectional spectral scanning technology, the spectral and spatial information of high-spectral images are extracted through point-by-point convolution and multi-scale convolution neural networks, and the spectral and spatial information of high-spectral images are fused to multi-scale Features to improve classification performance.
Effectively extracting high-level semantic information in hyperspectral images improves classification performance in small samples and improves the accuracy of hyperspectral land classification.
Smart Images

Figure CN118941860A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a hyperspectral classification method based on a local enhanced selective state space model. Background Art
[0002] Hyperspectral images contain rich spectral information ranging from visible light to infrared bands, and have the characteristics of integrating spatial information and spectral information. They are widely used in agricultural detection, military reconnaissance, environmental monitoring, geological detection and other fields. As the basic technology of hyperspectral applications, hyperspectral image classification assigns a category label to each pixel and has received great attention. However, hyperspectral image classification still faces challenges such as high hyperspectral dimension, unbalanced number of labeled samples, and spectral variability.
[0003] In recent years, the development of deep learning has promoted the progress of hyperspectral classification technology. Deep learning models represented by convolutional neural networks and self-attention mechanism networks have received a lot of attention and achieved remarkable success in hyperspectral image classification, and have long dominated. Convolutional neural networks are famous for utilizing local contextual information through shift-invariant convolutions and have long dominated.
[0004] However, the local receptive field and shared static weights of convolutional neural networks limit their ability to model long-range dependencies and cannot effectively capture global context. Networks based on self-attention mechanisms have dynamic weights and the ability to capture global information, but are limited by quadratic computational complexity, resulting in huge computational overhead. Recently, selective state-space models have rapidly emerged in long sequence modeling with linear complexity as a strong competitor to traditional self-attention mechanism networks, but their application in hyperspectral image classification is still in its infancy, and the selective state-space model has not been improved to be suitable for hyperspectral classification tasks.
[0005] In view of this, it is necessary to provide a hyperspectral classification method based on a locally enhanced selective state space model to overcome the above defects. Summary of the invention
[0006] The purpose of the present invention is to provide a hyperspectral classification method based on a locally enhanced selective state space model. A multi-directional local scanning mechanism is used to improve the existing selective state space model. According to the characteristics of hyperspectral images, a bidirectional spectral scanning technology is used to extract the spectral information of the central area of the hyperspectral image. The spectral and spatial information are fused to improve the hyperspectral classification performance. The hyperspectral ground object classification task is better achieved, and the classification performance in the case of small samples is improved. It is applied to the field of intelligent processing of aerial hyperspectral images.
[0007] In order to achieve the above object, the technical solution provided by the present invention is implemented as follows: a hyperspectral classification method based on a local enhanced selective state space model comprises the following steps:
[0008] S1, reads the three-dimensional hyperspectral image and the category label corresponding to each pixel;
[0009] S2, normalize the values in the hyperspectral image and map the data to the interval [0, 1];
[0010] S3, taking each pixel of the three-dimensional hyperspectral image as the center point, cropping the neighborhood of the center point with a fixed size into image blocks, and constructing an image block dataset, where the label of the center point pixel is set as the category label;
[0011] S4, randomly select data from the image block dataset as the training set, and the rest of the data as the test set;
[0012] S5, train the neural network model on the training set, input the image blocks of the training set into the convolution dimensionality reduction backbone built with point-by-point convolution as the basic unit, and obtain the feature map y∈R after point-by-point convolution dimensionality reduction H×W×D , using a multi-directional local scanning mechanism to enhance the selective state space model, and transform the reduced feature map y∈R H×W×D Divide the image y into multiple local windows p Then, the information between different spectral bands is integrated to obtain the hyperspectral feature map y stack Input selective state space model;
[0013] S6, extract the feature map y∈R after dimensionality reduction H×W×D The central area of the image block y is obtained center ∈R s ×s×D , and then scan bidirectionally in the spectral dimension and combine the selective state space model to obtain the regional image block y center ∈R s×s×D The spatial and spectral characteristics of
[0014] S7, uses a multi-scale convolutional neural network to fuse spatial and spectral features, removes redundant information, and retains valuable spatial structure information;
[0015] S8, uses a fully connected feedforward neural network to build a classification head and predict category information; trains the proposed model on the training set, uses an adaptive moment estimation optimizer to iteratively update the model parameters, and tests the performance of the model on the test set to verify the effectiveness of the proposed model.
[0016] Preferably, step S5 further comprises step S51, for an input training set The image block is calculated as follows:
[0017]
[0018] Among them, H, W and C represent the length, width and number of bands of the image respectively. Indicates that the output feature map is at the cth position (h, w) out The value of the channel, Indicates that the input feature map is at position (h, w) in The value of the channel, Indicates that the convolution kernel is in the input channel c in and output channel c out The weight between out Less than c in .
[0019] Preferably, step S5 further includes step S52, wherein the feature map obtained after point-by-point convolution dimensionality reduction is y∈R H×W×D , using the multi-directional local scanning mechanism to enhance the selective state space model, the feature map y∈R after dimensionality reduction H×W×D Divide into multiple local window images y p , the calculation process is:
[0020]
[0021] Partition means dividing the feature map into windows of size h×w in the dimensions of height and width, n h and n w Respectively represent the number of windows in the height and width directions;
[0022] S53, the local window image y p Rearrange:
[0023]
[0024] Among them, Rearrange means that different local window images y p rearrange;
[0025] The rearranged local window image y R The internal height and width dimensions are transposed to obtain the transposed local window image Y T :
[0026]
[0027] Among them, Transpose means transposing in the height and width dimensions;
[0028] S54, the transposed local window image y T Flip in the spatial dimension to get the flipped local window image [y RF,y TF ]:
[0029] [y RF ,y TF ]=Flip([y R ,y T ]),
[0030] Among them, Flip means flipping in the spatial dimension.
[0031] S56, flatten and stack all the above local window images, integrate the information between different spectral bands of the local window images, and obtain the hyperspectral feature map y stack :
[0032] y stack ∈R 4×L×C =Stack(Flatten([y R ,y T ,y RF ,y TF ])),
[0033] Flatten means flattening and arranging in sequence in the spatial dimension, and Stack means stacking these sequences together to integrate the obtained hyperspectral feature map y stack It has four sequences of local window horizontal, vertical, forward and reverse;
[0034] S57, hyperspectral feature map stack The four sequences of input selective state space model are processed as shown in the formula:
[0035] y spatial ∈R L×C =Sum(SSM(y stack )),
[0036] Among them, Sum means adding the four output results, SSM means selective state space model,
[0037] h′ t =Ah t-1 +By t
[0038] The calculation formula is: t =Ch′ t ,
[0039] Among them, y t Indicates input y stack The tth value in h t is the state representation from the input sequence to the output sequence, z represents the output, A is the high-order polynomial mapping operator matrix, B and C are transformed by the input y stackThe linear mapping is obtained, and the mapping formula is:
[0040] B=w b y stack +b b
[0041] C=w c y stack +b c ,
[0042] Among them, w and b are trainable parameters.
[0043] Preferably, step S6 further includes step S61, and the calculation formula for extracting the spectral feature is as follows:
[0044] y flatten ∈R S×D =Flatten(y center ∈R w×w×D ),
[0045] Among them, Flatten means flattening in the spatial dimension, S = w × w;
[0046] y ssm ∈R S×D =SSM(y flatten ∈R S×D ),
[0047] Among them, SSM represents the selective spatial state model, which scans bidirectionally in the spectral dimension;
[0048] y spectral ∈R 1×C =Sigmoid(Average(y ssm )),
[0049] Among them, Average means taking the average value in the spatial dimension, and the sigmoid function is used to generate the control signal of the spectral dimension. Its calculation formula is:
[0050]
[0051] Multiply the attention signal of the spectral channel with the spatial feature to obtain the feature of the fusion of the spectral feature and the spatial feature. The calculation formula is: feature =y spectral ⊙y spatial
[0052] Here, ⊙ represents the Hadamard product.
[0053] Preferably, step S7 further includes step S71, and the calculation process is as follows:
[0054] y s1 =DWConv3×3 (PWConv(y feature ))
[0055] y s2 =PWConv(DWConv 5×5 (y s1 ))
[0056] y signal =Gelu(PWConv(y feature ))
[0057] z out =y signal ☉y s2 ,
[0058] Among them, y feature represents input, PWConv represents point-by-point convolution, DWConv 3×3 Denotes a depth-separable convolution with a convolution kernel of 3×3, DWConv 5×5 represents a depth-wise separable convolution with a convolution kernel of 5×5, Gelu represents a Gaussian error linear function, and ⊙ represents a Hadamard product.
[0059] Compared with the existing technology, the beneficial effect is that it comprehensively considers the characteristics of aerial hyperspectral images, combines the local state space multi-directional scanning mechanism and the bidirectional spectral scanning mechanism, can effectively extract high-level semantic information in hyperspectral images, better realizes the hyperspectral ground object classification task, and improves the classification performance in small sample conditions.
[0060] Other features and advantages of the present invention will be set forth in the following description, and part will be apparent from the description, or can be understood by implementation of the present invention. Features and advantages of the present invention can be realized and obtained by the elements and combinations specifically pointed out in the appended application scope. These and other features of the present invention will become more clearly understood according to the following description and the appended claims, or can be understood by the implementation of the embodiments described in the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0062] Figure 1 The present invention provides a flowchart of a hyperspectral classification method based on a local enhanced selective state space model.
[0063] Figure 2 Schematic diagram of a multi-scale convolutional neural network in an embodiment of the present invention.
[0064] Figure 3 It is a comparison chart of the classification results of the embodiments of the present invention on the Indian Pines dataset.
[0065] Figure 4 It is a comparison chart of the classification results of the embodiments of the present invention on the WHU-Hi-LongKou dataset. DETAILED DESCRIPTION
[0066] In order to make the purpose, technical solution and beneficial technical effect of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be understood that the specific implementation methods described in this specification are only for explaining the present invention, not for limiting the present invention.
[0067] See also Figure 1 to Figure 2 The present invention provides a hyperspectral classification method based on a local enhanced selective state space model, comprising the following steps:
[0068] S1, reads the three-dimensional hyperspectral image and the category label corresponding to each pixel;
[0069] S2, normalize the values in the hyperspectral image and map the data to the interval [0, 1];
[0070] Specifically, step S2 also includes step S21. In this embodiment, two public data sets, Indian Pines (IP) and WHU-Hi-LongKou (LK), are used as examples to normalize the values in the hyperspectral image and map the values to the interval [0, 1].
[0071] The processing process is expressed as follows:
[0072]
[0073] Among them, x max and x min They represent the maximum and minimum values of the numerical values in the hyperspectral image respectively; the hyperspectral image is divided into multiple three-dimensional image blocks to construct a data set, and the background image blocks that do not contain category information are removed.
[0074] S3, taking each pixel of the three-dimensional hyperspectral image as the center point, cropping the neighborhood of the center point with a fixed size into image blocks, and constructing an image block dataset, where the label of the center point pixel is set as the category label;
[0075] S4, randomly select a certain proportion (1%) of the data in the image block dataset as the training set, and the rest of the data as the test set;
[0076] S5, train the neural network model on the training set, input the image blocks of the training set into the convolution dimensionality reduction backbone built with point-by-point convolution as the basic unit, and obtain the feature map y∈R after point-by-point convolution dimensionality reduction without changing the spatial resolution through a 1×1 convolution kernel. H×W×D , using a multi-directional local scanning mechanism to enhance the selective state space model, and transform the reduced feature map y∈R H×W×D Divide the image y into multiple local windows p Then, the information between different spectral bands is integrated to obtain the hyperspectral feature map y stack Input selective state space model can reduce computational burden;
[0077] Specifically, step S5 further includes step S51, for an input training set y∈R H×W×D The image block is calculated as follows:
[0078]
[0079] Among them, H, W and C represent the length, width and number of bands of the image respectively. Indicates that the output feature map is at the cth position (h, w) out The value of the channel, Indicates that the input feature map is at position (h, w) in The value of the channel, Indicates that the convolution kernel is in the input channel c in and output channel c out The weight between out Less than c in .
[0080] Specifically, step S5 also includes step S52, the feature map obtained after point-by-point convolution dimensionality reduction is y∈R H×W×D , using the multi-directional local scanning mechanism to enhance the selective state space model, the feature map y∈R after dimensionality reduction H×W×D Divide into multiple local window images y p , the calculation process is:
[0081]
[0082] Partition means dividing the feature map into windows of size h×w in the dimensions of height and width, n h and n w Respectively represent the number of windows in the height and width directions;
[0083] S53, the local window image yp Rearrange:
[0084]
[0085] Among them, Rearrange means that different local window images y p rearrange;
[0086] The rearranged local window image y R Transpose the internal height and width dimensions to obtain the transposed local window image y T :
[0087]
[0088] Among them, Transpose means transposing in the height and width dimensions;
[0089] S54, the transposed local window image y T Flip in the spatial dimension to get the flipped local window image [y RF ,y TF ]:
[0090] [y RF ,y TF ]=Flip([y R ,y T ),
[0091] Among them, Flip means flipping in the spatial dimension.
[0092] S56, the above-mentioned local window image (y p ,y R ,y T , [y RF ,y TF ]) are flattened and stacked together to integrate the information between different spectral bands of the local window image to obtain the hyperspectral feature map y stack :
[0093] y stack ∈R 4×L×C =Stack(Flatten([y R ,y T ,y RF ,y TF ])),
[0094] Flatten means flattening and arranging in sequence in the spatial dimension, and Stack means stacking these sequences together to integrate the obtained hyperspectral feature map y stack It has four sequences of local window horizontal, vertical, forward and reverse;
[0095] S57, hyperspectral feature map stack The four sequences of input selective state space model are processed as shown in the formula:
[0096] y spatial ∈R L×C =Sum(SSM(y stack )),
[0097] Among them, Sum means adding the four output results, SSM means selective state space model,
[0098] h′ t =Ah t-1 +By t
[0099] The calculation formula is: t =Ch′ t ,
[0100] Among them, y t Indicates input y stack The tth value in h t is the state representation from the input sequence to the output sequence, z represents the output, A is the high-order polynomial mapping operator matrix, B and C are transformed by the input y stack The linear mapping is obtained, and the mapping formula is:
[0101] B=w b y stack +b b C=w c y stack +b c ,
[0102] Among them, w and b are trainable parameters
[0103] S6, extract the feature map y∈R after dimensionality reduction H×W×D The central area of the image block y is obtained center ∈R s ×s×D , and then scan bidirectionally in the spectral dimension and combine the selective state space model to obtain the regional image block y center ∈R s×s×D The spatial and spectral characteristics of
[0104] Specifically, step S6 also includes step S61, the calculation formula for extracting spectral features is as follows:
[0105] y flatten ∈R S×D =Flatten(y center ∈R w×w×D ),
[0106] Among them, Flatten means flattening in the spatial dimension, S = w × w;
[0107] y ssm ∈R S×D =SSM(y flatten ∈R S×D ,
[0108] Among them, SSM represents the selective spatial state model, and the spatial features are obtained by bidirectional scanning in the spectral dimension;
[0109] y specral ∈R 1×C =Sigmoid(Average(y ssm )),
[0110] Among them, Average means taking the average value in the spatial dimension, and the sigmoid function is used to generate the control signal of the spectral dimension. Its calculation formula is:
[0111]
[0112] Multiply the attention signal of the spectral channel with the spatial feature to obtain the feature of the fusion of the spectral feature and the spatial feature. The calculation formula is: feature =y spectral ⊙y spatial
[0113] Where ⊙ represents the Hadamard product, y feature It is obtained by multiplying the spatial features and the spectral features.
[0114] S7, uses a multi-scale convolutional neural network to fuse spatial and spectral features, removes redundant information, and retains valuable spatial structure information;
[0115] Specifically, step S7 also includes step S71, and the calculation process is as follows:
[0116] y s1 =DWCov 3×3 (PWConv(y featne ))
[0117] y s2 =PWConv(DWConv 5×5 (y s1 ))
[0118] y signal =Gelu(PWConv(y feature ))
[0119] z out =y signal ☉y s2 ,
[0120] Among them, y feature represents input, PWConv represents point-by-point convolution, DWConv 3×3 Denotes a depth-separable convolution with a convolution kernel of 3×3, DWConv 5×5 represents a depth-wise separable convolution with a convolution kernel of 5×5, Gelu represents a Gaussian error linear function, and ⊙ represents a Hadamard product.
[0121] S8, uses a fully connected feedforward neural network to build a classification head to predict category information; uses an adaptive moment estimation optimizer to iteratively update model parameters, trains and optimizes the selective state space model on the training set, and obtains an improved selective space state model.
[0122] The performance of the improved selective spatial state model is tested on the test set to verify the effectiveness of the improved selective spatial state model. In this embodiment, the overall accuracy (OA), average accuracy (AA) and Kappa coefficient (K) are used to measure the classification accuracy, and compared with the mainstream benchmark models SSRN, DBDA, SSFTT, and SSTN. The IP data set uses 10% as the training set and 90% as the test set, and the LK data set uses 1% as the training set and the remaining 99% as the test set.
[0123] The SSRN method comes from Zhong, Z., Li, J., Luo, Z., Chapman, M., 2018b. Spectral–Spatial Residual Network for Hyperspectral Image Classification: A3-DDeepLearning Framework. IEEE Transactions on Geoscience and Remote Sensing56, 847-858.
[0124] The SSTN method comes from Zhong, Z., Li, Y., Ma, L., Li, J., Zheng, W.-S., 2022. Spectral–Spatial Transformer Network for Hyperspectral Image Classification: AFactorized Architecture Search Framework. IEEE Transactions on Geoscience and Remote Sensing 60, 1-15.
[0125] The SSFTT method comes from Sun, L., Zhao, G., Zheng, Y., Wu, Z., 2022. Spectral–SpatialFeature Tokenization Transformer for Hyperspectral Image Classification. IEEE Transactions on Geoscience and Remote Sensing 60, 1-14.
[0126] The DBDA method comes from Li, R., Zheng, S., Duan, C., Yang, Y., Wang, X., 2020. Classification of Hyperspectral Image Based on Double-Branch Dual-Attention Mechanism Network, Remote Sensing. MDPI AG, p.582.
[0127] The classification results on the IP and LK datasets are shown in Tables 1 and 2. It can be seen that on the two datasets, compared with different mainstream classification models, the improved selective spatial state model proposed in the present invention achieves the best performance in the three indicators of classification accuracy OA, AA and κ.
[0128] Table 1 IP dataset
[0129]
[0130] Table 2 LK dataset
[0131]
[0132]
[0133] On the other hand, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the hyperspectral classification method based on the local enhanced selective state space model when executing the program.
[0134] In yet another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the hyperspectral classification method based on the local enhanced selective state space model are implemented.
[0135] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the hyperspectral classification method based on the local enhanced selective state space model are implemented.
[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0137] In particular, according to some embodiments of the present disclosure, the process described above can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.
[0138] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a mission data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated mission data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0139] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperTextTransferProtocol), and may be interconnected with any form or medium of digital task data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an adhoc peer-to-peer network), as well as any currently known or future developed network.
[0140] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: in response to detecting a query operation on a production collaboration document in a switch production line management application, determines the network connection status of the switch production line management application; in response to determining that the network connection status of the switch production line management application represents an offline state, replaces the web page entry information corresponding to the production collaboration document with target entry file information, and loads target web page resource information to display the web page of the production collaboration document offline in the switch production line management application, wherein the target entry file information is file information of a pre-downloaded entry file corresponding to the web page of the production collaboration document, and the target web page resource information is locally stored resource information corresponding to the web page; in response to determining that the network connection status of the switch production line management application represents an online state, and the web page resource information corresponding to the production collaboration document is not stored locally, downloads the web page resource information of the web page from the production line document server, wherein the web page resource information includes an entry file and resource information; according to the web page resource information, displays the web page of the production collaboration document in the switch production line management application, and stores the web page resource information in a local database.
[0141] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages or a combination thereof, including product-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0142] The present invention is not limited to what is described in the specification and implementation modes, and therefore additional advantages and modifications can be easily realized by those skilled in the art. Therefore, without departing from the spirit and scope of the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details, representative devices, and illustrative examples shown and described herein.
Claims
1. A hyperspectral classification method based on a locally enhanced selective state-space model, It is characterized in that The following steps are involved: S1, reads the three-dimensional hyperspectral image and the category label corresponding to each pixel; S2, normalize the values in the hyperspectral image and map the data to the interval [0, 1]; S3, taking each pixel of the three-dimensional hyperspectral image as the center point, cropping the neighborhood of the center point with a fixed size into image blocks, and constructing an image block dataset, where the label of the center point pixel is set as the category label; S4, randomly select data from the image block dataset as the training set, and the rest of the data as the test set; S5, train the neural network model on the training set, input the image blocks of the training set into the convolution dimensionality reduction backbone built with point-by-point convolution as the basic unit, and obtain the feature map y∈R after point-by-point convolution dimensionality reduction H×W×D , using a multi-directional local scanning mechanism to enhance the selective state space model, and transform the reduced feature map y∈R H×W×D Divide the image y into multiple local windows p Then, the information between different spectral bands is integrated to obtain the hyperspectral feature map y stack Input selective state space model; S6, use the spectral feature extraction module to extract the feature map y∈R obtained after dimensionality reduction H×W×D The central area of the image block y is obtained center ∈R s×s×D , and then scan bidirectionally in the spectral dimension and combine the selective state space model to obtain the regional image block y center ∈R s×s×D The spatial and spectral characteristics of S7, uses a multi-scale convolutional neural network to fuse spatial and spectral features, removes redundant information, and retains valuable spatial structure information; S8, uses a fully connected feedforward neural network to build a classification head to predict category information; trains and optimizes the selective state space model on the training set.
2. The hyperspectral classification method based on the local enhanced selective state space model as claimed in claim 1, It is characterized in that Step S5 also includes step S51, for an input training set The image block is calculated as follows: Among them, H, W and C represent the length, width and number of bands of the image respectively. Indicates that the output feature map is at the cth position (h, w) out The value of the channel, Indicates that the input feature map is at position (h, w) in The value of the channel, Indicates that the convolution kernel is in the input channel c in and output channel c out The weight between out Less than c in .
3. The hyperspectral classification method based on the local enhanced selective state space model as claimed in claim 2, It is characterized in that Step S5 also includes step S52, the feature map obtained after point-by-point convolution dimensionality reduction is y∈R H×W×D , using the multi-directional local scanning mechanism to enhance the selective state space model, the feature map y∈R after dimensionality reduction H×W×D Divide into multiple local window images y p , the calculation process is: Partition means dividing the feature map into windows of size h×w in the dimensions of height and width, n h and n w Respectively represent the number of windows in the height and width directions; S53, the local window image y p Rearrange: Among them, Rearrange represents the different local window images y p rearrange; The rearranged local window image y R The internal height and width dimensions are transposed to obtain the transposed local window image Y T : Among them, Transpose means transposing in the height and width dimensions; S54, the transposed local window image y T Flip in the spatial dimension to get the flipped local window image [y RF ,y TF ]: [and RF .and TF ]=Flip([and R ,and T ]), Among them, Flip means flipping in the spatial dimension. S56, flatten and stack all the above local window images, integrate the information between different spectral bands of the local window images, and obtain the hyperspectral feature map y stack : and stack ∈R 4×L×C =Stack(Flatten([and R ,and T ,and RF ,and TF ]), Flatten means flattening and arranging in sequence in the spatial dimension, and Stack means stacking these sequences together to integrate the obtained hyperspectral feature map y stack It has four sequences of local window horizontal, vertical, forward and reverse; S57, hyperspectral feature map stack The four sequences of input selective state space model are processed as shown in the formula: y spatial ∈R L×C =sum(SSM(y stack )), Among them, Sum means adding the four output results, SSM means selective state space model, h′ t =Ah t-1 +By t The calculation formula is: t =Ch′ t , Among them, y t Indicates input y stack The tth value in h t It is the state representation from the input sequence to the output sequence, z represents the output, A is the high-order polynomial mapping operator matrix, B and C are transformed by the input y stack The linear mapping is obtained, and the mapping formula is: B=w b y stack +b b C=w c y stack +b c , Among them, w and b are trainable parameters.
4. The hyperspectral classification method based on the local enhanced selective state space model as claimed in claim 1, It is characterized in that Step S6 also includes step S61, the calculation formula for extracting spectral features is as follows: and flatten ∈R S×D =Flatten(and center ∈R w×w×D ), Among them, Flatten means flattening in the spatial dimension, S = w × w; y ssm ∈R S×D =SSM(y flatten ∈R S×D , Among them, SSM represents the selective spatial state model, which scans bidirectionally in the spectral dimension; y spatial ∈R 1×C =Sigmoid(Average(y ssm )), Among them, Average means taking the average value in the spatial dimension, and the sigmoid function is used to generate the control signal of the spectral dimension. Its calculation formula is: Multiply the attention signal of the spectral channel by the spatial feature to obtain the feature of the fusion of the spectral feature and the spatial feature. The calculation formula is: feature =y spectral ⊙y spatial Here, ⊙ represents the Hadamard product.
5. The hyperspectral classification method based on the local enhanced selective state space model as claimed in claim 1, It is characterized in that Step S7 also includes step S71, and the calculation process is as follows: y s1 =DWConv 3×3 (PWConv(y feature )) y s2 =PWConv(DWConv 5×5 (y s1 )) the signal =Gelu(PWConv(y feature )) z out =y signal ⊙y s2 , Among them, y feature represents input, PWConv represents point-by-point convolution, DWConv 3×3 Denotes a depth-separable convolution with a convolution kernel of 3×3, DWConv 5×5 represents a depth-wise separable convolution with a convolution kernel of 5×5, Gelu represents a Gaussian error linear function, and ⊙ represents a Hadamard product.
Citation Information
Patent Citations
Parallel method for neural network training
CN116596035A
AU2021101584A4
Cited By
Hyperspectral image classification method and system based on spatial-spectral complementary Mama
CN119810554A
