BNN-based hyperspectral image classification method, apparatus and device, and memory

Abstract

The invention relates to a BNN-based hyperspectral image classification method, apparatus and device, and a memory. According to the method, a BNN-based hyperspectral image classification model is designed, excellent classification performance is realized while a CNN model is compressed, and a self-adaptive gradient scaling module based on a multi-scale pyramid compression attention mechanism is used as a full-precision real value bypass branch of a basic BNN network in a model training process. The method is used for flexibly adjusting the gradient in the back propagation process to reduce the gradient error, so that the binary neural network is better optimized without increasing the reasoning overhead, and meanwhile, the representation capability of the network is kept in forward propagation. Compared with an existing binaryzation or quantification method, the method has the advantages that the classification accuracy of the hyperspectral image is remarkably improved, and meanwhile, a high compression ratio is kept.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a BNN-based hyperspectral image classification method, device, equipment and memory. Background Art

[0002] With the rapid development of the Internet of Things and artificial intelligence technologies, hyperspectral remote sensing is increasingly showing the characteristics of low cost, high mobility, integration, intelligence and real-time processing capabilities. At present, in addition to satellite-based hyperspectral remote sensing technology, lightweight and miniaturized hyperspectral remote sensing technology based on small unmanned aerial vehicles (UAVs) has a wide range of application needs and practical value in the fields of battlefield situation awareness, environmental monitoring, security and disaster relief. At the same time, hyperspectral remote sensing is rapidly developing in the direction of low cost, high mobility, integration, intelligence and strong real-time capabilities. Real-time hyperspectral target detection requires simultaneous processing of the acquired data, especially when the computing and storage resources of the application platform are limited. In order to achieve instant decision analysis, the location and response intensity of the target must be distinguished in real time or near real time.

[0003] Whether it is a single-scene hyperspectral image classification method, a cross-scene hyperspectral image classification method, or a hyperspectral image open set recognition method, deep convolutional neural network (CNN) plays a key role as a backbone model for feature extraction. However, the success of CNN in artificial intelligence applications comes at the cost of a large amount of storage resources and extremely high computational complexity. In addition, the improvement of spectral and spatial resolution of hyperspectral images directly leads to a rapid increase in data dimensions, causing the "dimensionality disaster" problem, which further aggravates the computational complexity of the classification model. However, for edge AI computing devices such as ARM (Advanced Reduced Instruction Set Machine) platform, FPGA (Field Programmable Gate Array) platform and SoC (System-on-Chip) on satellites or drones, the deployment and efficient operation of high-precision deep learning models face challenges due to limited computing resources and strict latency requirements. Therefore, the huge storage and computing overheads bring difficulties to the real-time processing of hyperspectral remote sensing target classification, which makes it of great research value to compress and accelerate the deep learning algorithm model for hyperspectral image classification.

[0004] According to existing research results, in hyperspectral image classification models, most of the number of parameters and computational overhead are concentrated in the convolutional layers. If the operation of these layers can be effectively optimized, memory usage and computational requirements will be significantly reduced. Neural network parameter quantization refers to compressing the original network by reducing the number of bits required to store weights and activation values. Binary neural networks (BNNs) are an extreme form of quantization in which weights and activation values ​​are binarized to achieve fast inference and minimize memory consumption. However, in the field of remote sensing, BNN models suitable for fast classification of hyperspectral images are still scarce and need to be improved. Summary of the invention

[0005] Based on this, it is necessary to provide a BNN-based hyperspectral image classification method, device, equipment and memory to address the above technical problems.

[0006] A hyperspectral image classification method based on BNN, the method comprising:

[0007] Acquire hyperspectral images and preprocess them to obtain training samples.

[0008] The pre-built BNN-based hyperspectral image classification model is trained with training samples to obtain a trained BNN-based hyperspectral image classification model; the BNN-based hyperspectral image classification model includes: a basic BNN network and several adaptive gradient scaling modules based on a multi-scale pyramid compression attention mechanism that are only used in the training process; the adaptive gradient scaling module serves as a full-precision real-valued bypass branch of the basic BNN network during the training process, which is used to adjust the gradient during the back-propagation process while maintaining the representation ability of the network in the forward propagation.

[0009] The basic BNN network in the trained BNN-based hyperspectral image classification model is used to classify the preprocessed hyperspectral image to be classified, and the hyperspectral image classification result is obtained.

[0010] A hyperspectral image classification device based on BNN, the device comprising:

[0011] The training sample determination module is used to obtain a hyperspectral image and preprocess the hyperspectral image to obtain a training sample.

[0012] The BNN-based hyperspectral image classification model training module is used to train the pre-built BNN-based hyperspectral image classification model with training samples to obtain a trained BNN-based hyperspectral image classification model; the BNN-based hyperspectral image classification model includes: a basic BNN network and several adaptive gradient scaling modules based on a multi-scale pyramid compression attention mechanism that are only used during the training process; the adaptive gradient scaling module serves as a full-precision real-valued bypass branch of the basic BNN network during the training process, and is used to adjust the gradient during the back-propagation process while maintaining the representation capability of the network during the forward propagation.

[0013] The hyperspectral image classification module is used to classify the pre-processed hyperspectral image to be classified using the basic BNN network in the trained BNN-based hyperspectral image classification model to obtain the hyperspectral image classification result.

[0014] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above methods when executing the computer program.

[0015] A computer readable memory stores a computer program, which implements the steps of any of the above methods when executed by a processor.

[0016] The above-mentioned BNN-based hyperspectral image classification method, device, equipment and memory, the method designs a BNN-based hyperspectral image classification model, achieves excellent classification performance while compressing the CNN model, and uses an adaptive gradient scaling module based on a multi-scale pyramid compression attention mechanism as a full-precision real-valued bypass branch of the basic BNN network during model training, which is used to flexibly adjust the gradient during the back-propagation process to reduce the gradient error, thereby better optimizing the binary neural network without increasing the inference overhead, while maintaining the network's representation ability in the forward propagation. Compared with existing binarization or quantization methods, the method significantly improves the accuracy of hyperspectral image classification while maintaining a high compression ratio. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Schematic diagram of a process of a hyperspectral image classification method based on BNN in one embodiment;

[0018] Figure 2 Schematic diagram of a hyperspectral image classification framework based on BNN in another embodiment;

[0019] Figure 3 Schematic diagram of a binarization convolution block connected to an adaptive gradient scaling module in another embodiment;

[0020] Figure 4Flowchart of the operation of a pyramid compression attention (PSA) module in another embodiment;

[0021] Figure 5 Schematic diagram of a compressed pyramid connection (SPC) module with S=3 in another embodiment;

[0022] Figure 6 A flowchart of the SE module in another embodiment;

[0023] Figure 7 A detailed structural diagram of CR in another embodiment;

[0024] Figure 8 A flowchart of network training and testing based on confidence-aware loss in another embodiment, wherein (a) is standard supervised training, (c) is supervised training using the SuperLoss loss function, and (b) is the testing process;

[0025] Fig. 9 Four structural diagrams of implementing the adaptive gradient scaling module in another embodiment, where (a) is BiTurn, (b) is the method of document 2, (c) is ECA, and (d) is SE;

[0026] Fig.10 is a structural block diagram of a hyperspectral image classification device based on BNN in one embodiment;

[0027] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] In this application, a BNN-based hyperspectral image classification method is proposed. In the hyperspectral image classification task, the hyperspectral image is defined as H∈R h×w×d , where h, w and d represent the spatial height, width and number of spectral bands of the three-dimensional hyperspectral image respectively. The goal of hyperspectral image classification is to obtain the spectral vector x for each pixel. i ∈R d Assign a class label y i ∈{1,2,...,N}. The labeled training dataset is defined as D s ={(x i ,y i ),i=1,2,...,P}, and the unlabeled test data set is denoted as D t ={x i,i=P+1,P+2,...,P+Q}, where P and Q are the number of samples in the training set and the test set respectively. s Learn from x i to y i The mapping relationship is generalized to the test set D t . Convolutional neural networks (CNNs) are particularly good at capturing such mapping relationships. However, due to their large number of parameters and computational requirements, deep convolutional neural networks are difficult to deploy directly on edge computing platforms with limited resources or strict low latency requirements, such as digital signal processors (DSPs) and field programmable gate arrays (FPGAs), which are usually standard processors for airborne or airborne data processing. Binary neural networks (BNNs) provide an effective way to solve the lightweight problem of hyperspectral image classification models through extreme parameter compression ratio and acceleration capabilities. A hyperspectral image classification model based on BNN (referred to as BiTurn) is proposed.

[0030] In one embodiment, Figure 1 As shown, a hyperspectral image classification method based on BNN is provided, and the method includes the following steps:

[0031] Step 100: Acquire a hyperspectral image and preprocess the hyperspectral image to obtain a training sample.

[0032] Step 102: Use training samples to train the pre-built BNN-based hyperspectral image classification model to obtain a trained BNN-based hyperspectral image classification model; the BNN-based hyperspectral image classification model includes: a basic BNN network and several adaptive gradient scaling modules based on a multi-scale pyramid compression attention mechanism that are only used in the training process; the adaptive gradient scaling module serves as a full-precision real-valued bypass branch of the basic BNN network during the training process, and is used to adjust the gradient during the back-propagation process while maintaining the representation capability of the network during the forward propagation.

[0033] Specifically, the hyperspectral image classification model based on BNN is as follows: Figure 2 shown.

[0034] The BNN-based hyperspectral image classification model includes a basic BNN network and several adaptive gradient scaling (AGS) modules based on a multi-scale pyramid compression attention mechanism that are only used during training. The basic BNN network follows the BNN basic network structure of a one-dimensional convolutional neural network constructed in accordance with the literature 1 (Mei S, Chen X, Zhang Y, et al. Accelerating convolutional neural network-based hyperspectral image classification by step activation quantization. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60: 1-12). According to the common operations of BNN in classification tasks, the activation binarization function of the last layer of the network is removed. Instead, a dropout layer with a dropout rate of 0.5 and a full-precision fully connected layer are connected at the end for classification.

[0035] In order to solve the serious gradient error problem of the traditional straight-through estimator during back-propagation and the problem of training falling into suboptimal solutions, an adaptive gradient scaling (AGS) module based on a multi-scale pyramid compression attention mechanism is designed to recalibrate the gradient. Specifically, a data-driven residual connection is added during training. The AGS module consists of multiple layers of networks that are connected before the sign function that binarizes the activations, such as Figure 3 As shown. SPC stands for compressed pyramid connection convolutional neural network structure, GAP stands for global average pooling layer, and FC stands for fully connected layer. In order to maintain the consistency of using only sign functions in the forward calculation stage, the output of this module is forced to be positive. Through this operation, it is equivalent to multiplying each gradient unit by an adaptive positive scaling factor in the backpropagation stage without affecting the forward calculation process. More importantly, the AGS module only exists during training and is removed in the inference stage, thus avoiding the increase of additional parameters and calculations.

[0036] During the training process, the BNN-based hyperspectral image classification model adds several full-precision real-valued bypass branches to the basic BNN network (a full-precision adaptive gradient scaling module is embedded on the binarization module of the basic BNN network) to provide bypass full-precision gradient branches for the back-propagation process, which are used to adjust the gradient during the back-propagation process while maintaining the representation ability of the network in the forward propagation. After the model training is completed, the adaptive gradient scaling module will be removed.

[0037] Step 104: using the trained basic BNN network in the BNN-based hyperspectral image classification model to classify the preprocessed hyperspectral image to be classified, to obtain a hyperspectral image classification result.

[0038] In the above-mentioned BNN-based hyperspectral image classification method, the method designs a BNN-based hyperspectral image classification model, achieves excellent classification performance while compressing the CNN model, and uses an adaptive gradient scaling module based on a multi-scale pyramid compression attention mechanism as a full-precision real-valued bypass branch of the basic BNN network during model training, which is used to flexibly adjust the gradient during backpropagation to reduce the gradient error, thereby better optimizing the binary neural network without increasing the inference overhead, while maintaining the network's representation ability in forward propagation. Compared with existing binarization or quantization methods, the hyperspectral image classification accuracy of this method is significantly improved while maintaining a high compression ratio.

[0039] In one embodiment, the basic BNN network includes a convolution module, seven binarization modules, a dropout layer, and a full-precision fully connected layer; wherein the convolution module includes a convolution layer and a batch normalization layer; the binarization module includes a convolution layer for performing a binarization operation and a batch normalization layer; step 102 includes: inputting the training sample into the convolution module to obtain a convolution feature; inputting the convolution feature into the first adaptive gradient scaling module to obtain a first adaptive feature; performing an element-by-element multiplication operation on the first adaptive feature and the convolution feature and inputting the result into the first binarization module, and then fusing the obtained feature with the convolution feature to obtain the first The first binary feature is input into the second adaptive gradient scaling module to obtain the second adaptive feature; the second adaptive feature and the first binary feature are multiplied element by element and then input into the second binary module, and then the obtained feature is fused with the first binary feature to output the second binary feature; the second binary feature is input into the third adaptive gradient scaling module to obtain the third adaptive feature; the third adaptive feature and the second binary feature are multiplied element by element and then input into the third binary module, the obtained feature is input into the fourth binary module, and then the obtained feature is fused with the second binary feature. The third binary feature is fused and the third binary feature is output; the third binary feature is input into the fourth adaptive gradient scaling module to obtain the fourth adaptive feature; the fourth adaptive feature and the third binary feature are element-wise multiplied and input into the fifth binary module, and then the obtained feature is fused with the third binary feature to output the fourth binary feature; the fourth binary feature is input into the fifth adaptive gradient scaling module to obtain the fifth adaptive feature; the fifth adaptive feature and the fourth binary feature are element-wise multiplied and input into the sixth binary module, and then the obtained feature is fused with the fourth binary feature to output the fifth binary feature. The fifth binary feature is input into the sixth adaptive gradient scaling module to obtain the sixth adaptive feature; the sixth adaptive feature and the fifth binary feature are element-wise multiplied and input into the seventh binary module, and then the obtained feature is fused with the fifth binary feature to output the sixth binary feature; the sixth binary feature is processed by the dropout layer and input into the full-precision fully connected layer for classification to obtain the hyperspectral image classification prediction result; the BNN-based hyperspectral image classification model is reversely trained according to the hyperspectral image classification prediction result and the training samples to obtain the trained BNN-based hyperspectral image classification model.

[0040] Specifically, the specific structure of the basic BNN network is as follows Figure 2As shown in (a), the basic BNN network consists of eight convolutional layers. Except for the first layer, all convolutional layers perform binarization operations, and each convolutional layer is connected to a normalization layer. Referring to the model proposed in Reference 1, the input and output channels of the third binary convolutional layer are not equal, so in BiTurn, the corresponding residual connection skips two stacked binarization (SBB) modules.

[0041] In one embodiment, the adaptive gradient scaling module includes: a multi-scale pyramid squeezing attention mechanism, a global average pooling layer, a first fully connected layer, a ReLU activation function, a second fully connected layer and a Sigmoid activation function; the multi-scale pyramid squeezing attention mechanism includes an SPC module with a multi-scale pyramid convolution structure, and a compression and excitation weight module; the convolution feature is input into the first adaptive gradient scaling module to obtain a first adaptive feature, including: inputting the convolution feature into the SPC module, first uniformly dividing the convolution feature in the channel dimension, and then processing all the division results respectively using convolution layers with convolution kernels of different scales, splicing all the processing results in the channel dimension, and finally processing the splicing results using the compression and excitation weight module to obtain an attention weight vector; the attention weight vector is processed by the global average pooling layer, the first fully connected layer, the ReLU activation function, the second fully connected layer and the Sigmoid activation function in sequence to obtain the first adaptive feature.

[0042] Specifically, Figure 4 The figure shows the operation flow chart of the pyramid compression attention (PSA) module in the adaptive gradient scaling module. The implementation of PSA mainly includes two steps. The full-precision feature vector generated by the front-end network is first input into the SPC module with a multi-scale pyramid convolution structure for multi-scale spectral feature learning. Then, the multi-scale feature vector is output to the compression and excitation (SE) weight module, in which the channel-level attention vector is calculated, that is, the attention to feature maps of different scales is extracted. The calculation process of the key component SPC module is shown in Figure 5 As shown in Figure 2. In SPC, “compression” refers to evenly dividing the feature maps in the channel dimension, while “connection” refers to connecting the feature maps in the channel dimension.

[0043] In one embodiment, the mathematical expression of the SPC module is:

[0044]

[0045] Z i =SE(F i ) (2)

[0046] F=Cat([F0,F1,...,F S-1 ]) (3)

[0047]

[0048] Where Z is the multi-scale attention weight vector, represents the cascade operation, K represents the size of the convolution kernel, G is the size of the convolution group, S is the number of convolution groups, and C is divisible by S. C is the channel dimension of the feature map input to the SPC module, and Z i ∈R C ′×1×1 is the attention weight vector, SE() represents the SE module, cat() represents the channel dimension concatenation operation, and F i ∈R C×H×W Represents feature maps of different scales, F∈R C×H×W is the overall multi-scale feature map.

[0049] In one embodiment, in the SE module, the attention weight of the cth channel is:

[0050] w c =σ(W1δ(W0(g c ))) (6)

[0051]

[0052] Among them, w c represents the attention weight of the cth channel, g c represents the result of global average pooling of the heat certificate in the input SE module, δ represents the ReLU operation, and represents the fully connected calculation (these two fully connected layers effectively integrate the linear information between channels and promote the interaction between high-dimensional information and low-dimensional information), σ represents the activation function (preferably, the Sigmoid function is usually used as the activation function to assign weights to channels, thereby more effectively extracting feature information), x c (i,j) represents the element in the i-th row and j-th column of the c-th channel.

[0053] Specifically, in the SPC module, the input channel dimension of each branch is C. In order to process the input tensor with convolution kernels of different scales without increasing the computational cost, multiple group convolutions are introduced to extract spectral information from the input feature map. This means that in the pyramid structure, multi-scale convolution kernels are used to generate feature maps with different spectral resolutions. This approach makes it possible to obtain richer spectral information from the input tensor. In addition, each group of convolutions independently learns spectral information of different scales and establishes local cross-channel interactions in a parallel manner, thereby compressing the channel dimension of the input feature vector. Finally, the feature maps F generated at different scales are iThey have the same channel dimension C = C / S, where i = 0, 1, ..., S-1, S is the number of convolution groups, and C is divisible by S. In addition, without increasing the number of parameters, SPC adopts a grouped convolution group size determination scheme as shown in formula (5), which represents the relationship between the multi-scale convolution kernel size and the convolution group size.

[0054] Therefore, the calculation of the generated multi-scale feature map is shown in formula (4). In formula (4), the size of the i-th convolution kernel is k i =2×(i+1)+1, the size of the i-th convolution group is G i . F i ∈R C×H×W Represents feature maps of different scales, where H and W are the height and width of its spatial dimension, respectively. In the experiments of this paper, taking the PU dataset as an example, the number of feature map channels input to the adaptive gradient scaling module is 24, and the size is 24×1×103. Therefore, the number of convolution groups in SPC is set to S=3, and G0=1, G1=4, and G2=8 are obtained. The convolution kernel sizes in the three convolution groups are 1×3, 1×5, and 1×7, respectively. The overall multi-scale feature map can be obtained by connection, as shown in formula (3). In formula (3), F i ∈R C×H×W is the overall multi-scale feature map obtained through the above process. Next, in PSA, the Squeeze and Excite (SE) attention mechanism is used to obtain the attention vector from the multi-scale feature map to obtain the weight information of feature maps of different scales, which is recorded as SE weight. The calculation formula of the attention weight vector is shown in formula (2).

[0055] The specific operation process of the SE module is as follows: Figure 6 As shown. This channel attention mechanism enables the network to selectively weight the training contribution of each channel, thereby producing more effective information output. Let X∈R C×H×W Represents the input feature map, where H, W, and C are the height, width, and number of channels of the input, respectively. The SE module consists of two parts: compression and excitation, which are used to encode global information and adaptively adjust the relationship between channels, respectively. Channel-level attention weights can be generated by global average pooling (GAP) to embed global spectral information into channel attention calculation. The operation is shown in formula (7).

[0056] In the SE module, the attention weight of the cth channel is calculated as shown in formula (6).

[0057] In general, the PSA module is able to integrate multi-scale spectral information and realize information interaction across channel dimensions, thereby generating pixel-level attention for feature vectors. In addition, in order to realize the interaction between attention information, the PSA module integrates multi-scale channel attention vectors in a cascade manner without changing the original channel attention vectors.

[0058] In formula (1), Z i It is from F i The attention value obtained in , and Z is the multi-scale attention weight vector, that is, the scaling factor E(a r θ E ).

[0059] Then, before the activation values ​​are fed into the sign function, they are scaled by the gradient scaling module with a scaling factor E(a r θ E ) are multiplied element by element to obtain the scaled and adjusted activation value. The whole process is derived as follows:

[0060]

[0061] In the formula, ⊙ represents element-by-element multiplication, E is the adaptive gradient adjustment module, and its parameter is denoted by θ E During the back-propagation phase, the AGS module affects the propagated gradients. When the gradients start to back-propagate from the end of the network, the activation gradients of the current layer are adjusted first, and then the activation and weight gradients of the previous shallower layers are adjusted in turn, as shown below:

[0062]

[0063] Inspired by reference 2 (Wei W, Song C, Zhang L, et al. Lightweighted hyperspectralimage classification network by progressive bi-quantization. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 1-14), the PSA-based adaptive gradient scaling module proposed in this application is different from the data-driven channel rescaling (DDCR) module in the Real2Bin method. In the CR method, during the training and inference stages, the binary activations after binary convolution are scaled at the channel level using unbinarized full-precision activations. The specific structure of CR is as follows Figure 7 As shown, the adjustment process of DDCR is derived as follows:

[0064]

[0065] Among them, E CR is the CR module, and the parameters of this module are recorded as By comparing formula (8) and formula (10), it can be seen that the convolution results of the two methods are different. Specifically, the proposed BNN model does not change the result of the binary convolution during the forward calculation process, while Real2Bin adjusts the result of the binary convolution. More importantly, thanks to the continuous results of the forward calculation, the AGS module in the proposed BNN model can be removed in the inference phase without adding additional parameters and computational overhead. However, in order to avoid the discontinuity of forward propagation between the training and inference phases, Real2Bin still needs to retain the CR module in the inference phase.

[0066] In addition, compared with the gradient scaling module similar to ECA-Net used in Reference 1, which only uses one-dimensional convolutional layers to design the channel attention mechanism, the multi-scale pyramid squeezed attention mechanism proposed in this application effectively integrates local and global attention by extracting multi-scale spectral feature vectors across channel dimensions to generate channel attention weights. This method realizes cross-dimensional information interaction and establishes long-range channel dependencies.

[0067] In one of the embodiments, the BNN-based hyperspectral image classification model also includes a dynamic curriculum learning strategy based on a confidence-aware loss function Superloss; the dynamic curriculum learning strategy based on the confidence-aware loss function Superloss means that Superloss can be directly inserted into the classification task loss function, and the confidence parameter of the Superloss loss function is its convergence value at the limit. The confidence parameter of the Superloss loss function is updated with the state of the task loss during the sample training process. During the progressive quantization training process, there is no need to modify the learning process of the network; according to the dynamic curriculum learning method, the loss of each sample is monitored during the training process to determine its importance in the training process.

[0068] Specifically, the curriculum learning strategy is a learning paradigm inspired by the cognitive processes of humans and animals. The paradigm is based on the intuitive observation that the human learning process naturally starts with simple conceptual problems and gradually transitions to more complex tasks. When the curriculum learning strategy is applied to the field of machine learning, it advocates that the model should first focus on learning simple samples and then move on to complex samples. During the training process, the training difficulty or importance of each sample needs to be considered, which is usually achieved by calculating the weight of each sample in the training iteration. This makes the samples that are easy to classify have a larger training weight in the early stages of training, while conversely, the samples that are difficult to classify have a smaller weight at the beginning of training.

[0069] Hyperspectral images are composed of a large number of pixels with different label categories, and the classification difficulty of each pixel sample varies. Traditional hyperspectral image classification algorithms only input all training samples into the network for training without considering the classification difficulty of each pixel sample, which may lead to unsatisfactory training results of the model. Considering that the classification difficulty of each pixel sample in the hyperspectral image varies, treating each sample equally during the training process may cause the optimization of the BNN model to deviate from the optimal direction. A dynamic curriculum learning progressive quantization strategy based on the confidence-aware loss function Superloss is proposed, such as Figure 2 As shown in (c). Based on this strategy, the model can monitor the classification error loss of each sample during training and determine its training weight, that is, its contribution to classification, thereby following a dynamic curriculum learning training mode from simple to difficult. In addition, Superloss can be directly inserted into the task loss function. Unlike other confidence-aware losses that need to wait for the confidence parameters to converge, the Superloss loss function directly uses its limit convergence value as the confidence parameter. There is no need to learn the confidence parameters sample by sample, but it is updated according to the task loss status of the sample, without incurring computational delays. The following sections will explain in detail the proposed adaptive gradient scaling module and the robust dynamic curriculum learning progressive quantization strategy.

[0070] In one embodiment, the confidence-aware loss function Superloss is:

[0071]

[0072] in, Total loss, Represents the convergence value of the confidence parameter of the Superloss loss function at the limit; l i represents the loss function for the classification task, represents the confidence loss function; λ is the regularization parameter; τ is the ideal threshold used to distinguish easy-to-classify samples from hard-to-classify samples according to their respective losses, F(x i θ B ) represents the network's response to sample x i Corresponding prediction, y i Represents sample x i The corresponding truth value of i ≥0 represents a learnable confidence parameter, θ B is the network parameter.

[0073] Specifically, in traditional binary neural network model optimization, all training samples are assigned the same importance level, which means that each pixel sample has the same loss weight term. The cross entropy loss of each sample is expressed as The model training optimization process minimizes the cross entropy loss as follows:

[0074]

[0075] in, represents the training dataset, which consists of C classes. F is a network with parameters θ B Binary neural network. j (x i θ B ) means to convert pixel sample x i The predicted probability of being classified as class j. i,j Represents pixel sample x i The true label of is the one-hot encoded vector of category j.

[0076] like Figure 8 As shown in Figure 2, we propose a dynamic curriculum learning strategy based on the confidence-aware loss function Superloss for progressive binarization training. The confidence-aware loss function can jointly learn network parameters and sample training weights (confidence). Compared with the traditional loss function In contrast, the confidence-aware loss function Added a learnable confidence parameter σ i ≥0 as input, which is consistent with each sample x i Related, and represents the network's response to sample x i Corresponding prediction F(x i θ B ) confidence and reliability. In short, for training samples with lower confidence, the gradient update of model parameters will be reduced, which is equivalent to reducing the training weights of samples with lower confidence. The confidence loss function used is The expression is as follows:

[0077]

[0078] Here, λ is the regularization parameter and τ is the ideal threshold used to distinguish between easy and hard-to-classify samples based on their respective losses.

[0079] In the embodiment, for the convenience of expression, τ is estimated as the moving average of the input loss during the training process. Abbreviated as Unlike general dynamic curriculum learning strategies that need to wait for the training of confidence parameters to converge, Superloss directly uses the converged value of the confidence parameter at the limit, and this converged value is only determined by the input loss Determine, as shown in formula (12).

[0080] Therefore, in Superloss, the confidence parameter does not need to be learned sample by sample, but is updated with the status of the training sample. In addition, Superloss has only one input parameter, which can be easily added to the original task loss without changing other training operations. Backpropagation starts from Superloss. The expression of Superloss is shown in formula (11).

[0081] like Figure 8 As shown in Figure 1, (a) is the standard supervised training, (c) is the supervised training using the SuperLoss loss function, and (b) is the testing process, which is the same as the standard supervised training method. The optimal parameters in formula (11) are has a closed form solution, and the loss is constrained to avoid infinite values, as shown in Equation (17) and Equation (18).

[0082]

[0083]

[0084] Where W represents the Lambert W function and λ is the regularization parameter.

[0085] During the back-propagation process, τ and All from l i The calculated value can be regarded as a constant. According to the analysis of Castells et al., under different λ values, the function image of Superloss is based on the input loss l i The variable meets expectations. This function increases the contribution of easy-to-classify samples to training (when l i <τ), while significantly reducing the input loss of difficult samples. When the regularization parameter tends to infinity, the optimal confidence parameter is close to 1. In this case, Superloss is calculated as follows:

[0086]

[0087] In the hyperspectral image classification task, Superloss is applied to the calculation of cross entropy loss as follows:

[0088]

[0089] In the experimental settings of different datasets, a fixed threshold τ = log C is used, where C is the number of categories of pixel samples in the dataset. Therefore, τ represents the cross entropy of the uniform distribution, which approximates the natural boundary between correct predictions and incorrect predictions. In summary, in the dynamic curriculum learning scheme adopted in this paper, the confidence parameter of Superloss does not need to be learned sample by sample or wait for convergence. Instead, it is updated with the state of the task loss during the sample training process and has an optimal closed-form solution. Therefore, in the proposed progressive quantization training process, there is no need to modify the learning process of the network. A new loss function Superloss is simply inserted on top of the original task loss. According to the dynamic curriculum learning method, the loss of each sample is monitored during training to determine its importance in the training process.

[0090] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0091] In a verification example, to demonstrate that BiTurn can improve classification performance while maintaining the compression ratio, classification experiments are conducted on five benchmark hyperspectral image datasets, including the Pavia University dataset, the Dioni dataset, the WHU-Hi-LongKou dataset, the WHU-Hi-HongHu dataset, and the WHU-Hi-HanChuan dataset.

[0092] The Pavia University (PU) dataset is a hyperspectral image captured over the University of Pavia campus in Italy using the Reflection Optical System Imaging Spectrometer (ROSIS-03). The dataset contains 103 spectral bands with wavelengths ranging from 0.43 to 0.86 microns and a spatial resolution of 1.3 meters. The land cover features include large and small strip-like areas, the image has a spatial size of 610×340, contains 9 land cover categories, and has a total of 42,776 labeled pixel samples.

[0093] The Dioni dataset is part of the "HyRANK Hyperspectral Satellite Dataset I", which was released in 2018. The dataset contains 176 spectral bands, with a spatial size of 250×1376 and a spatial resolution of 30 meters. The dataset includes 12 types of scattered land cover targets.

[0094] The WHU-Hi-LongKou dataset was captured on July 17, 2018, from 13:49 to 14:37 in Longkou Town, Hubei Province, China. The imaging platform used was a DJI Matrice600Pro drone equipped with an 8mm focal length Headwall Nano-Hyperspec imaging sensor. During data collection, the weather was clear and cloudless, the temperature was about 36°C, and the relative humidity was about 65%. The drone flew at an altitude of 500 meters and the image size was 550×400 pixels. The spectral wavelength range is 400-1000 nanometers, contains 270 bands, and the spatial resolution is about 0.463 meters.

[0095] The WHU-Hi-HongHu dataset was captured from 16:23 to 17:37 on November 20, 2017 in Honghu City, Hubei Province, China. The imaging platform used was a DJI Matrice 600Pro drone equipped with a Headwall Nano-Hyperspec ultra-micro hyperspectral imaging sensor with a focal length of 17mm. During data collection, the weather was cloudy, the temperature was about 8°C, and the relative humidity was about 55%. The drone flew at an altitude of 100 meters, and the spatial size of the image was 940×475 pixels. The spectral wavelength range is 400-1000 nanometers, contains 270 bands, and the spatial resolution is about 0.043 meters.

[0096] The WHU-Hi-HanChuan dataset was captured on June 17, 2016, from 17:57 to 18:46 in Hanchuan, Hubei Province, China. The imaging platform used was a Leica AibotX6 V1 drone equipped with a 17mm focal length Headwall Nano-Hyperspec imaging sensor. During data collection, the weather was clear and cloudless, the temperature was about 30°C, and the relative humidity was about 70%. The drone flew at an altitude of 250 meters and the image size was 1217×303 pixels. The spectral wavelength range is 400-1000 nanometers, contains 274 bands, and the spatial resolution is about 0.109 meters. It is worth noting that the WHU-Hi-HanChuan dataset was collected in the afternoon at a lower solar altitude angle, resulting in many areas in the image being covered by shadows.

[0097] (1) Experimental setup

[0098] According to the experimental methods described in references (1) and (2), in the data preprocessing stage, the value of each pixel sample is calculated as the average of the 9×9 neighborhood with the pixel as the geometric center to capture spatial information. All experiments were implemented using the Pytorch 2.0.0 deep learning framework on a hardware platform equipped with CUDA 12.1 and 24G memory Nvidia TITAN RTX. The model training used the Adam optimizer with a minimum batch size of 64 and the training was performed for 200 cycles. The initial learning rate was set to 1e-2 and decayed to 0.1 at the 100th and 150th cycles. In addition, τ = log(C), the hyperparameter λ = 1, and the number of convolution groups S = 3 were set. All experiments were run ten times, using different random seeds to average the results, thereby eliminating the impact of random initialization on classification performance. In the experimental evaluation of this paper, three classification performance indicators were used, including the overall accuracy (OA) of all categories, the average accuracy (AA) of each category, and the Kappa coefficient.

[0099] (2) Experimental results

[0100] 1) Comparative analysis experiment

[0101] This embodiment conducts comparative analysis experiments on five data sets to compare the classification performance of BiTurn with the baseline BNN method Bi-Real Net, the method in Reference 1, the method in Reference 2, and the corresponding full-precision network. Table 1 lists the experimental results of the PU data set, and Tables 2 to 5 give the results of the Dioni and WHU-Hi series data sets.

[0102] Table 1. Classification performance of different methods on PU dataset

[0103]

[0104] Table 2. Classification performance of different methods on the Dioni dataset

[0105]

[0106]

[0107] Table 3. Classification performance of different methods on the WHU-Hi-LongKou dataset

[0108]

[0109] Table 4. Classification performance of different methods on the WHU-Hi-HongHu dataset

[0110]

[0111]

[0112] Table 5. Classification performance of different methods on the WHU-Hi-HanChuan dataset

[0113]

[0114] By observing the data in the table, it can be seen that BiTurn achieves the best classification performance on all datasets compared with the three advanced 454-parameter quantization methods for hyperspectral image classification. Specifically, compared with the baseline method Bi-RealNet, BiTurn's overall accuracy on the PU, Dioni, WHU-Hi-LongKou, WHU-Hi-HongHu, and WHU-Hi-HanChuan datasets is improved by 18.25%, 6.1%, 7.05%, 3.96%, and 7.51%, respectively. More importantly, the method in Reference 1 uses a 1-bit weight and 2-bit activation quantization strategy, and BiTurn's overall classification accuracy on the PU dataset is 13.64% higher than that of the method in Reference 1, and there is also a significant performance improvement on the other four datasets. In addition, compared with the current state-of-the-art hyperspectral image classification algorithm based on binary neural networks (the method in Reference 2), the overall classification accuracy of BiTurn on the PU, Dioni, WHU-Hi-LongKou, WHU-Hi-HongHu, and WHU-Hi-HanChuan datasets is improved by 4.16%, 2.33%, 1.85%, 1.1%, and 2.99%, respectively. In addition, from the data in Tables 1, 4, and 5, it can be seen that the classification results of BiTurn on the PU, WHU-Hi-LongKou, and WHU-Hi-HanChuan datasets are better than the corresponding full-precision networks. This suggests that the quantization method may play the role of a regularizer, suppressing the overfitting of the model. Specifically, due to the insensitivity of binary models to small perturbations in the input, they are better able to handle noise and uncertainty in the data, enabling binary models to better generalize to new samples and maintain high classification performance in the presence of noise or limited data, showing better robustness.

[0115] From the classification experimental results of all comparison methods (True label map, Bi-Real Net, method in reference 1, method in reference 2, Full-precision network and BiTurn (this method)) on the PU, Dioni, WHU-Hi-LongKou, WHU-Hi-HongHu and WHU-Hi-HanChuan datasets, it can be seen that BiTurn maintains the classification accuracy of the corresponding full-precision network model as much as possible, surpasses it on some datasets, and shows stronger robustness.

[0116] Table 6. Comparison of memory usage and computational complexity of different methods in PU dataset experiments

[0117]

[0118] Table 6 compares the memory usage and floating-point operations (FLOPs) of different methods in the inference phase. W / A represents the bit width of weights and activations, and multiples of (1 / 1) represent the number of parallel branches. Lower memory usage and FLOPs mean reduced computational overhead on resource-constrained or low-latency platforms, and the best results are shown in bold. In addition, the corresponding classification performance comparison is presented in Table 7. According to the calculation method of reference 3 (Martinez B, Yang J, Bulat A, et al. Training binary neural networks with real-to-binary convo-lutions.arXiv preprint arXiv:2003.11535,2020) and reference 2, the memory usage indicator is the product of the number of parameters and the bit width, and the FLOPs calculation formula for the quantized convolution operator is ((c in ×c out ×k×l out ×b a ×b w ) / 64), where c out is the number of channels of the output feature map, c in is the number of channels of the input feature map, k is the kernel size of the one-dimensional convolution, l out is the length of the output feature map, b a is the activation bit width, b w is the weight quantization bit width.

[0119] Table 7. Comparison of average classification performance of different methods on PU dataset

[0120]

[0121] Among all the compared methods, Real2Bin uses a full-precision data-driven channel rescaling module similar to the attention mechanism, which adds additional computation and memory usage. The parallel binary convolution in GroupNet, as well as the 2-bit activation in the method of reference 1 and the LSQ method also increase computation and memory overhead. Although BiTurn also uses a full-precision adaptive gradient scaling module during network training, this module can be removed during the inference phase without affecting classification accuracy. Therefore, similar to the method of reference 2, the computation and memory usage of BiTurn during the inference phase are the same as those of the baseline method Bi-Real Net. In general, BiTurn improves the classification accuracy of the binary neural network model while maintaining the compression ratio.

[0122] (3) Ablation analysis experiment

[0123] To analyze the effectiveness of each component in BiTurn, we explored the impact of 508 individual modules on the classification results in experiments on five datasets. As shown in Table 8, BL represents the baseline BNN model Bi-Real Net, AGS-PSA is an adaptive gradient scaling module based on multi-scale pyramid squeezing and incentive attention mechanism, Superloss is a confidence-based loss function, and the best results in the table are marked in bold. The use of the AGS-PSA module and the robust dynamic curriculum learning strategy based on Superloss both help to improve the classification performance of the BNN model. Specifically, for the Dioni dataset, the dynamic curriculum learning strategy improves the overall accuracy (OA) by 5.71%, the adaptive gradient scaling module improves OA by 2.12%, and the combination of the two components improves OA by 6.01%. For the WHU-Hi-HanChuan dataset, the dynamic curriculum learning strategy improves OA by 6.17%, the adaptive gradient adjustment module improves OA by 1.06%, and the combined effect of the two components improves OA by 7.51%. This demonstrates the effectiveness of each module and shows how they enhance each other and work synergistically to improve the classification performance of the BNN model.

[0124] Table 8. Ablation analysis experimental results of BiTurn components

[0125]

[0126]

[0127] (4) Gradient Scaling Module Analysis Experiment

[0128] There are many ways to implement the gradient adaptive scaling module structure. For example, SENet (denoted as SE) performs an attention mechanism in the channel dimension; ECA-Net (denoted as ECA) performs an efficient channel attention mechanism in the spectral band dimension. The method in document 2 also follows the design idea of ​​ECA-Net, but removes the matrix transposition operations at the beginning and end. In addition, this application proposes an adaptive gradient adjustment module based on the pyramid squeeze-and-excitation attention mechanism, which includes the attention module of SENet. Fig. 9 The attention mechanism designs of BiTurn, the method in Reference 2, ECA, and SE are shown. In the figure, GAP represents the global average pooling operation, Trans represents the matrix transposition operation, and element-by-element multiplication represents element-by-element multiplication.

[0129] The experimental results of the four connection methods on the datasets PU, Dioni, WHU-Hi-LongKou, WHU-Hi-HongHu and WHU-Hi-HanChuan are presented in Table 9, with the best results highlighted in bold. It can be observed that the connection method used by BiTurn achieves the best classification performance on all five datasets. The method of Reference 2 and ECA achieved the second best classification results, while the SENet connection method performed the worst. It is speculated that unlike SENet, the ECA-Net connection method avoids the dimensionality reduction process when generating attention, thereby reducing the loss of classification accuracy. In addition, the pyramid squeeze-and-excitation attention mechanism is able to obtain multi-scale spectral feature maps in the channel dimension, learning more effective feature representations for hyperspectral images.

[0130] Table 9. Experimental results of different implementation methods of AGS module on several datasets

[0131]

[0132]

[0133] (5) Parameter analysis experiment

[0134] This example discusses the effect of the regularization parameter λ on the classification performance in Superloss. In the experiment of the CIFAR-10 dataset, λ is set to 1, while in the CIFAR-100 dataset, λ is set to 0.25. Since the number of sample categories in the hyperspectral image is small, the experiment in this example also sets λ to 1 by default.

[0135] The classification performance under two λ value scenarios was compared and the results are shown in Table 10. When λ is set to 1, although BiTurn achieves the highest average classification performance in ten random seed initialization experiments on five datasets, the classification performance on the Dioni dataset and the WHU-Hi-HanChuan dataset is not stable enough and the standard deviation is large. This also shows that setting λ to a fixed value has certain limitations, and in practical applications, the value of λ should be adjusted appropriately.

[0136] Table 10 Classification performance of the proposed method under different regularization parameter λ settings

[0137]

[0138]

[0139] From the experimental results, we can see that the CNN-based hyperspectral image classification algorithm has achieved excellent performance. However, deep learning models usually face the problem of over-parameterization, which inevitably leads to significant memory consumption and computing resource overhead. This makes it difficult to directly deploy such models on airborne or satellite-borne embedded computing platforms in real-time operating environments with strict latency requirements and limited resources.

[0140] To address this problem, the NNN-based hyperspectral image classification model (BiTurn) proposed in this application achieves excellent classification performance while compressing the CNN model. BiTurn uses an adaptive gradient scaling module based on a multi-scale pyramid squeeze attention mechanism to learn multi-scale spectral features, and flexibly adjusts the gradient during the binary neural network training process to reduce the gradient error, thereby better optimizing the binary neural network without increasing the inference overhead. In addition, considering the fragility of BNN and its sensitivity to initialization, a dynamic curriculum learning strategy based on a robust confidence-aware loss function Superloss is adopted to finely weigh the weights of training samples during training to achieve progressive quantization and enhance model performance. Experimental results show that compared with existing binarization or quantization methods, BiTurn significantly improves the hyperspectral image classification accuracy on five widely used public datasets while maintaining a high compression ratio.

[0141] In one embodiment, Fig.10 As shown, a hyperspectral image classification device based on BNN is provided, comprising: a training sample determination module, a hyperspectral image classification model training module based on BNN and a hyperspectral image classification module, wherein:

[0142] The training sample determination module is used to obtain a hyperspectral image and preprocess the hyperspectral image to obtain a training sample.

[0143] The BNN-based hyperspectral image classification model training module is used to train the pre-built BNN-based hyperspectral image classification model with training samples to obtain a trained BNN-based hyperspectral image classification model; the BNN-based hyperspectral image classification model includes: a basic BNN network and several adaptive gradient scaling modules based on a multi-scale pyramid compression attention mechanism that are only used during the training process; the adaptive gradient scaling module serves as a full-precision real-valued bypass branch of the basic BNN network during the training process, and is used to adjust the gradient during the back-propagation process while maintaining the representation capability of the network during the forward propagation.

[0144] The hyperspectral image classification module is used to classify the pre-processed hyperspectral image to be classified using the basic BNN network in the trained BNN-based hyperspectral image classification model to obtain the hyperspectral image classification result.

[0145] In one of the embodiments, the basic BNN network includes a convolution module, seven binarization modules, a dropout layer, and a full-precision fully connected layer; wherein the convolution module includes a convolution layer and a batch normalization layer; the binarization module includes a convolution layer for performing a binarization operation and a batch normalization layer; the BNN-based hyperspectral image classification model training module is also used to input the training sample into the convolution module to obtain the convolution feature; the convolution feature is input into the first adaptive gradient scaling module to obtain the first adaptive feature; the first adaptive feature and the convolution feature are element-wise multiplied and input into the first binarization module, and then the obtained feature is combined with the convolution feature. Convolution features are fused to obtain the first binary feature; the first binary feature is input into the second adaptive gradient scaling module to obtain the second adaptive feature; the second adaptive feature and the first binary feature are multiplied element by element and input into the second binary module, and then the obtained feature is fused with the first binary feature to output the second binary feature; the second binary feature is input into the third adaptive gradient scaling module to obtain the third adaptive feature; the third adaptive feature and the second binary feature are multiplied element by element and input into the third binary module, and the obtained feature is input into the fourth binary module, and then the obtained feature is combined with the first binary feature. The second binary feature is fused to output the third binary feature; the third binary feature is input into the fourth adaptive gradient scaling module to obtain the fourth adaptive feature; the fourth adaptive feature and the third binary feature are element-wise multiplied and input into the fifth binary module, and then the obtained feature is fused with the third binary feature to output the fourth binary feature; the fourth binary feature is input into the fifth adaptive gradient scaling module to obtain the fifth adaptive feature; the fifth adaptive feature and the fourth binary feature are element-wise multiplied and input into the sixth binary module, and then the obtained feature is fused with the fourth binary feature to output the fifth. The fifth binary feature is input into the sixth adaptive gradient scaling module to obtain the sixth adaptive feature; the sixth adaptive feature and the fifth binary feature are element-wise multiplied and input into the seventh binary module, and then the obtained feature is fused with the fifth binary feature to output the sixth binary feature; the sixth binary feature is processed by the dropout layer and input into the full-precision fully connected layer for classification to obtain the hyperspectral image classification prediction result; the BNN-based hyperspectral image classification model is reversely trained according to the hyperspectral image classification prediction result and the training samples to obtain the trained BNN-based hyperspectral image classification model.

[0146] In one of the embodiments, the adaptive gradient scaling module includes: a multi-scale pyramid squeezing attention mechanism, a global average pooling layer, a first fully connected layer, a ReLU activation function, a second fully connected layer and a Sigmoid activation function; the multi-scale pyramid squeezing attention mechanism includes an SPC module with a multi-scale pyramid convolution structure, and a compression and excitation weight module; a BNN-based hyperspectral image classification model training module is also used to input convolution features into the SPC module, first uniformly divide the convolution features in the channel dimension, and then process all the division results respectively using convolution layers with convolution kernels of different scales, splice all the processing results in the channel dimension, and finally process the splicing results using the compression and excitation weight module to obtain an attention weight vector; the attention weight vector is processed by the global average pooling layer, the first fully connected layer, the ReLU activation function, the second fully connected layer and the Sigmoid activation function in sequence to obtain a first adaptive feature.

[0147] In one embodiment, the mathematical expressions of the SPC module are shown in formulas (1) to (5).

[0148] In one embodiment, in the SE module, the attention weight of the cth channel is as shown in formulas (6) and (7).

[0149] In one of the embodiments, the BNN-based hyperspectral image classification model in the device also includes a dynamic curriculum learning strategy based on a confidence-aware loss function Superloss; the dynamic curriculum learning strategy based on the confidence-aware loss function Superloss means that Superloss can be directly inserted into the classification task loss function, and the confidence parameter of the Superloss loss function is its convergence value at the limit. The confidence parameter of the Superloss loss function is updated with the state of the task loss during the sample training process. During the progressive quantization training process, there is no need to modify the learning process of the network; according to the dynamic curriculum learning method, the loss of each sample is monitored during the training process to determine its importance in the training process.

[0150] In one embodiment, the confidence-aware loss function Superloss is shown in formulas (11) to (14).

[0151] The specific definition of the hyperspectral image classification device based on BNN can be found in the definition of the hyperspectral image classification method based on BNN above, which will not be repeated here. Each module in the above-mentioned hyperspectral image classification device based on BNN can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0152] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a hyperspectral image classification method based on BNN is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0153] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0154] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above method embodiment when executing the computer program.

[0155] In one embodiment, a computer readable memory is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0156] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable memory. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0157] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0158] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A hyperspectral image classification method based on BNN, characterized in that: The method comprises: Acquiring a hyperspectral image, and preprocessing the hyperspectral image to obtain a training sample; The training samples are used to train a pre-built BNN-based hyperspectral image classification model to obtain a trained BNN-based hyperspectral image classification model; the BNN-based hyperspectral image classification model includes: a basic BNN network and several adaptive gradient scaling modules based on a multi-scale pyramid compression attention mechanism that are only used in the training process; the adaptive gradient scaling module serves as a full-precision real-valued bypass branch of the basic BNN network during the training process, and is used to adjust the gradient during the back-propagation process while maintaining the representation capability of the network during the forward propagation; The basic BNN network in the trained BNN-based hyperspectral image classification model is used to classify the preprocessed hyperspectral image to be classified, and the hyperspectral image classification result is obtained.

2. The method according to claim 1, characterized in that: The basic BNN network includes a convolution module, seven binarization modules, a dropout layer, and a full-precision fully connected layer; wherein the convolution module includes a convolution layer and a batch normalization layer; the binarization module includes a convolution layer for performing a binarization operation and a batch normalization layer; The training samples are used to train the pre-built BNN-based hyperspectral image classification model to obtain a trained BNN-based hyperspectral image classification model, including: Inputting the training sample into the convolution module to obtain convolution features; Inputting the convolution feature into the first adaptive gradient scaling module to obtain a first adaptive feature; The first adaptive feature and the convolution feature are multiplied element by element and then input into the first binarization module, and then the obtained feature is fused with the convolution feature to obtain a first binarization feature; Inputting the first binary feature into the second adaptive gradient scaling module to obtain a second adaptive feature; The second adaptive feature and the first binary feature are multiplied element by element and then input into the second binary module, and then the obtained feature is merged with the first binary feature to output the second binary feature; Inputting the second binary feature into the third adaptive gradient scaling module to obtain a third adaptive feature; Performing an element-by-element multiplication operation on the third adaptive feature and the second binary feature, and inputting the resultant feature into the third binary module, inputting the resultant feature into the fourth binary module, and then fusing the resultant feature with the second binary feature, and outputting a third binary feature; Inputting the third binary feature into the fourth adaptive gradient scaling module to obtain a fourth adaptive feature; The fourth adaptive feature and the third binary feature are multiplied element by element and then input into the fifth binary module, and then the obtained feature is merged with the third binary feature to output a fourth binary feature; Inputting the fourth binary feature into the fifth adaptive gradient scaling module to obtain a fifth adaptive feature; The fifth adaptive feature and the fourth binary feature are multiplied element by element and then input into the sixth binary module, and then the obtained feature is merged with the fourth binary feature to output the fifth binary feature; Inputting the fifth binary feature into the sixth adaptive gradient scaling module to obtain a sixth adaptive feature; The sixth adaptive feature and the fifth binary feature are multiplied element by element and then input into the seventh binary module, and then the obtained feature is merged with the fifth binary feature to output the sixth binary feature; Inputting the sixth binary feature into the full-precision fully connected layer for classification after being processed by the dropout layer to obtain a hyperspectral image classification prediction result; The BNN-based hyperspectral image classification model is reversely trained according to the hyperspectral image classification prediction result and the training samples to obtain a trained BNN-based hyperspectral image classification model.

3. The method according to claim 2, characterized in that The adaptive gradient scaling module includes: a multi-scale pyramid squeezing attention mechanism, a global average pooling layer, a first fully connected layer, a ReLU activation function, a second fully connected layer, and a Sigmoid activation function; The multi-scale pyramid squeeze attention mechanism includes an SPC module with a multi-scale pyramid convolution structure and a compression and excitation weight module; Inputting the convolution feature into the first adaptive gradient scaling module to obtain a first adaptive feature, comprising: The convolution feature is input into the SPC module, the convolution feature is firstly evenly divided in the channel dimension, and then all the division results are processed by convolution layers with convolution kernels of different scales, all the processing results are spliced ​​in the channel dimension, and finally the splicing results are processed by the compression and excitation weight module to obtain the attention weight vector; The attention weight vector is processed in sequence by the global average pooling layer, the first fully connected layer, the ReLU activation function, the second fully connected layer and the Sigmoid activation function to obtain a first adaptive feature.

4. The method according to claim 3, characterized in that: The mathematical expression of the SPC module is: FROM i =SE(F i ) F=Cat([F0,F1,...,F S-1 ]) Where Z is the multi-scale attention weight vector, represents the cascade operation, K represents the size of the convolution kernel, G is the size of the convolution group, S is the number of convolution groups, and C is divisible by S. C is the channel dimension of the feature map input to the SPC module, and Z i ∈R C′×1×1 is the attention weight vector, SE() represents the SE module, cat() represents the channel dimension concatenation operation, and F i ∈R C×H×W Represents feature maps of different scales, i = 0, 1, 2, ..., S-1, F∈R C×H×W is the overall multi-scale feature map.

5. The method according to claim 4, characterized in that In the SE module, the attention weight of the cth channel is: w c =σ(W1δ(W0(g c ))) Among them, w c represents the attention weight of the cth channel, g c represents the result of global average pooling of the heat certificate in the input SE module, and δ represents the ReLU operation and represents the fully connected calculation, σ represents the activation function, x c (i,j) represents the element in the i-th row and j-th column of the c-th channel.

6. The method according to claim 1, characterized in that The BNN-based hyperspectral image classification model also includes a dynamic curriculum learning strategy based on a confidence-aware loss function Superloss; The dynamic course learning strategy based on the confidence-aware loss function Superloss means that Superloss can be directly inserted into the classification task loss function, and the confidence parameter of the Superloss loss function is its convergence value at the limit. The confidence parameter of the Superloss loss function is updated with the state of the task loss during the sample training process. During the progressive quantization training process, there is no need to modify the learning process of the network; Following the approach of dynamic curriculum learning, the loss of each sample is monitored during training to determine its importance in the training process.

7. The method according to claim 6, characterized in that The confidence-aware loss function Superloss is: L λ (l i ,s i )=(l i -t)s i +λ(logσ i ) 2 l i =l CE (F(x i ;θ B ),y i ) Among them, SL λ (l i )Total loss, Represents the convergence value of the confidence parameter of the Superloss loss function at the limit; l i Represents the loss function of the classification task, L λ (l i ,σ i ) represents the confidence loss function; λ is the regularization parameter; τ is the ideal threshold used to distinguish easy-to-classify samples from hard-to-classify samples according to their respective losses, F(x i θ B ) represents the network's response to sample x i Corresponding prediction, y i Represents sample x i The corresponding truth value of i ≥0 represents a learnable confidence parameter, θ B is the network parameter.

8. A hyperspectral image classification device based on BNN, characterized in that: The device comprises: A training sample determination module is used to obtain a hyperspectral image and preprocess the hyperspectral image to obtain a training sample; A BNN-based hyperspectral image classification model training module is used to train a pre-built BNN-based hyperspectral image classification model using the training samples to obtain a trained BNN-based hyperspectral image classification model; the BNN-based hyperspectral image classification model includes: a basic BNN network and a number of adaptive gradient scaling modules based on a multi-scale pyramid compression attention mechanism that are only used during the training process; the adaptive gradient scaling module serves as a full-precision real-valued bypass branch of the basic BNN network during the training process, and is used to adjust the gradient during the back-propagation process while maintaining the representation capability of the network during the forward propagation; The hyperspectral image classification module is used to classify the pre-processed hyperspectral image to be classified using the basic BNN network in the trained BNN-based hyperspectral image classification model to obtain the hyperspectral image classification result.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer readable memory having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images