Twin multi-instance learning hyperspectral target detection network and detection method based on Transform

Abstract

According to the twin multi-instance learning hyperspectral target detection network and detection method based on Transform, through a spectrum embedding module, a semantic feature extraction module and target feature enhancement fusion operation, in the training process, the target features are extracted, and the target features are extracted; the network inputs a positive sample packet pair, namely two positive sample packets or a negative sample packet pair, namely one positive sample packet and one negative sample packet through a twin structure, and extracts spectral features of each sample through a spectrum embedding module to obtain packet embedding; the positive sample packet is embedded through a semantic feature extraction module, the semantic relationship between a target and a background is adaptively learned through a self-attention mechanism, the obtained semantic features are further converted into weight scores through a full-connection network, and the positive sample packet is embedded into weighted fusion to obtain representative target features; the negative sample packet is embedded without semantic feature extraction; jointly optimizing the model through InfoNCE comparison loss and cross entropy loss, back-propagating and updating network parameters, and strengthening the distinguishing capability of a target and a background; according to the method, the coding capability of spectrum embedding is effectively improved, and the detection distinguishing effect of target and background features is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of hyperspectral image detection, and particularly relates to a Siamese multi-instance learning hyperspectral target detection network and detection method based on Transformer. Background Art

[0002] As an important data form in the field of remote sensing, hyperspectral images combine rich spectral information and spatial resolution, and can record the characteristics of target objects in dozens to hundreds of continuous narrow bands. Compared with ordinary images, each pixel of a hyperspectral image not only contains two-dimensional geometric spatial information within the target area, but also has approximately continuous one-dimensional spectral information. Its characteristic of "combining spatial and spectral information" makes hyperspectral images of great value in target detection, classification, and analysis. Hyperspectral target detection is the process of detecting and identifying specific targets from complex scenes using hyperspectral imaging technology. Its core idea is to analyze spectral information to discover the unique spectral characteristics presented by specific targets in multiple narrow bands, thereby achieving the distinction between targets and the background. With the continuous progress of hyperspectral remote sensing technology, hyperspectral target detection has become a key research direction in multiple fields such as precision agriculture, environmental monitoring, resource exploration, and military reconnaissance. However, due to the influence of the complexity of hyperspectral data characteristics and application environments, hyperspectral target detection faces several significant challenges in technical applications: (1) The problem of inaccurate annotation: Due to the complexity and diversity of the ground environment, as well as the difference between GPS positioning accuracy (about 5 meters) and hyperspectral resolution (about 1 meter), it is very difficult to accurately label hyperspectral images pixel by pixel. This annotation error not only reduces the accuracy of annotation but also poses an obstacle to the training of supervised learning models; (2) The imbalance in the distribution of target and non-target pixels: In many actual scenarios, target pixels only account for a very small part of the total pixels, and the ratio of target to non-target may even reach a huge ratio of 1:10,000. This unevenness in data distribution significantly increases the difficulty of model learning, easily leading to overfitting of the model to the background class and ignoring the accuracy of target detection; (3) The sub-pixel aliasing phenomenon in hyperspectral images: A single pixel in a hyperspectral image may contain multiple substances, and its spectral characteristics are presented as a mixture of the spectra of multiple substances, which makes it more difficult to distinguish target and background features, especially when the target spectral characteristics are weak or the background is complex and variable. These challenges together constitute the core technical bottleneck in hyperspectral target detection and also pose higher requirements on existing algorithms and models.

[0003] To address the problem of inaccurate annotation in hyperspectral target detection, weakly supervised learning methods have gradually become a research hotspot. Weakly supervised learning is a method that can effectively train a model even when the annotation information of the training data is incomplete, inaccurate, or even noisy. Among them, Multiple Instance Learning (MIL), as a typical framework of weakly supervised learning, is particularly suitable for hyperspectral target detection tasks due to its flexible annotation method and wide applicability. The core feature of multiple instance learning is that the data is organized in the form of "bags", each bag consists of several instances, and the label of the bag only indicates whether there is at least one target instance in the bag, without the need to annotate each instance separately. In actual hyperspectral target detection, the region containing target pixels can be labeled as a positive bag, while the pure background region is labeled as a negative bag. This modeling method effectively reduces the dependence on pixel-level accurate annotation and overcomes the problem of insufficient training data caused by difficult and costly annotation. In addition, the multiple instance learning framework can efficiently learn the differential features between the target and the background on weakly labeled data by inferring instance-level features from bag-level labels, thereby improving the detection ability of the model in complex scenarios. This method is particularly suitable for scenarios where the ground environment in hyperspectral images is complex, multi-target aliasing exists, and labels are scarce. Therefore, multiple instance learning has become an important research direction in the field of hyperspectral target detection, showing broad application prospects and theoretical value.

[0004] The existing implementation scheme closest to the present invention is the MILES algorithm [Chen Y, Bi J, Wang J Z. MILES: Multiple-instance learning via embedded instance selection [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(12): 1931-1947], and the CS-attention MINN algorithm [Bi Q, Qin K, Li Z, et al. A multiple-instance densely-connected ConvNet for aerial scene classification [J]. IEEE Transactions on Image Processing, 2020, 29: 4911-4926.]. The MILES algorithm obtains the feature mapping of each packet in the space composed of all examples by mapping each data packet to the space, and then trains an SVM classifier to obtain a classifier at the packet level. During testing, the positive examples in the positive packet are determined by inversely inferring the contribution of each example in the positive packet to the classification of the packet. The CS-attention MINN algorithm proposes a multiple-instance pooling and attention module based on channel space attention. This module is trainable and can highlight the examples related to the scene label to determine the positive examples in the positive packet. The disadvantages are low detection accuracy, weak ability to distinguish between target and background features, and low generalization ability. Summary of the Invention

[0005] To overcome the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a Transformer-based siamese multi-instance learning hyperspectral target detection network and detection method. Through a spectral embedding module, a semantic feature extraction module, and an operation of enhancing and fusing target features, during the training process, the network inputs positive sample bags pairs (i.e., two positive sample bags) or negative sample bags pairs (i.e., one positive sample bag and one negative sample bag) through a siamese structure, and extracts the spectral features of each instance through the spectral embedding module to obtain bag embeddings; the positive sample bag embeddings pass through the semantic feature extraction module, and the self-attention mechanism is used to adaptively learn the semantic relationship between the target and the background. The obtained semantic features are further transformed into weight scores through a fully connected network, and the positive bag embeddings are weighted and fused to obtain representative target features; during the fusion, the weight scores are enhanced through Gaussian perturbation to obtain more representative target feature representations to balance the quantity difference between the target and the background; since the negative sample bag embeddings do not contain targets, no feature extraction is performed; to strengthen the discrimination ability between the target and the background, the model is jointly optimized through the InfoNCE contrast loss and the cross-entropy loss, and the network parameters are updated through backpropagation; the present invention effectively improves the encoding ability of spectral embedding and enhances the discrimination effect between target and background features; the network of the present invention improves the diversity of target features by adding Gaussian perturbation to balance the quantity gap between the target and the background, and combines the joint optimization of InfoNCE contrast learning and classification tasks, successfully improving the accuracy, generalization ability, and anti-noise performance of hyperspectral image target detection. Compared with the prior art, it has stronger adaptability and higher detection accuracy.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A Transformer-based siamese multi-instance learning hyperspectral target detection network, comprising a spectral embedding module, a semantic feature extraction module, and a target feature enhancement and fusion module; wherein:

[0008] The spectral embedding module is composed of three convolutional blocks and a fully connected layer; each convolutional block contains a convolutional layer, a PReLU activation layer, and a max pooling layer; and is responsible for extracting effective spectral features from the input hyperspectral data points.

[0009] The semantic feature extraction module includes a positional encoding layer and a Transformer encoder. The positional encoding layer uses a sine positional encoding method for encoding to ensure that the sequential information of the hyperspectral data is retained; the Transformer encoder is composed of multiple stacked encoder layers, and each layer contains a multi-head self-attention mechanism to enhance the adaptive modeling ability of target and background features, realize enhancing the representativeness of target features, and improve the discrimination ability between target and background features.

[0010] The target feature enhancement and fusion module is used to add semantic weights during target feature embedding and generate diverse target feature representations through Gaussian perturbation, so as to enhance the representativeness of target features and improve the discrimination ability between target and background features.

[0011] A twin multi-instance learning hyperspectral target detection method based on Transformer, the specific operation steps are as follows:

[0012] Step 1. Obtain a training sample set and a test sample set, which are used to train the neural network to learn the feature distributions of hyperspectral targets and backgrounds, and are evaluated in the test stage;

[0013] Step 2. Based on the training sample set obtained in Step 1, construct a twin multi-instance neural network W based on Transformer, which is used to extract the deep features of hyperspectral data, and realize the feature contrast learning between target and background under the twin structure, as well as the network structure for classifier training, so as to prepare for subsequent training;

[0014] Step 3. Adopt the training sample set obtained in Step 1 and, based on the twin multi-instance neural network W based on Transformer constructed in Step 2, perform iterative training on the twin multi-instance neural network W based on Transformer to obtain an optimized target detection model, so that it can accurately distinguish targets and backgrounds, and train a classifier with good detection performance;

[0015] Step 4. Use the twin multi-instance neural network W based on Transformer trained in Step 3 to perform target detection on the test sample set obtained in Step 1 to obtain hyperspectral target detection results.

[0016] The specific method of Step 1 is as follows:

[0017] Step (1a) Select C types of spectra from the spectral library, select one type as the target spectrum, and the other C - 1 types as background spectra, where C ≥ 3. The selected spectra are used to construct a hyperspectral training sample set and a test sample set. Among them, the mixture of the target spectrum and background spectra generates target category pixel points, and the mixture of background spectra generates background category pixel points;

[0018] Step (1b) Set the average target spectrum proportion value of target category pixel points during the overall mixing generation process as Pt, where 0 < Pt ≤ 0.25, that is, among all target category pixel points, the average proportion of the target spectrum is Pt; for any target category pixel point, its spectrum is composed of a weighted combination of the target spectrum and C - 1 background spectra. Let the specific proportion value of the target spectrum be β 0 , and the specific proportion value of the background spectrum be β 1 , β 2 , …, βC-1 Randomly assigned, and β 0 +β 1 +β 2 +…+β C-1 = 1; For background category pixel points, only C - 1 kinds of background spectra are mixed and generated according to the randomly assigned ratio β 1 , β 2 , …, β C-1 , and β 1 +β 2 +…+β C-1 = 1; According to the ratio values β 0 , β 1 , β 2 , …, β C-1 Perform weighted summation on the spectra to generate highly mixed noisy data. Set the value range of the signal-to-noise ratio SNR to 20 ≤ SNR ≤ 40 dB, and ensure that the target category pixel points contain the target spectrum and the background category pixel points do not contain the target spectrum;

[0019] Step (2a) Set up a spectral embedding module for extracting effective spectral features from the input hyperspectral pixel points; The spectral embedding module consists of multiple convolutional blocks and a fully connected layer. Each convolutional block contains a convolutional layer, a PReLU activation layer, and a max pooling layer. Local spectral features are extracted through convolutional operations, and the pooling operation reduces the feature dimension and enhances robustness; The number of channels of each convolutional block gradually increases to extract deeper-level feature information layer by layer; The data processed by all convolutional layers is mapped to a feature space of a fixed dimension through a fully connected layer, and the Tanh activation function is used to enhance the feature representation ability;

[0020] Step (2b) Set up a semantic feature extraction module. The semantic feature extraction module includes a position encoding layer and a Transformer encoder; The position encoding layer uses the sin / cos method to provide position information for each element in the input sequence; The Transformer encoder contains multiple cascaded encoders, and each layer of the Transformer encoder uses a multi-head self-attention mechanism; When calculating self-attention, the traditional vector inner product similarity QK T is replaced with a calculation method based on cosine similarity , and its expression is:

[0021]

[0022] Step (2c) The semantic features output by the semantic feature extraction module are processed by a fully connected layer. The fully connected layer converts the semantic features into a weight score vector ω for each hyperspectral pixel point; Through the target feature enhancement and fusion module, the weight score vector ω is subjected to sparse normalization processing, that is, less than the set threshold The weight score will be set to 0; then, set the Gaussian perturbation ∈ to follow a normal distribution with a mean of 0 and a variance of σ 2 Add the Gaussian perturbation ∈ to the weight score vector ω, so that the weight score vector ω generates a weight score vector ω' with the same distribution; repeat adding the Gaussian perturbation ∈ to the weight score vector ω M times to obtain M different weight score vectors ω' 1 , ω' 2 ,…, ω' M , these weight score vectors ω' 1 , ω' 2 ,…, ω' M respectively participate in the fusion of the sample bag embeddings to obtain M different representative features of the sample bag embeddings; the expression for adding the Gaussian perturbation ∈ to the weight score vector ω is:

[0023] ω' = ω × (1 + ∈), ∈~N(0, σ 2 );

[0024] Step (2d) sets the contrast loss function and the cross-entropy loss function; two sample bags B 1 and B 2 of the input training set samples respectively are processed by the spectral embedding module and the semantic feature extraction module to obtain the features of the corresponding two sample bags, and the contrast loss is jointly calculated using the features of the two sample bags, and the cross-entropy loss is calculated separately for the features of each sample bag; the expression of the loss function is:

[0025]

[0026] Jointly optimizing the two loss functions of the contrast loss and the cross-entropy loss, the Transformer-based Siamese multi-instance neural network W can more accurately learn the feature discrimination between the target and the background in the object detection task and alleviate the negative impact of class imbalance on the training process;

[0027] Loss total = γLoss cls + Loss cts

[0028] where Loss cls is the cross-entropy loss for the classification task, Loss cts is the contrast loss; N is the number of features participating in training the classifier, p n is the predicted confidence value of the feature x n , y n is the label of the bag to which x n belongs. When x n is the representative target feature, then y n = 1. When x n is the background pixel feature, then yn = 0; sim(·, ·) represents calculating the similarity of features, represents the representative target feature of the positive bag pair i and the similarity between represents the representative target feature of the positive bag in the negative bag pair j and the k-th background pixel feature in the negative bag the similarity between; τ is the temperature scaling coefficient, used to adjust the distribution of similarities between positive and negative sample bag pairs to avoid over-penalizing the similarity of negative bags; γ is the proportion of the classification loss in the total loss.

[0029] The specific method for iteratively training the Transformer-based Siamese multi-instance neural network W in step 3 is as follows:

[0030] Step (3a) Based on the Transformer-based Siamese multi-instance neural network W constructed in step 2, initialize the training process. Let the current iteration number be t and the maximum iteration number be T, T ≥ 100. Set the Transformer-based Siamese multi-instance neural network in the current parameter state as W t ; At the start of training, set the initial state as t = 1;

[0031] Step (3b) Use the training set as the input of the current Transformer-based Siamese multi-instance neural network W t for forward propagation, where (B 1 , B 2 ) i represents the i-th sample bag pair in the training set, is the positive sample bag pair, is the negative sample bag pair; The spectral embedding module extracts the spectral features of the pixels in each sample bag of the training set to obtain the pixel spectral feature set All the positive sample bag embeddings E in the feature set F + have to be input into the semantic feature extraction module for processing and then fused into the representative target feature e + , and at the same time, Gaussian perturbation is enhanced for the fusion weights to generate representative target features equivalent to the number of background pixel features; The negative sample bag embeddings directly release the background pixel features e - contained in the negative sample bags; F is transformed into the set where represents the representative target feature of represents the representative target feature bag after Gaussian perturbation enhancement of represents Released background pixel feature packets;

[0032] Each sample packet in the set F' of step (3c) pairs the feature (e 1 ,e 2 ) i to participate in the calculation of the contrastive loss Loss cts ; and participate in the training of the classifier and calculate the cross-entropy loss Loss cls ; After obtaining the total loss based on the loss function calculate the parameter gradient of the neural network W t of the Transformer-based Siamese multi-instance neural network W, and use the stochastic gradient descent (SGD) algorithm to update the parameters θ w of the Transformer-based Siamese multi-instance neural network W to obtain the updated, i.e., the current, parameters of the Transformer-based Siamese multi-instance neural network W t , and the update formulas are respectively:

[0033]

[0034] where η represents the gradient descent parameter, represents the parameters of the current Transformer-based Siamese multi-instance neural network W t , represents the parameters of the Transformer-based Siamese multi-instance neural network W before update, represents the partial derivative operation;

[0035] Step (3d) determines whether the current iteration number t is equal to the maximum iteration number T. If so, obtain the trained multi-instance neural network W of self-attention semantic modeling * ; Otherwise, let t = t + 1, and return to step (3b) to repeat the execution.

[0036] The specific method for the above step 4 to obtain the hyperspectral target detection result is:

[0037] Use the test sample set as the input of the multi-instance neural network W * trained in step 3 for target detection, and output the probability that each pixel belongs to the target.

[0038] A Transformer-based Siamese multi-instance learning hyperspectral target detection device includes:

[0039] A memory for storing computer programs;

[0040] A processor for implementing the Transformer-based twin multi-instance learning hyperspectral target detection method according to any one of steps 1 to 4 when executing the computer program.

[0041] A computer-readable storage medium storing a computer program, which can perform hyperspectral target detection based on the Transformer-based twin multi-instance learning hyperspectral target detection method when the computer program is executed by a processor.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0043] The application of the Transformer-based twin multi-instance neural network in hyperspectral target detection in the present invention. The constructed network mainly includes a spectral embedding module and a semantic feature extraction module. By mapping the positive sample bag data to positive sample bag embeddings and expanding them into a sequence form, the semantic feature extraction module uses the self-attention mechanism to adaptively focus on and extract the target information in the positive sample bag, which can effectively enhance the accurate expression ability of target features, improve the discrimination of target features in detection, has strong adaptability and generalization ability, and has the technical advantage of improving the detection accuracy in a weakly supervised scenario. Moreover, in order to better conform to the characteristics of hyperspectral data, the operation of the self-attention mechanism of Transformer adopts the form of cosine similarity, which can reduce the influence of feature scale on attention calculation, make the feature representation more robust, and at the same time alleviate the numerical instability problem in traditional dot-product attention calculation, and has the technical characteristics of being more in line with the feature distribution of hyperspectral data, being able to better model the relative relationship between the target and background categories and improving the model robustness.

[0044] The present invention adopts the InfoNCE contrast learning mechanism for pairs of bags under the twin multi-instance learning architecture. The InfoNCE loss is good at maximizing the relative difference between positive and negative samples in natural language processing or multi-modal alignment tasks to learn effective representations. The input data of the twin multi-instance neural network naturally contains randomly paired positive bag pairs and negative bag pairs. Combining the method of InfoNCE contrast learning and introducing a temperature scaling parameter at the same time enables the model to pay more balanced attention to target and background features, effectively separating the representative target features and background features in the embedding space, and enabling better representation and discrimination of the target and background.

[0045] The present invention adopts an enhanced fusion method for representative target features with Gaussian perturbation. The number of target pixels is usually less than that of background pixels. To balance the quantity gap between the target and the background, before the combination of positive bag embedding and semantic weight fusion, Gaussian perturbation is added to the semantic weight for enhancement. After slight, uniformly symmetric random perturbation, more diverse weights are generated, and thus more effective representative target features are fused. The balance between target and background features is conducive to training a more robust classifier.

[0046] In summary, compared with the existing hyperspectral target detection methods MILES and CS-attention MINN, the AUC value of the Transformer-based Siamese multi-instance learning hyperspectral target detection method proposed by the present invention has been significantly improved, indicating that this method effectively alleviates the model overfitting phenomenon caused by the problems of large noise and spectral aliasing between the target and the background, enhances the feature representation and distinguishability of the target and the background, thereby improving the target detection accuracy, and thus has important practical application value. Brief Description of the Drawings

[0047] Figure 1 It is the implementation flowchart of the method of the present invention.

[0048] Figure 2 It is the schematic diagram of the spectral curves of the target and the background used in the simulation experiment of the present invention. Detailed Embodiment

[0049] The present invention will be further described in detail below with reference to the drawings.

[0050] A Transformer-based Siamese multi-instance learning hyperspectral target detection network includes a spectral embedding module, a semantic feature extraction module, and a target feature enhancement fusion module; wherein:

[0051] The spectral embedding module is composed of three convolutional blocks and a fully connected layer; each convolutional block contains a convolutional layer, a PReLU activation layer, and a max pooling layer; and is responsible for extracting effective spectral features from the input hyperspectral data points;

[0052] The semantic feature extraction module includes a position encoding layer and a Transformer encoder. The position encoding layer uses the sine position encoding method for encoding to ensure that the sequential information of the hyperspectral data is retained; the Transformer encoder is composed of multiple stacked encoder layers, and each layer contains a multi-head self-attention mechanism to enhance the adaptive modeling ability of the target and background features, realize enhancing the representativeness of the target features, and improve the distinguishability of the target and background features;

[0053] The target feature enhancement and fusion module is used to add semantic weights during target feature embedding and generate diverse target feature representations through Gaussian perturbation, so as to enhance the representativeness of target features and improve the discrimination ability between target and background features.

[0054] A twin multi-instance learning hyperspectral target detection method based on Transformer, such as Figure 1 , the specific operation steps are as follows:

[0055] Step 1. Obtain the training sample set and the test sample set, which are used to train the neural network to learn the feature distributions of hyperspectral targets and backgrounds and are evaluated in the test stage; the specific method is:

[0056] Step (1a) Select C types of spectra from the spectral library, choose one type as the target spectrum, and the other C - 1 types as the background spectra, where C ≥ 3. In this example, C = 4, and the spectra of 4 minerals, namely The Verde Antique, The Phyllite, The Pyroxenite, and The Red Slate, are selected. Figure 2 shows the visualized spectral curves of these 4 mineral spectra. The spectral dimension is 211, and the Red Slate spectrum is selected as the target spectrum; the selected spectra will be used to construct the hyperspectral training sample set and the test sample set, where the mixture of the target spectrum and the background spectra generates target class pixel points, and the mixture of the background spectra generates background class pixel points.

[0057] Step (1b) Set the average target spectrum proportion value Pt of the target class pixel points in the overall mixing generation process, where 0 < Pt ≤ 0.25, that is, the average proportion of the target spectrum in all target class pixel points is Pt; for any target class pixel point, its spectrum is composed of the weighted combination of the target spectrum and C - 1 background spectra. Let the specific proportion value of the target spectrum be β 0 , and the specific proportion values of the background spectra be β 1 , β 2 , …, β C-1 are randomly assigned, and β 0 + β 1 + β 2 + … + β C-1 = 1; for the background class pixel points, they are only generated by the mixture of C - 1 background spectra according to the randomly assigned proportions β 1 , β 2 , …, β C-1 , and β 1 + β 2 + … + β C-1 = 1; according to the proportion values β 0 , β 1, β 2 , …, β C-1 Perform a weighted sum of the spectra to generate highly mixed noisy data. Set the value range of the signal-to-noise ratio SNR to 20 ≤ SNR ≤ 40 dB, and ensure that the target class pixel points contain the target spectrum, while the background class pixel points do not contain the target spectrum. In this example, the signal-to-noise ratio is set to 30 dB, and the values of Pt are set to 0.1, 0.15, and 0.25 respectively to generate 3 groups of mixed hyperspectral data, and add Gaussian white noise of 30 dB.

[0058] Step (1c) sets the packet size to N bag , and aggregates all the target class pixel points and background class pixel points generated by mixing in step (1b) into packets B according to the scale of N bag . The packets containing target class pixel points are marked as positive sample packets B + , and the packets composed only of background class pixel points are marked as negative sample packets B - ; Divide all the aggregated positive sample packets B + and negative sample packets B - into a training set and a test set. The training set is denoted as To ensure that the Transformer-based Siamese multi-instance neural network W can be effectively trained, parameter-tuned, and performance-evaluated at different stages. In this example, the training set, validation set, and test set all contain 200 positive packets and 300 negative packets, and each packet contains 25 pixel points.

[0059] Step 2. Based on the training sample set obtained in Step 1, construct a Transformer-based Siamese multi-instance neural network W to extract deep features of hyperspectral data, and implement feature contrast learning between the target and the background under the Siamese structure, as well as the network structure for classifier training, to prepare for subsequent training; the specific method is as follows:

[0060] Step (2a) sets up a spectral embedding module to extract effective spectral features from the input hyperspectral pixel points; in this embodiment, the spectral embedding module consists of three convolutional blocks and a fully connected layer; each convolutional block contains a convolutional layer, a PReLU activation layer, and a max pooling layer; the convolutional kernel size is set to 1×3, the stride is 1, the pooling kernel size is 1×2, and the stride is 1×2; the number of input channels of the first convolutional block is 1, and the number of output channels is 20; the number of output channels of the second convolutional block is 128; the number of output channels of the third convolutional block is 64; the data after all convolutional layers passes through a fully connected layer, which contains 128 neurons; the data processed by all convolutional layers is mapped to a feature space of a fixed dimension through the fully connected layer, and the Tanh activation function is used to enhance the feature representation ability;

[0061] Step (2b) sets up a semantic feature extraction module, which consists of a position encoding layer and a Transformer encoder; the position encoding layer uses the sin / cos method to provide position information for each element in the input sequence; the Transformer encoder contains 12 cascaded encoders, and each layer of the Transformer encoder uses an 8-head self-attention mechanism; when calculating self-attention, the Transformer encoder replaces the traditional vector inner product similarity QK T with a calculation method based on cosine similarity The expression is:

[0062]

[0063] In step (2c), the semantic features output by the semantic feature extraction module are processed by a fully connected layer, and the fully connected layer converts the semantic features into a weight score vector ω for each hyperspectral pixel point; through the target feature enhancement and fusion module, the weight score vector ω is subjected to sparse normalization, that is, the weight scores less than the set threshold will be set to 0; then, a Gaussian perturbation ∈ is set to follow a normal distribution with a mean of 0 and a variance of σ 2 , and the Gaussian perturbation ∈ is added to the weight score vector ω, so that the weight score vector ω generates a weight score vector ω' with the same distribution; the Gaussian perturbation ∈ is added to the weight score vector ω M times repeatedly to obtain M different weight score vectors ω' 1 , ω' 2 , …, ω' M , and these weight score vectors ω' 1 , ω' 2 , …, ω' M participate in the fusion of sample bag embeddings respectively to obtain M different representative features of sample bag embeddings; the expression for adding the Gaussian perturbation ∈ to the weight score vector ω is:

[0064] ω' = ω × (1 + ∈), ∈ ~ N(0, σ 2 );

[0065] In step (2d), a contrast loss function and a cross-entropy loss function are set; two sample bags B 1 and B 2 of the input training set samples are processed by the spectral embedding module and the semantic feature extraction module to obtain the features of the corresponding two sample bags, and the contrast loss is jointly calculated using the features of the two obtained sample bags, and the cross-entropy loss is calculated separately for the features of each sample bag; the expression of the loss function is:

[0066]

[0067] By jointly optimizing the two loss functions of contrastive loss and cross-entropy loss, the Transformer-based Siamese multi-instance neural network W can more accurately learn the feature discrimination between the target and the background in the object detection task and alleviate the negative impact of class imbalance on the training process;

[0068] Loss total = γLoss cls + Loss cts

[0069] where Loss cls is the cross-entropy loss for the classification task, and Loss cts is the contrastive loss; N is the number of features participating in training the classifier, p n is the predicted confidence value of feature x n , y n is the label of the bag to which x n belongs. When x n is a representative target feature, then y n = 1. When x n is a background pixel feature, then y n = 0; sim(·,·) represents calculating the similarity of features, represents the similarity between the representative target feature of the positive bag pair i and , and represents the similarity between the representative target feature of the positive bag in the negative bag pair j and the k-th background pixel feature in the negative bag; τ is the temperature scaling coefficient used to adjust the distribution of similarities between positive and negative sample bag pairs and avoid over-penalizing the similarity of negative bags; γ is the proportion of the classification loss in the total loss; in this example, γ = 0.04 and τ = 0.2.

[0070] Step 3. Using the training sample set obtained in Step 1 and based on the Transformer-based Siamese multi-instance neural network W constructed in Step 2, perform iterative training on the Transformer-based Siamese multi-instance neural network W to obtain an optimized object detection model that can accurately distinguish the target and the background and train a classifier with good detection performance; the specific method is as follows:

[0071] Step (3a) Based on the Transformer-based Siamese multi-instance neural network W constructed in Step 2, initialize the training process. Let the current iteration number be t and the maximum iteration number be T, T ≥ 100. Set the Transformer-based Siamese multi-instance neural network in the current parameter state as W t ; at the start of training, set the initial state as t = 1; ​

[0072] Step (3b) takes the training set as the input for the current Transformer-based Siamese multi-instance neural network W t to perform forward propagation, where (B 1 , B 2 ) i represents the i-th sample bag pair in the training set, is a positive sample bag pair, is a negative sample bag pair; the spectral embedding module extracts the spectral features of the pixels in each sample bag of the training set to obtain the pixel spectral feature set All positive sample bag embeddings E in the feature set F + are input into the semantic feature extraction module for processing and then fused into the representative target feature e + , and at the same time, Gaussian perturbation enhancement is performed on the fusion weights to generate representative target features equivalent to the number of background pixel features; the negative sample bag embeddings directly release the background pixel features e - contained in the negative sample bags; F is transformed into the set where represents 's representative target feature, represents 's representative target feature packet after Gaussian perturbation enhancement, represents 's released background pixel feature packet;

[0073] Step (3c) Each sample bag pair feature (e 1 , e 2 ) i in the set F' participates in the calculation of the contrastive loss Loss cts ; and participate in the training of the classifier and calculate the cross-entropy loss Loss cls ; After obtaining the total loss , based on the loss function calculate the parameter gradient of the neural network W t , and use the stochastic gradient descent (SGD) algorithm to update the parameters θ w of the Transformer-based Siamese multi-instance neural network W to obtain the updated, i.e., the current parameters of the Transformer-based Siamese multi-instance neural network W t , and the update formulas are respectively:

[0074]

[0075] where η represents the gradient descent parameter, Denote the parameters of the current Transformer-based Siamese multi-instance neural network \(W\) t ; Denote the parameters of the Transformer-based Siamese multi-instance neural network \(W\) before update Denote the partial derivative operation;

[0076] (3d) Determine whether the current iteration number \(t\) is equal to the maximum iteration number \(T\). If so, obtain the trained multi-instance neural network \(W\) for self-attention semantic modeling * ; Otherwise, let \(t = t + 1\), and return to step (3b) to repeat the execution.

[0077] Step 4. Use the Transformer-based Siamese multi-instance neural network \(W\) trained in step 3 to perform object detection on the test sample set obtained in step 1, and obtain the hyperspectral object detection result.

[0078] Compare the AUC values of the ROC curves of the objects in the test data sets with the target spectral ratio values \(P_t\) of 0.1, 0.15, and 0.25 obtained by the method proposed in the present invention, the MILES method, and the CS-attention MINN method. The results are shown in Table 1.

[0079] Table 1

[0080]

[0081]

[0082] It can be seen from Table 1 that the proposed Transformer-based Siamese multi-instance learning hyperspectral object detection method in the present invention has a significantly improved AUC value compared with the existing hyperspectral object detection methods MILES and CS-attention MINN. This shows that this method effectively alleviates the model overfitting phenomenon caused by the problems of large noise and spectral aliasing between the target and the background, enhances the feature representation of the target and the background and the distinguishability of the features, thereby improving the object detection accuracy, and thus has important practical significance.

[0083] As can be seen from Table 1, the AUC values of the method proposed in the present invention on datasets of three specifications are all better than those of the existing methods MILES and CS-attention MINN. On the dataset with Pt = 0.25, the target features of this dataset are relatively obvious and target detection is relatively easy. The comparison methods MILES and CS-attention MINN both perform well on this dataset. However, the AUC of the method of the present invention reaches 0.920, which is 0.013 higher than that of the CS-attention MINN method with a relatively high AUC, and the improvement rate is 1.4%. This result shows that the method of the present invention not only maintains high precision but also further optimizes the detection performance. At the same time, on the dataset with Pt = 0.15, the target features are weak, and the AUC value of the method of the present invention reaches 0.847, which is 0.015 higher than that of the CS-attention MINN with a relatively high AUC value, and the improvement rate is 1.8%. The experimental results of these two datasets show that the method of the present invention has obvious advantages in the ability to distinguish between targets and backgrounds, has stronger feature learning ability, and its detection performance is significantly better than that of the existing methods.

[0084] On the dataset with Pt = 0.1, that is, the dataset with relatively weak target features and great detection difficulty, the improvement of the method of the present invention is particularly obvious. The AUC value of this dataset is 0.815, which is 0.071 higher than the AUC (0.744) of the MILES method, and the improvement rate is 9.5%; it is 0.022 higher than the AUC (0.793) of the CS-attention MINN method, and the improvement rate is 2.8%. This result shows that the method of the present invention can still effectively capture target information in the case of relatively difficult target detection, showing strong anti-interference and anti-noise capabilities and better robustness. This advantage benefits from the self-attention mechanism adopted in the present invention, which can accurately focus on the target information in the sample packet and calculate the self-attention weight through cosine similarity, so as to optimize according to the characteristics of hyperspectral data.

[0085] In summary, the method of the present invention shows excellent detection performance and robustness under various dataset specifications. Especially when dealing with hyperspectral data with weak target features or high noise, it shows stronger target and background feature extraction and discrimination capabilities, demonstrating its significant advantages in hyperspectral target detection.

Claims

1. A Transformer-based twin multi-instance learning hyperspectral target detection network, characterized in that: It includes a spectral embedding module, a semantic feature extraction module, and a target feature enhancement fusion module; among which: The spectral embedding module consists of multiple convolutional blocks and a fully connected layer; each convolutional block contains a convolutional layer, a PReLU activation layer, and a maximum pooling layer; it is responsible for extracting effective spectral features from the input hyperspectral data points; The semantic feature extraction module includes a position encoding layer and a Transformer encoder. The position encoding layer is encoded using sinusoidal position encoding to ensure that the order information of the hyperspectral data is preserved. The Transformer encoder consists of multiple stacked encoder layers, each of which contains a multi-head self-attention mechanism to enhance the adaptive modeling capability of target and background features, enhance the representativeness of target features, and improve the ability to distinguish between target and background features. The target feature enhancement fusion module is used to add semantic weights when embedding target features, and generate diversified target feature representations through Gaussian perturbations to enhance the representativeness of target features and improve the ability to distinguish between target and background features.

2. A Transformer-based twin multi-instance learning hyperspectral target detection method, characterized in that: The specific steps are as follows: Step 1. Obtain training sample sets and test sample sets to train the neural network to learn the characteristic distribution of hyperspectral targets and backgrounds, and evaluate them in the test phase; Step 2. Based on the training sample set obtained in step 1, a Transformer-based twin multi-instance neural network W is constructed to extract the deep features of the hyperspectral data and realize the feature comparison learning of the target and the background under the twin structure, as well as the network structure of the classifier training, to prepare for subsequent training; Step 3. Using the training sample set obtained in step 1, the Transformer-based twin multi-instance neural network W constructed in step 2 is iteratively trained to obtain an optimized target detection model, so that it can distinguish between targets and backgrounds, and train a classifier with good detection performance; Step 4. Use the Transformer twin multi-instance neural network W trained in step 3 to perform target detection on the test sample set obtained in step 1 to obtain the hyperspectral target detection results.

3. The Transformer-based twin multi-instance learning hyperspectral target detection method according to claim 2, characterized in that: The specific method of step 1 is: Step (1a) selects C types of spectra from the spectral library, selects one type as the target spectrum, and the other C-1 types as the background spectrum, where C≥3, and the selected spectra are used to construct a hyperspectral training sample set and a test sample set, wherein the mixture of the target spectrum and the background spectrum generates the target category pixel point, and the mixture of the background spectrum generates the background category pixel point; Step (1b) sets the average target spectral proportion value of the target category pixel points in the overall mixing generation process as Pt, where 0 < Pt ≤ 0.25, that is, among all the target category pixel points, the average proportion of the target spectrum is Pt; for any target category pixel point, its spectrum is composed of a weighted combination of the target spectrum and C - 1 background spectra. Let the specific proportion value of the target spectrum be β0, and the specific proportion values of the background spectra be β1, β2, …, β C-1 Randomly assign, and β0 + β1 + β2 + … + β C-1 = 1; for the background category pixel points, they are only generated by mixing the C - 1 background spectra according to the randomly assigned proportions β1, β2, …, β C-1 And β1 + β2 + … + β C-1 = 1; according to the proportion values β0, β1, β2, …, β C-1 Perform weighted summation on the spectra to generate highly mixed noisy data. Set the value range of the signal - to - noise ratio SNR as 20 ≤ SNR ≤ 40 dB, and ensure that the target category pixel points contain the target spectrum and the background category pixel points do not contain the target spectrum; Step (1c) Set the packet size to N bag , all target category pixels and background category pixels generated by mixing in step (1b) are mixed according to N bag The size of the data is aggregated into data packet B, and the packet containing the target category pixels is marked as positive sample packet B + , the package consisting only of background category pixels is marked as negative sample package B - ; Pack all the aggregated positive samples into B + and negative sample bag B - Divided into training set and test set, the training set is denoted as Each sample in the training set contains two sample packages B1 and B2 to ensure that the Transformer twin multi-instance neural network W can be effectively trained, parameter tuned and performance evaluated at different stages.

4. The Transformer-based twin multi-instance learning hyperspectral target detection method according to claim 2, characterized in that: The specific method of step 2 to construct the Transformer-based twin multi-instance neural network W is: Step (2a) sets a spectral embedding module for extracting effective spectral features from the input hyperspectral pixel points; the spectral embedding module is composed of multiple convolution blocks and fully connected layers, each convolution block includes a convolution layer, a PReLU activation layer and a maximum pooling layer, and extracts local spectral features through convolution operations, and the pooling operation reduces the feature dimension and enhances robustness; the number of channels of each convolution block is gradually increased to extract deeper feature information layer by layer; the data processed by all convolution layers are mapped to a fixed-dimensional feature space through a fully connected layer, and a Tanh activation function is used to enhance the feature representation capability; Step (2b) sets a semantic feature extraction module, which includes a position encoding layer and a Transformer encoder. The position encoding layer uses a sin / cos method to provide position information for each element in the input sequence. The Transformer encoder includes multiple layers of cascaded encoders, and each layer of the Transformer encoder uses a multi-head self-attention mechanism. When calculating the self-attention, the Transformer encoder converts the traditional vector inner product similarity QK T Replaced with cosine similarity The calculation method is as follows: In step (2c), the semantic features output by the semantic feature extraction module are processed by the fully connected layer, which converts the semantic features into a weight score vector ω for each hyperspectral pixel; the weight score vector ω is sparsely normalized by the target feature enhancement fusion module, that is, it is less than the set threshold. The weight score will be set to 0; then, set the Gaussian perturbation ∈ to have a mean of 0 and a variance of σ 2 Normal distribution, add Gaussian perturbation ∈ to the weight score vector ω, so that the weight score vector ω generates a weight score vector ω' of the same distribution; repeat adding M Gaussian perturbations ∈ to the weight score vector ω to obtain M different weight score vectors ω'1, ω'2, ..., ω' M , these weight score vectors ω'1,ω'2,…,ω' M Participate in the fusion of sample package embedding respectively, and obtain representative features of M different sample package embeddings; The expression for adding Gaussian perturbation ∈ to the weight score vector ω is: ω'=ω×(1+∈),∈~N(0,σ 2 ); Step (2d) sets the contrast loss function and the cross entropy loss function; the two sample packages B1 and B2 of the training set samples are respectively inputted, and the features of the two corresponding sample packages are obtained by processing the spectral embedding module and the semantic feature extraction module, and the contrast loss is calculated jointly using the features of the two sample packages, and the cross entropy loss is calculated separately using the features of each sample package; the expression of the loss function is: By jointly optimizing the contrast loss and cross entropy loss, the Transformer-based twin multi-instance neural network W can more accurately learn the feature distinction between the target and the background in the target detection task and alleviate the negative impact of category imbalance on the training process. Loss total =γLoss cls +Loss cts Among them, Loss cls is the cross entropy loss for the classification task, Loss cts is the contrast loss; N is the number of features involved in training the classifier, p n is feature x n The prediction confidence value, y n is x n The label of the package to which it belongs. n is the representative target feature, then y n =1, when x n is the background pixel feature, then y n =0; sim(·,·) indicates the similarity of the calculated features, Represents the representative target feature of positive bag pair i and The similarity between Represents the representative target features of the positive bag in the negative bag pair j and the kth background pixel feature in the negative bag ; τ is the temperature scaling coefficient, which is used to adjust the distribution of similarities between positive and negative sample packages to avoid excessive punishment of negative package similarity; γ is the proportion of classification loss in the total loss.

5. The Transformer-based twin multi-instance learning hyperspectral target detection method according to claim 2, characterized in that: The specific method of iteratively training the Transformer-based twin multi-instance neural network W in step 3 is: Step (3a) Initialize the training process based on the Transformer-based twin multi-instance neural network W constructed in step 2, set the current number of iterations to t, the maximum number of iterations to T, T ≥ 100, and set the Transformer-based twin multi-instance neural network under the current parameter state to W t ; At the beginning of training, set the initial state to t = 1; Step (3b) is to convert the training set As the current Transformer-based twin multi-instance neural network W t The input is forward propagated, where (B1, B2) i represents the i-th sample package pair in the training set, is a positive sample bag pair, is a negative sample package pair; the spectral embedding module extracts the spectral features of the pixels in each sample package of the training set to obtain the pixel spectral feature set All positive samples in feature set F are embedded in E + All of them are input into the semantic feature extraction module for processing and then fused into representative target features e + , and Gaussian perturbation enhancement is performed on the fusion weights to generate representative target features equivalent to the number of background pixel features; the negative sample package embedding directly releases the background pixel features e contained in the negative sample package - ; F is transformed into a set in represent Representative target features, represent The representative target feature package after Gaussian perturbation enhancement, represent Released background pixel feature package; Step (3c) Each sample packet in set F' has features (e1, e2) i Participate in calculating the contrast loss Loss cts ; and Participate in classifier training and calculate cross entropy loss Loss cls ; Get the total loss Then, based on the loss function Calculate the neural network W t The parameter gradient of the Transformer-based twin multi-instance neural network W is calculated using the stochastic gradient descent (SGD) algorithm. w Update to obtain the updated current Transformer-based twin multi-instance neural network W t The update formulas of the parameters are: Where η represents the gradient descent parameter, Represents the current Transformer-based twin multi-instance neural network W t Parameters, represents the parameters of the Transformer-based twin multi-instance neural network W before updating, represents the partial derivative operation; Step (3d) determines whether the current number of iterations t is equal to the maximum number of iterations T. If so, the trained multi-instance neural network W for self-attention semantic modeling is obtained. * ; Otherwise, set t = t + 1 and return to step (3b) and repeat.

6. A Transformer-based twin multi-instance learning hyperspectral target detection device, characterized in that: include: Memory for storing computer programs; A processor is used to implement the Transformer-based twin multi-instance learning hyperspectral target detection method described in any one of steps 1 to 4 when executing the computer program.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it is possible to perform hyperspectral target detection based on a Transformer twin multi-instance learning hyperspectral target detection method.

Images