Transform-based unsupervised cross-spectrum identification pairing and full-spectrum spectrum synthesis method

Abstract

The invention discloses an unsupervised cross-spectrum identification pairing and full-spectrum spectrum synthesis method based on Transform, and relates to the technical field of hyperspectral image processing and computer vision, in particular to a cross-spectrum identification pairing and full-spectrum spectrum synthesis method. The invention aims to solve the problems that in the prior art, the hyperspectral target detection precision is low, weak spectrum differences are difficult to distinguish, and spectrums of different spectrum segments are difficult to fuse. The method comprises the following steps: constructing a space-spectrum attention fusion mechanism, extracting a spectrum curve, and calculating a correlation weight between a neighborhood and a center spectrum; carrying out deep modeling on the spectrum sequence by utilizing Transform; and a self-supervised learning strategy is combined to train an encoder, and pixel-level target detection is realized by using cosine similarity. And extracting group spectral features and carrying out cross-image target type matching by adopting a Hungary algorithm. And finally, generating a continuous and complete high-quality full-spectrum spectrum through non-synchronous target registration and a full-spectrum weighted fusion framework.

Description

Technical Field

[0001] This invention relates to the fields of hyperspectral image processing and computer vision technology, and in particular to a method for cross-spectral recognition and pairing and full-spectral spectral synthesis. Background Technology

[0002] With the continuous development of remote sensing, drone, and sensor technologies, hyperspectral imaging has gradually become an important means of target identification and monitoring. Hyperspectral images are characterized by extremely high spectral resolution, capable of acquiring spectral information of targets across hundreds of consecutive bands. Therefore, compared to traditional RGB or multispectral images, hyperspectral data has unparalleled advantages in distinguishing the material composition of target objects, precise identification, and detecting minute differences.

[0003] Currently, Earth observation models are undergoing a profound transformation from traditional single-satellite detailed surveys to multi-satellite network collaborative observations. Single satellites, limited by their orbital return periods, struggle to achieve high-frequency revisits to specific areas, failing to meet the continuous monitoring needs of dynamic targets. Satellite clusters or constellation systems, through the collaborative deployment and self-organizing network of multiple satellites in different orbits, can significantly improve the temporal resolution of observations, enabling wide-area inspections, regional surveys, and tracking surveillance of key areas. This technological evolution makes "relay-style" continuous tracking of dynamic targets possible. However, this also brings new technical challenges: once a target is captured by satellite A in a constellation at a certain time and within a certain spectral range, how can it be accurately re-identified and synthesized from subsequent images acquired by satellite B, which may have different spectral ranges and variations in target attitude, viewing angle, illumination, and atmospheric conditions? This challenge faces the following difficulties:

[0004] The spectral features are subtle, making it difficult to distinguish similar targets, and traditional methods struggle to capture local spectral details.

[0005] Existing deep learning-based hyperspectral target detection methods are mostly supervised learning methods, requiring a large number of labeled target and background samples. However, hyperspectral image annotation is costly and difficult, especially since accurately labeled data is scarce in real-world scenarios, limiting the applicability of these methods in unlabeled environments and making it difficult to meet practical application needs.

[0006] The spectral bands acquired by different sensors (such as visible light and infrared) have differences in viewing angle, illumination, and atmospheric attenuation, making data fusion difficult. Existing technologies struggle to effectively extract features within a very small neighborhood and achieve high-precision matching and fusion across spectral bands. Summary of the Invention

[0007] The purpose of this invention is to solve the problems of low accuracy in hyperspectral target detection, difficulty in distinguishing weak spectral differences, and difficulty in fusion of different spectral bands in the existing technology. It proposes an unsupervised cross-spectral identification and pairing and full-spectral synthesis method based on Transformer to achieve high-precision target detection, cross-spectral target type matching, and full-spectral data generation.

[0008] The specific process of an unsupervised cross-spectral identification and pairing and full-spectral synthesis method based on Transformer is as follows:

[0009] Step 1: Acquire the hyperspectral image to be tested (image 1), and construct the original spectral matrix based on hyperspectral image 1. ; Obtain the hyperspectral image 2 to be tested, and construct the original spectral matrix based on the hyperspectral image 2. The first hyperspectral image to be tested is a visible light image; the second hyperspectral image to be tested is a short-wave infrared image; there are two types of targets to be detected in the first hyperspectral image to be tested. and ;

[0010] There are two types of targets to be detected in the hyperspectral image 2. and ;

[0011] Step 2: Process the original spectral matrix For each pixel spectrum, a spatial-spectral attention weight matrix based on neighborhood spectral relationships is constructed; the fused spectral matrix is ​​then obtained based on the spatial-spectral attention weight matrix. For the original spectral matrix For each pixel spectrum, a spatial-spectral attention weight matrix based on neighborhood spectral relationships is constructed; the fused spectral matrix is ​​then obtained based on the spatial-spectral attention weight matrix. ;

[0012] Step 3: Analyze the fused spectral matrix Each spectrum is divided into overlapping spectral segments to obtain overlapping patches for each spectrum; each spectrum after overlapping spectral segmentation is mapped to a feature vector through a linear embedding layer; the spectral matrix is ​​then fused. The feature vectors corresponding to all spectra in the sequence are combined, and a learnable global embedding is concatenated at the beginning of the sequence. , to obtain the sequence ;Change the sequence With learnable location embedding By adding elements one by one, we obtain the sequence. ;

[0013] For the fused spectral matrix Each spectrum is divided into overlapping spectral segments to obtain overlapping patches for each spectrum; each spectrum after overlapping spectral segmentation is mapped to a feature vector through a linear embedding layer; the spectral matrix is ​​then fused. The feature vectors corresponding to all spectra in the sequence are combined, and a learnable global embedding is concatenated at the beginning of the sequence. , to obtain the sequence ;Change the sequence With learnable location embedding By adding elements one by one, we obtain the sequence. ;

[0014] Step 4: From the sequence Random sampling Each pixel spectrum is used as a training sample. For random sampling Gaussian blur is applied to each pixel in the pixel spectrum, and each pixel with Gaussian blur is used as an enhancement sample. ; train samples Input Online Encoder Online encoder Output features; enhance samples Input momentum encoder Momentum encoder Output characteristics; in-circuit encoder The output features are mapped to embedding vectors through the projection head. Momentum encoder The output features are mapped to embedding vectors through the projection head. ;

[0015] From sequence Random sampling Each pixel spectrum is used as a training sample. For random sampling Gaussian blur is applied to each pixel in the pixel spectrum, and each pixel with Gaussian blur is used as an enhancement sample. ; train samples Input Online Encoder Online encoder Output features; enhance samples Input momentum encoder Momentum encoder Output characteristics; in-circuit encoder The output features are mapped to embedding vectors through the projection head. Momentum encoder The output features are mapped to embedding vectors through the projection head. ;

[0016] The projection head consists of a first MLP and a second MLP in sequence;

[0017] Step 5: Based on embedding vectors and embedding vector Calculate the InfoNCE loss to obtain the trained online encoder. Based on embedding vectors and embedding vector Calculate the InfoNCE loss to obtain the trained online encoder. ;

[0018] Step 6: Fuse the spectral matrix Input the trained online encoder The trained online encoder Output features ; Fuse the spectral matrix Input the trained online encoder The trained online encoder Output features ;

[0019] Step 7: Classify the target category of the hyperspectral image 1 to be tested. Prior spectrum and categories Prior spectrum Mapped to embedding vector and ; Target category of the hyperspectral image 2 to be tested Prior spectrum and categories Prior spectrum Mapped to embedding vector and ;

[0020] Step 8: Calculate the features The spectrum of each pixel in and cosine similarity and A target response map is generated based on the cosine similarity of the spectra of all pixels; each pixel in the target response map is normalized to obtain the response value of each pixel; a threshold is set. If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. and If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. and ; detect the target The corresponding pixels are marked with rectangles; the detected target The corresponding pixels are marked with a rectangle;

[0021] Calculation features The spectrum of each pixel in and cosine similarity and A target response map is generated based on the cosine similarity of the spectra of all pixels; each pixel in the target response map is normalized to obtain the response value of each pixel; a threshold is set. If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. and If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. and ; detect the target The corresponding pixels are marked with rectangles; the detected target The corresponding pixels are marked with a rectangle;

[0022] Step 9: Target corresponding to the hyperspectral image 1 to be tested and The group features are calculated from the pixel spectral matrices within the rectangular bounding box; the group features are then Z-score normalized to obtain the normalized target corresponding to the hyperspectral image 1 of the test. Population spectral eigenvectors and target The population spectral eigenvectors;

[0023] Target corresponding to hyperspectral image 2 under test and The group features are calculated from the pixel spectral matrices within the rectangular bounding box; the group features are then Z-score normalized to obtain the target corresponding to the normalized hyperspectral image 2. Population spectral eigenvectors and target The population spectral eigenvectors;

[0024] Calculate the cosine distance between the standardized population spectral feature vectors corresponding to hyperspectral image 1 and hyperspectral image 2, and form the cost matrix based on the cosine distance. The Hungarian algorithm is used to analyze the cost matrix. The optimal match is obtained by solving the problem. The optimal match includes the type correspondence, spectral distance and matching confidence.

[0025] Step 10: Obtain the full-spectrum hyperspectral image based on the optimal match obtained in Step 9 and Step 10.

[0026] The beneficial effects of this invention are as follows:

[0027] This invention discloses an unsupervised cross-spectral identification, pairing detection, and full-spectral synthesis method based on Transformer.

[0028] This method first constructs a spatial-spectral attention fusion mechanism, extracts spectral curves using 3×3 minimal neighborhoods, and calculates the correlation weights between the neighborhood and the central spectrum to enhance local features; it then uses a lightweight Transformer with a cross-token feedforward layer to perform deep modeling of the spectral sequence; finally, it trains the encoder using a self-supervised learning strategy based on spectral instance discrimination, and achieves pixel-level target detection using cosine similarity.

[0029] After the target is detected, the group spectral features are extracted and the Hungarian algorithm is used to perform target type matching across images.

[0030] Finally, by employing asynchronous target registration and a full-spectrum weighted fusion framework, a continuous and complete high-quality full-spectrum spectrum is generated. This invention effectively solves problems such as scarce labeled targets, large differences in target features across multiple spectral bands, and misalignment in multi-source spectral fusion. Attached Figure Description

[0031] Figure 1 This is a flowchart of the present invention; Figure 2 This is a diagram showing the target detection results in the visible light band of a hyperspectral image. Figure 3 This is a diagram showing the target detection results in the short-wave infrared band of a hyperspectral image. Figure 4 This is a matching result diagram of the visible light band in the hyperspectral image; Figure 5 This is a matching result diagram of the short-wave infrared band in the hyperspectral image; Figure 6 This is a diagram showing the spectral fusion results for the entire scene. Figure 7 The image shows the result of the target spectrum fusion. Detailed Implementation

[0032] Specific Implementation Method 1: The specific process of this implementation method for unsupervised cross-spectral identification and pairing and full-spectral synthesis based on Transformer is as follows:

[0033] Step 1: Acquire the hyperspectral image to be tested (image 1), and construct the original spectral matrix based on hyperspectral image 1. ; Obtain the hyperspectral image 2 to be tested, and construct the original spectral matrix based on the hyperspectral image 2. The first hyperspectral image to be tested is a visible light image; the second hyperspectral image to be tested is a short-wave infrared image; there are two types of targets to be detected in the first hyperspectral image to be tested. and ;

[0034] There are two types of targets to be detected in the hyperspectral image 2. and ;

[0035] Step 2: Process the original spectral matrix For each pixel spectrum, a spatial-spectral attention weight matrix based on neighborhood spectral relationships is constructed; the fused spectral matrix is ​​then obtained based on the spatial-spectral attention weight matrix. For the original spectral matrix For each pixel spectrum, a spatial-spectral attention weight matrix based on neighborhood spectral relationships is constructed; the fused spectral matrix is ​​then obtained based on the spatial-spectral attention weight matrix. ;

[0036] Step 3: Analyze the fused spectral matrix Each spectrum is divided into overlapping spectral parts to obtain overlapping patches for each spectrum; each spectrum after overlapping spectral parts is mapped to a feature vector through a linear embedding layer (fully connected layer); the spectral matrix is ​​then fused. The feature vectors corresponding to all spectra in the sequence are combined, and a learnable global embedding is concatenated at the beginning of the sequence. , to obtain the sequence ;Change the sequence With learnable location embedding By adding elements one by one, we obtain the sequence. ;

[0037] For the fused spectral matrix Each spectrum is divided into overlapping spectral parts to obtain overlapping patches for each spectrum; each spectrum after overlapping spectral parts is mapped to a feature vector through a linear embedding layer (fully connected layer); the spectral matrix is ​​then fused. The feature vectors corresponding to all spectra in the sequence are combined, and a learnable global embedding is concatenated at the beginning of the sequence. , to obtain the sequence ;Change the sequence With learnable location embedding By adding elements one by one, we obtain the sequence. ;

[0038] Step 4: From the sequence Random sampling Each pixel spectrum is used as a training sample. For random sampling Gaussian blur is applied to each pixel in the pixel spectrum, and each pixel with Gaussian blur is used as an enhancement sample. ; train samples Input Online Encoder Online encoder Output features; enhance samples Input momentum encoder Momentum encoder Output characteristics; in-circuit encoder The output features are mapped to embedding vectors through the projection head. Momentum encoder The output features are mapped to embedding vectors through the projection head. ;

[0039] From sequence Random sampling Each pixel spectrum is used as a training sample. For random sampling Gaussian blur is applied to each pixel in the pixel spectrum, and each pixel with Gaussian blur is used as an enhancement sample. ; train samples Input Online Encoder Online encoder Output features; enhance samples Input momentum encoder Momentum encoder Output characteristics; in-circuit encoder The output features are mapped to embedding vectors through the projection head. Momentum encoder The output features are mapped to embedding vectors through the projection head. ;

[0040] The projection head consists of a first MLP and a second MLP; the dimensions of the projection head remain unchanged before and after input and output.

[0041] Step 5: Based on embedding vectors and embedding vector Calculate the InfoNCE loss to obtain the trained online encoder. Based on embedding vectors and embedding vector Calculate the InfoNCE loss to obtain the trained online encoder. ;

[0042] Step 6: Fuse the spectral matrix Input the trained online encoder The trained online encoder Output features ; Fuse the spectral matrix Input the trained online encoder The trained online encoder Output features ;

[0043] Step 7: Classify the target category of the hyperspectral image 1 to be tested. Prior spectrum and categories Prior spectrum Mapped to embedding vector and ; Target category of the hyperspectral image 2 to be tested Prior spectrum and categories Prior spectrum Mapped to embedding vector and ;

[0044] Step 8: Calculate the features The spectrum of each pixel in and cosine similarity and A target response map is generated based on the cosine similarity of the spectra of all pixels; each pixel in the target response map is normalized to obtain the response value of each pixel; a threshold is set. If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. and If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. and ; detect the target The corresponding pixels are marked with rectangles; the detected target The corresponding pixels are marked with a rectangle;

[0045] Calculation features The spectrum of each pixel in and cosine similarity and A target response map is generated based on the cosine similarity of the spectra of all pixels; each pixel in the target response map is normalized to obtain the response value of each pixel; a threshold is set. If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. and If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. and ; detect the target The corresponding pixels are marked with rectangles; the detected target The corresponding pixels are marked with a rectangle;

[0046] Step 9: Target corresponding to the hyperspectral image 1 to be tested and The group features are calculated from the pixel spectral matrices within the rectangular bounding box; the group features are then Z-score normalized to obtain the normalized target corresponding to the hyperspectral image 1 of the test. Population spectral eigenvectors and target The population spectral eigenvectors;

[0047] Target corresponding to hyperspectral image 2 under test and The group features are calculated from the pixel spectral matrices within the rectangular bounding box; the group features are then Z-score normalized to obtain the target corresponding to the normalized hyperspectral image 2. Population spectral eigenvectors and target The population spectral eigenvectors;

[0048] Calculate the cosine distance between the standardized population spectral feature vectors corresponding to hyperspectral image 1 and hyperspectral image 2, and form the cost matrix based on the cosine distance. The Hungarian algorithm is used to analyze the cost matrix. The optimal match is obtained by solving the problem. The optimal match includes the type correspondence, spectral distance and matching confidence.

[0049] At this point, targets of the same type have been matched in the two hyperspectral images;

[0050] Step 10: Obtain the full-spectrum hyperspectral image based on the optimal match obtained in Step 9 and Step 10.

[0051] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that, in step one, the hyperspectral image to be measured, 1, is acquired, and an original spectral matrix is ​​constructed based on hyperspectral image 1. ; Obtain the hyperspectral image 2 to be tested, and construct the original spectral matrix based on the hyperspectral image 2. The first hyperspectral image to be tested is a visible light image; the second hyperspectral image to be tested is a short-wave infrared image; there are two types of targets to be detected in the first hyperspectral image to be tested. and There are two types of targets to be detected in the hyperspectral image 2. and The specific process is as follows:

[0052] Step 11: Acquire the hyperspectral image to be tested , ;

[0053] There are two types of targets to be detected in the hyperspectral image 1. and ;

[0054] in, It is a real number; Image length, ; Image width, ; For the number of bands, ;

[0055] The hyperspectral image to be tested Included Each pixel is a row, and the spectrum contained in each pixel is... Each band is used as a column, and the dimension is constructed as follows: The original spectral matrix , Used for subsequent spectral enhancement and feature representation learning;

[0056] Steps 1 and 2: Acquire the hyperspectral image to be tested , ;

[0057] There are two types of targets to be detected in the hyperspectral image 2. and ;

[0058] in, It is a real number; Image length, ; Image width, ; For the number of bands, ;

[0059] The hyperspectral image to be tested Included Each pixel is a row, and the spectrum contained in each pixel is... Each band is used as a column, and the dimension is constructed as follows: The original spectral matrix , This is used for subsequent spectral enhancement and feature representation learning. Other steps and parameters are the same as in Specific Implementation Method 1.

[0060] Specific Implementation Method Three: This implementation method differs from Specific Implementation Method One or Two in that, in step two, the original spectral matrix... For each pixel spectrum, a spatial-spectral attention weight matrix based on neighborhood spectral relationships is constructed; the fused spectral matrix is ​​then obtained based on the spatial-spectral attention weight matrix. For the original spectral matrix For each pixel spectrum, a spatial-spectral attention weight matrix based on neighborhood spectral relationships is constructed; the fused spectral matrix is ​​then obtained based on the spatial-spectral attention weight matrix. The specific process is as follows:

[0061] Step 2.1: Process the original spectral matrix For each pixel spectrum, a spatial-spectral attention weight matrix based on neighborhood spectral relationships is constructed; the fused spectral matrix is ​​then obtained based on the spatial-spectral attention weight matrix. The specific process is as follows:

[0062] Step 211: Extract the original spectral matrix Each pixel in Nine spectral vectors within a 3×3 neighborhood , include ; Represents pixels The 3×3 neighborhood of the first Spectral vectors;

[0063] ; ;

[0064] Step 2.12: Calculation With central spectrum The cosine similarity is expressed as:

[0065] ,

[0066] in, Represents the 2-norm; express With central spectrum Cosine similarity;

[0067] Steps 2-13, based on With central spectrum cosine similarity Construct a spatial-spectral attention weight matrix based on neighborhood spectral relationships. ; ;

[0068] Step 214: Obtain the fused spectral matrix based on the spatial-spectral attention weight matrix. ; indicates as:

[0069]

[0070]

[0071] in, Represents the 2-norm; express With central spectrum Cosine similarity; Represents pixels The fused spectral vector; superscript This indicates the transpose; Represents the fused spectral matrix;

[0072] Step 22: Analyze the original spectral matrix. For each pixel spectrum, a spatial-spectral attention weight matrix based on neighborhood spectral relationships is constructed; the fused spectral matrix is ​​then obtained based on the spatial-spectral attention weight matrix. The specific process is as follows:

[0073] Step 221: Extract the original spectral matrix Each pixel in Nine spectral vectors within a 3×3 neighborhood , include ; Represents pixels The 3×3 neighborhood of the first Spectral vectors;

[0074] ; ;

[0075] Step 222, Calculation With central spectrum The cosine similarity is expressed as:

[0076] ,

[0077] in, Represents the 2-norm; express With central spectrum Cosine similarity;

[0078] Steps two through three, based on With central spectrum cosine similarity Construct a spatial-spectral attention weight matrix based on neighborhood spectral relationships. ; ;

[0079] Step 224: Obtain the fused spectral matrix based on the spatial-spectral attention weight matrix. ; indicates as:

[0080]

[0081]

[0082] in, Represents the 2-norm; express With central spectrum Cosine similarity; Represents each pixel The fused spectral vector; superscript This indicates the transpose; This represents the fused spectral matrix.

[0083] Other steps and parameters are the same as in specific implementation method one or two.

[0084] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that, in step three, the fused spectral matrix... Each spectrum is divided into overlapping spectral parts to obtain overlapping patches for each spectrum; each spectrum after overlapping spectral parts is mapped to a feature vector through a linear embedding layer (fully connected layer); the spectral matrix is ​​then fused. The feature vectors corresponding to all spectra in the sequence are combined, and a learnable global embedding is concatenated at the beginning of the sequence. , to obtain the sequence ;Change the sequence With learnable location embedding By adding elements one by one, we obtain the sequence. ;

[0085] For the fused spectral matrix Each spectrum is divided into overlapping spectral parts to obtain overlapping patches for each spectrum; each spectrum after overlapping spectral parts is mapped to a feature vector through a linear embedding layer (fully connected layer); the spectral matrix is ​​then fused. The feature vectors corresponding to all spectra in the sequence are combined, and a learnable global embedding is concatenated at the beginning of the sequence. , to obtain the sequence ;Change the sequence With learnable location embedding By adding elements one by one, we obtain the sequence. ;

[0086] The specific process is as follows:

[0087] Step 31: Analyze the fused spectral matrix Each spectrum is divided into overlapping spectral parts to obtain overlapping patches for each spectrum; each spectrum after overlapping spectral parts is mapped to a feature vector through a linear embedding layer (fully connected layer); the spectral matrix is ​​then fused. The feature vectors corresponding to all spectra in the sequence are combined, and a learnable global embedding is concatenated at the beginning of the sequence. (Unknown, a variable), obtain the sequence ;Change the sequence With learnable location embedding By adding elements one by one, we obtain the sequence. The specific process is as follows:

[0088] Step 3.11: Analyze the fused spectral matrix. Each spectrum is divided into overlapping spectral segments to obtain overlapping patches for each spectrum; the specific process is as follows:

[0089] Fusion spectral matrix After convolution kernel size stride Patch length A 1D deep convolutional layer generates Each spectral embedding label, i.e., a single spectrum is divided into One overlapping spectral patch;

[0090] Step 3.12: After the overlapping spectral segmentation, each spectrum is mapped to a feature vector through a linear embedding layer (fully connected layer); the specific process is as follows:

[0091] Each length is The spectral patch, after passing through the fully connected layer, is mapped to... eigenvectors; Representing dimension, ;

[0092] Each spectrum, after passing through the fully connected layer, is mapped to... eigenvectors;

[0093] Step 3.13: Fusion of the Spectral Matrix All spectra After passing through the fully connected layer, the resulting feature vectors are arranged into a sequence, and a learnable global embedding is concatenated at the beginning of the sequence. (Unknown, a variable), obtain the sequence ;

[0094] will sequence With learnable location embedding By adding elements one by one, we obtain the sequence. ;

[0095] , ;

[0096] Step 3.2: Analyze the fused spectral matrix. Each spectrum is divided into overlapping spectral parts to obtain overlapping patches for each spectrum; each spectrum after overlapping spectral parts is mapped to a feature vector through a linear embedding layer (fully connected layer); the spectral matrix is ​​then fused. The feature vectors corresponding to all spectra in the sequence are combined, and a learnable global embedding is concatenated at the beginning of the sequence. (Unknown, a variable), obtain the sequence ;Change the sequence With learnable location embedding By adding elements one by one, we obtain the sequence. The specific process is as follows:

[0097] Step 321: Analyze the fused spectral matrix Each spectrum is divided into overlapping spectral segments to obtain overlapping patches for each spectrum; the specific process is as follows:

[0098] Fusion spectral matrix After convolution kernel size stride Patch length A 1D deep convolutional layer generates Each spectral embedding label, i.e., a single spectrum is divided into One overlapping spectral patch;

[0099] Step 3.22: After the overlapping spectral segmentation, each spectrum is mapped to a feature vector through a linear embedding layer (fully connected layer); the specific process is as follows:

[0100] Each length is The spectral patch, after passing through the fully connected layer, is mapped to... eigenvectors; Representing dimension, ;

[0101] Each spectrum, after passing through the fully connected layer, is mapped to... eigenvectors;

[0102] Step 3: Fusion of the spectral matrix All spectra After passing through the fully connected layer, the resulting feature vectors are arranged into a sequence, and a learnable global embedding is concatenated at the beginning of the sequence. , to obtain the sequence ;

[0103] will sequence With learnable location embedding By adding elements one by one, we obtain the sequence. ;

[0104] , .

[0105] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0106] Specific Implementation Method Five: This implementation method differs from one of Specific Implementation Methods One to Four in that, in step four, from the sequence... Random sampling Each pixel spectrum is used as a training sample. For random sampling Gaussian blur is applied to each pixel in the pixel spectrum, and each pixel with Gaussian blur is used as an enhancement sample. ; train samples Input Online Encoder Online encoder Output features; enhance samples Input momentum encoder Momentum encoder Output characteristics; in-circuit encoder The output features are mapped to embedding vectors through the projection head. Momentum encoder The output features are mapped to embedding vectors through the projection head. ;

[0107] From sequence Random sampling Each pixel spectrum is used as a training sample. For random sampling Gaussian blur is applied to each pixel in the pixel spectrum, and each pixel with Gaussian blur is used as an enhancement sample. ; train samples Input Online Encoder Online encoder Output features; enhance samples Input momentum encoder Momentum encoder Output characteristics; in-circuit encoder The output features are mapped to embedding vectors through the projection head. Momentum encoder The output features are mapped to embedding vectors through the projection head. The projection head consists of a first MLP and a second MLP in sequence; the dimensions of the projection head remain unchanged before and after input and output.

[0108] The specific process is as follows:

[0109] Step 41: From the sequence Random sampling Each pixel spectrum is used as a training sample. For random sampling Gaussian blur is applied to each pixel in the pixel spectrum, and each pixel with Gaussian blur is used as an enhancement sample. ; train samples Input Online Encoder Online encoder Output features; enhance samples Input momentum encoder Momentum encoder Output characteristics; in-circuit encoder The output features are mapped to embedding vectors through the projection head. Momentum encoder The output features are mapped to embedding vectors through the projection head. The specific process is as follows:

[0110] Step 4.11: From the sequence Random sampling Each pixel spectrum is used as a training sample. For random sampling Gaussian blur is applied to each pixel in the pixel spectrum, and each pixel with Gaussian blur is used as an enhancement sample. ;

[0111] Step 4.12: Transfer the training samples Input Online Encoder Online encoder Output features; the specific process is as follows:

[0112] Training samples Features are obtained after the normalization layer. ;feature Features are obtained through a multi-head attention mechanism (8 heads). ;feature and Features are obtained by adding elements one by one. ;feature Features are obtained after the normalization layer. ;feature Features are obtained after a fully connected layer ;feature Features are obtained through 1D depthwise convolution. ;feature Features are obtained after a fully connected layer Features Features are obtained after the normalization layer. ;feature Features are obtained through a multi-head attention mechanism (8 heads). ;feature and characteristics Features are obtained by adding elements one by one. ;feature Features are obtained after the normalization layer. ;feature Features are obtained after a fully connected layer ;feature Features are obtained through 1D depthwise convolution. ;feature Features are obtained after a fully connected layer ;

[0113] Step 413: Enhance the sample Input momentum encoder Momentum encoder Output features; the specific process is as follows:

[0114] Enhanced Samples Features are obtained after the normalization layer. ;feature Features are obtained through a multi-head attention mechanism (8 heads). ;feature and Features are obtained by adding elements one by one. ;feature Features are obtained after the normalization layer. ;feature Features are obtained after a fully connected layer ;feature Features are obtained through 1D depthwise convolution. ;feature Features are obtained after a fully connected layer Features Features are obtained after the normalization layer. ;feature Features are obtained through a multi-head attention mechanism (8 heads). ;feature and Features are obtained by adding elements one by one. ;feature Features are obtained after the normalization layer. ;feature Features are obtained after a fully connected layer ;feature Features are obtained through 1D depthwise convolution. ;feature Features are obtained after a fully connected layer ;

[0115] Step 414: Online Encoder The output features are mapped to embedding vectors through the projection head. ;

[0116] Momentum encoder The output features are mapped to embedding vectors through the projection head. ;

[0117] The projection head consists of a first MLP and a second MLP; the dimensions of the projection head remain unchanged before and after input and output.

[0118] Step 42, from the sequence Random sampling Each pixel spectrum is used as a training sample. For random sampling Gaussian blur is applied to each pixel in the pixel spectrum, and each pixel with Gaussian blur is used as an enhancement sample. ; train samples Input Online Encoder Online encoder Output features; enhance samples Input momentum encoder Momentum encoder Output characteristics; in-circuit encoder The output features are mapped to embedding vectors through the projection head. Momentum encoder The output features are mapped to embedding vectors through the projection head. The specific process is as follows:

[0119] Step 421, from the sequence Random sampling Each pixel spectrum is used as a training sample. ;

[0120] Random sampling Gaussian blur is applied to each pixel in the pixel spectrum, and each pixel with Gaussian blur is used as an enhancement sample. ;

[0121] Step 422: Transfer the training samples Input Online Encoder Online encoder Output features; the specific process is as follows:

[0122] Training samples Features are obtained after the normalization layer. ;feature Features were obtained through a multi-head attention mechanism (8 heads). ;feature and Features are obtained by adding elements one by one. ;feature Features are obtained after the normalization layer. ;feature Features are obtained after a fully connected layer ;feature Features are obtained through 1D depthwise convolution. ;feature Features are obtained after a fully connected layer Features Features are obtained after the normalization layer. ;feature Features were obtained through a multi-head attention mechanism (8 heads). ;feature and characteristics Features are obtained by adding elements one by one. ;feature Features are obtained after the normalization layer. ;feature Features are obtained after a fully connected layer ;feature Features are obtained through 1D depthwise convolution. ;feature Features are obtained after a fully connected layer ;

[0123] Steps 4, 2, and 3: Enhance the sample Input momentum encoder Momentum encoder Output features; the specific process is as follows:

[0124] Enhanced Samples Features are obtained after the normalization layer. ;feature Features were obtained through a multi-head attention mechanism (8 heads). ;feature and Features are obtained by adding elements one by one. ;feature Features are obtained after the normalization layer. ;feature Features are obtained after a fully connected layer ;feature Features are obtained through 1D depthwise convolution. ;feature Features are obtained after a fully connected layer Features Features are obtained after the normalization layer. ;feature Features were obtained through a multi-head attention mechanism (8 heads). ;feature and Features are obtained by adding elements one by one. ;feature Features are obtained after the normalization layer. ;feature Features are obtained after a fully connected layer ;feature Features are obtained through 1D depthwise convolution. ;feature Features are obtained after a fully connected layer ;

[0125] Step 424, Online Encoder The output features are mapped to embedding vectors through the projection head. ;

[0126] Momentum encoder The output features are mapped to embedding vectors through the projection head. ;

[0127] The projection head consists of a first MLP and a second MLP; the dimensions of the projection head remain unchanged before and after input and output.

[0128] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0129] Specific Implementation Method Six: This implementation method differs from Specific Implementation Methods One to Five in that step five is based on embedding vectors. and embedding vector Calculate the InfoNCE loss to obtain the trained online encoder. Based on embedding vectors and embedding vector Calculate the InfoNCE loss to obtain the trained online encoder. The specific process is as follows:

[0130] Step 51: Based on embedding vectors and embedding vector Calculate the InfoNCE loss to obtain the trained online encoder. The specific process is as follows:

[0131] Step 5.11: Embed the vector The negative samples are sequentially stored in a FIFO queue Q of length K=1000 to form a negative sample set. ;in, Indicates the first One negative sample;

[0132] Step 512: Based on embedding vectors and embedding vector Calculate the InfoNCE loss; expressed as:

[0133]

[0134] in, Indicates the first InfoNCE loss of the spectrum per pixel, Indicates the temperature coefficient. Indicates the first One negative sample;

[0135] Optimization of online encoder parameters based on InfoNCE loss This loss allows the model to face "the current batch of positive samples + a massive number of negative samples in the queue" in each training iteration, thus improving the feature discrimination ability.

[0136] Momentum encoder parameters Instead of performing independent gradient updates, the momentum encoder parameters are updated solely through a weighted average of the parameters from the synchronous online encoder. ; indicates as:

[0137]

[0138] in, Indicates the momentum coefficient. ;

[0139] Step 513: Repeat steps 511 and 512 until all training samples are available. corresponding embedding vector After all operations are completed, the trained online encoder is obtained. The encoder can extract high-dimensional spectral features of pixels;

[0140] Step 52: Based on embedding vectors and embedding vector Calculate the InfoNCE loss to obtain the trained online encoder. The specific process is as follows:

[0141] Step 521: Embed the vector The negative samples are sequentially stored in a FIFO queue Q of length K=1000 to form a negative sample set. ;

[0142] in, Indicates the first One negative sample;

[0143] Step 522: Based on embedding vectors and embedding vector Calculate the InfoNCE loss; expressed as:

[0144]

[0145] in, Indicates the first InfoNCE loss of the spectrum per pixel, Indicates the temperature coefficient. Indicates the first One negative sample;

[0146] Optimization of online encoder parameters based on InfoNCE loss This loss allows the model to face "the current batch of positive samples + a massive number of negative samples in the queue" in each training iteration, thus improving the feature discrimination ability.

[0147] Momentum encoder parameters Instead of performing independent gradient updates, the momentum encoder parameters are updated solely through a weighted average of the parameters from the synchronous online encoder. ; indicates as:

[0148]

[0149] in, Indicates the momentum coefficient. ;

[0150] Step 523: Repeat steps 521 and 522 until all training samples are available. corresponding embedding vector After all operations are completed, the trained online encoder is obtained. The encoder can extract high-dimensional spectral features of pixels.

[0151] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0152] Specific Implementation Method Seven: This implementation method differs from one of Specific Implementation Methods One to Six in that, in step seven, the target category of the hyperspectral image 1 to be tested is... Prior spectrum and categories Prior spectrum Mapped to embedding vector and ; Target category of the hyperspectral image 2 to be tested Prior spectrum and categories Prior spectrum Mapped to embedding vector and The specific process is as follows:

[0153] Step 71: Classify the target category of the hyperspectral image 1 to be tested. Prior spectrum Mapped to embedding vector The specific process is as follows:

[0154] Construct the original spectral matrix based on the hyperspectral image to be tested;

[0155] Each spectrum in the fused spectral matrix is ​​divided into overlapping spectral patches to obtain overlapping patches for each spectrum. Each spectrum after overlapping spectral partitioning is mapped to a feature vector through a linear embedding layer (fully connected layer). The feature vectors corresponding to all spectra in the fused spectral matrix are combined into a sequence, and a learnable global embedding is concatenated at the beginning of the sequence to obtain the final sequence. The final sequence is then element-wise added to the learnable positional embedding to obtain the final sequence. ;

[0156] will sequence Input the online encoder trained in step 51 Obtain the embedding vector ;

[0157] Step 72: Classify the target category of the hyperspectral image 1 to be tested. Prior spectrum Mapped to embedding vector The specific process is as follows:

[0158] Construct the original spectral matrix based on the hyperspectral image to be tested;

[0159] Each spectrum in the fused spectral matrix is ​​divided into overlapping spectral patches to obtain overlapping patches for each spectrum. Each spectrum after overlapping spectral partitioning is mapped to a feature vector through a linear embedding layer (fully connected layer). The feature vectors corresponding to all spectra in the fused spectral matrix are combined into a sequence, and a learnable global embedding is concatenated at the beginning of the sequence to obtain the final sequence. The final sequence is then element-wise added to the learnable positional embedding to obtain the final sequence. ;

[0160] will sequence Input the online encoder trained in step 51 Obtain the embedding vector ;

[0161] Step 73: Classify the target category of the hyperspectral image 2 to be tested. Prior spectrum Mapped to embedding vector The specific process is as follows:

[0162] The original spectral matrix is ​​constructed based on the hyperspectral image to be tested.

[0163] Each spectrum in the fused spectral matrix is ​​divided into overlapping spectral patches to obtain overlapping patches for each spectrum. Each spectrum after overlapping spectral partitioning is mapped to a feature vector through a linear embedding layer (fully connected layer). The feature vectors corresponding to all spectra in the fused spectral matrix are combined into a sequence, and a learnable global embedding is concatenated at the beginning of the sequence to obtain the final sequence. The final sequence is then element-wise added to the learnable positional embedding to obtain the final sequence. ;

[0164] Will Input the online encoder trained in step 52 Obtain the embedding vector ;

[0165] Step 74: Classify the target category of the hyperspectral image 2 to be tested. Prior spectrum Mapped to embedding vector The specific process is as follows:

[0166] The original spectral matrix is ​​constructed based on the hyperspectral image to be tested.

[0167] Each spectrum in the fused spectral matrix is ​​divided into overlapping spectral patches to obtain overlapping patches for each spectrum. Each spectrum after overlapping spectral partitioning is mapped to a feature vector through a linear embedding layer (fully connected layer). The feature vectors corresponding to all spectra in the fused spectral matrix are combined into a sequence, and a learnable global embedding is concatenated at the beginning of the sequence to obtain the final sequence. The final sequence is then element-wise added to the learnable positional embedding to obtain the final sequence. ;

[0168] Will Input the online encoder trained in step 52 Obtain the embedding vector .

[0169] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0170] Specific Implementation Method Eight: This implementation method differs from any of Specific Implementation Methods One to Seven in that the calculation of features in step eight is... The spectrum of each pixel in and cosine similarity and A target response map is generated based on the cosine similarity of the spectra of all pixels; each pixel in the target response map is normalized to obtain the response value of each pixel; a threshold is set. If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. and If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. and ; detect the target The corresponding pixels are marked with rectangles; the detected target The corresponding pixels are marked with a rectangle;

[0171] Calculation features The spectrum of each pixel in and cosine similarity and A target response map is generated based on the cosine similarity of the spectra of all pixels; each pixel in the target response map is normalized to obtain the response value of each pixel; a threshold is set. If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. and If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. and ; detect the target The corresponding pixels are marked with rectangles; the detected target The corresponding pixels are marked with a rectangle; the specific process is as follows:

[0172] Step 81: Calculate the characteristics The spectrum of each pixel in and cosine similarity and A target response map is generated based on the cosine similarity of the spectra of all pixels; each pixel in the target response map is normalized to obtain the response value of each pixel; a threshold is set. If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. and If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. and ; detect the target The corresponding pixels are marked with rectangles; the detected target The corresponding pixels are marked with a rectangle; the specific process is as follows:

[0173] Step 811: Calculate the characteristics Spectrum of each pixel and cosine similarity , is represented as:

[0174] ,

[0175] Features All pixel spectra and cosine similarity Rearrange in descending order, and calculate the cosine similarity of each row. satisfy Move to the next row and rearrange in descending order until all cosine similarities are reached. After sorting, a two-dimensional target response map is obtained;

[0176] Each pixel in the target response map is normalized to obtain the response value of each pixel; the higher the response value, the more likely the pixel belongs to the target category.

[0177] Set threshold If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. or If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. or ;

[0178] Step 812: Calculate the characteristics Spectrum of each pixel and cosine similarity , is represented as:

[0179] ,

[0180] Features All pixel spectra and cosine similarity Rearrange in descending order, and calculate the cosine similarity of each row. satisfy Move to the next row and rearrange in descending order until all cosine similarities are reached. After sorting, a two-dimensional target response map is obtained;

[0181] Each pixel in the target response map is normalized to obtain the response value of each pixel; the higher the response value, the more likely the pixel belongs to the target category.

[0182] Set threshold If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. or If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. or ;

[0183] Step 813: The targets detected in Steps 811 and 812... The corresponding pixels are marked with a rectangle;

[0184] The targets detected in steps 811 and 812 The corresponding pixels are marked with a rectangle;

[0185] Step 82: Calculate the characteristics The spectrum of each pixel in and cosine similarity and A target response map is generated based on the cosine similarity of the spectra of all pixels; each pixel in the target response map is normalized to obtain the response value of each pixel; a threshold is set. If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. and If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. and ; detect the target The corresponding pixels are marked with rectangles; the detected target The corresponding pixels are marked with a rectangle; the specific process is as follows:

[0186] Step 821: Calculate the characteristics Spectrum of each pixel and cosine similarity , is represented as:

[0187] ,

[0188] Features All pixel spectra and cosine similarity Rearrange in descending order, and calculate the cosine similarity of each row. satisfy Move to the next row and rearrange in descending order until all cosine similarities are reached. After sorting, a two-dimensional target response map is obtained;

[0189] Each pixel in the target response map is normalized to obtain the response value of each pixel; the higher the response value, the more likely the pixel belongs to the target category.

[0190] Set threshold If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. or If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. or ;

[0191] Step 822: Calculate the characteristics Spectrum of each pixel and cosine similarity , is represented as:

[0192] ,

[0193] Features All pixel spectra and cosine similarity Rearrange in descending order, and calculate the cosine similarity of each row. satisfy Move to the next row and rearrange in descending order until all cosine similarities are reached. After sorting, a two-dimensional target response map is obtained;

[0194] Each pixel in the target response map is normalized to obtain the response value of each pixel; the higher the response value, the more likely the pixel belongs to the target category.

[0195] Set threshold If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. or If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. or ;

[0196] Step 823: The targets detected in Steps 821 and 822... The corresponding pixels are marked with a rectangle;

[0197] The targets detected in steps 821 and 822 The corresponding pixels are marked with a rectangle.

[0198] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0199] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods One to Eight in that, in step nine, the target corresponding to the hyperspectral image 1 to be tested... and The group features are calculated from the pixel spectral matrices within the rectangular bounding box; the group features are then Z-score normalized to obtain the normalized target corresponding to the hyperspectral image 1 of the test. Population spectral eigenvectors and target The group spectral feature vector; the target corresponding to the hyperspectral image 2 under test. and The group features are calculated from the pixel spectral matrices within the rectangular bounding box; the group features are then Z-score normalized to obtain the target corresponding to the normalized hyperspectral image 2. Population spectral eigenvectors and target The population spectral feature vector is calculated; the cosine distance between the normalized population spectral feature vector corresponding to the hyperspectral image 1 and the population spectral feature vector corresponding to the hyperspectral image 2 is calculated, and the cost matrix is ​​formed based on the cosine distance. The Hungarian algorithm is used to analyze the cost matrix. The optimal match is obtained by solving the problem. The optimal match includes the type correspondence, spectral distance, and match confidence. At this point, targets of the same class in the two hyperspectral images have been matched. The specific process is as follows:

[0200] Step 91: Target corresponding to the hyperspectral image 1 to be tested and Pixel spectral matrix within the rectangular frame and Calculate the population features separately; perform Z-score normalization on the population features to obtain the normalized population spectral feature vector corresponding to the measured hyperspectral image 1; the specific process is as follows:

[0201] 1) Population characteristics include mean spectrum, standard deviation spectrum, coefficient of variation, and mean of first derivative;

[0202] Mean spectrum : ;

[0203] in, This represents a single pixel within the bounding boxes of all objects of the same type in an image. This represents the total number of pixels of the same type in an image;

[0204] Standard deviation spectrum : ;

[0205] coefficient of variation : ;

[0206] First derivative mean : ;

[0207] in, Represents the first of the same type of target in a graph. The average value of each band, Represents the first of the same type of target in a graph. The average value of each band;

[0208] The mean spectrum, standard deviation spectrum, coefficient of variation, and mean of the first derivative are combined into a vector to form the population characteristics;

[0209] 2) Perform Z-score normalization on the population features to obtain the normalized population spectral feature vector corresponding to the measured hyperspectral image 1; the specific process is as follows:

[0210] Z-score normalization is performed on the population features to obtain the normalized population spectral feature vector corresponding to the hyperspectral image 1 under test. Each class of hyperspectral image 1 obtains a feature vector.

[0211] Step 92: Target corresponding to the hyperspectral image 2 under test and The group characteristics are calculated from the pixel spectral matrix within the rectangular frame;

[0212] Z-score normalization is performed on the population features to obtain the normalized population spectral feature vector corresponding to the hyperspectral image 2 under test. Each class of hyperspectral image 2 obtains a feature vector.

[0213] Step 93: Calculate the cosine distance between the standardized population spectral feature vector corresponding to the hyperspectral image 1 and the population spectral feature vector corresponding to the hyperspectral image 2, and form the cost matrix based on the cosine distance. The Hungarian algorithm is used to analyze the cost matrix. The solution is performed to obtain the optimal match, which includes type correspondence, spectral distance, and matching confidence. The specific process is as follows:

[0214] Step 931: Calculate the cosine distance between the standardized population spectral feature vector corresponding to the hyperspectral image 1 and the population spectral feature vector corresponding to the hyperspectral image 2, and form the cost matrix based on the cosine distance. The specific process is as follows:

[0215] Cost Matrix It is The matrix, ;

[0216] in, The cosine distance between the first group feature vector of the standardized hyperspectral image 1 and the first group feature vector of the standardized hyperspectral image 2 is represented. The cosine distance between the first group feature vector of the standardized hyperspectral image 1 and the second group feature vector of the standardized hyperspectral image 2 is represented. The cosine distance between the second group feature vector of the standardized hyperspectral image 1 and the first group feature vector of the standardized hyperspectral image 2 is represented. The cosine distance between the second group feature vector of the standardized hyperspectral image 1 and the second group feature vector of the standardized hyperspectral image 2 is represented.

[0217] Step 932: Use the Hungarian algorithm to process the cost matrix. The solution is performed to obtain the optimal match, which includes type correspondence, spectral distance, and matching confidence. The specific process is as follows:

[0218] 1) Cost matrix Each element in the matrix is ​​row-normalized to obtain the row-normalized cost matrix. ;

[0219] 2) The cost matrix after row normalization Each element in the matrix is ​​column-normalized to obtain the column-normalized cost matrix. ;

[0220] 3) The cost matrix after column normalization is normalized using the method of finding the minimum coverage line. The minimum number of coverage lines is obtained through processing.

[0221] If the minimum number of coverage lines is equal to the column-normalized cost matrix If the order is 2, then the column-normalized cost matrix is... The position corresponding to the current zero element (the first image is in the column, the second image is in the row, then there are two rows and two columns, corresponding to the two categories of each image, if...) If it's 0, then it means that category 1 of the first image and category 1 of the second image belong to the same category. If it is 0, it means that category 1 of the first image and category 2 of the second image are of the same category, and so on, which is the optimal match;

[0222] Otherwise, repeat steps 1)-3) until the optimal match is obtained.

[0223] The other steps and parameters are the same as those in one of the specific implementation methods one to eight.

[0224] Specific Implementation Method Ten: This implementation method differs from any of Specific Implementation Methods One to Nine in that, in step ten, the full-spectrum hyperspectral image is obtained based on the optimal matching obtained in step nine; the specific process is as follows:

[0225] 1) Target in two hyperspectral images and The rectangular area is covered by a mask;

[0226] Canny edge detection was performed on one of the two hyperspectral images to obtain a binarized edge image. ;

[0227] Canny edge detection was performed on another image from the two hyperspectral images to obtain a binarized edge image. ;

[0228] Binarized edge image Shi-Tomasi corner detection is performed inside to obtain the feature point set P1;

[0229] Binarized edge image Shi-Tomasi corner detection is performed inside to obtain the feature point set P2;

[0230] The FLANN method is used to filter initial matching pairs from feature point sets P1 and P2, removing pairs with ambiguous matches to obtain the initial matching pair set. Use the RANSAC method to remove the initial set of matching pairs. The anomalies were obtained. and the corresponding affine matrix Using affine matrices Affine transformation is performed on one of the two hyperspectral images to obtain... This enables spatial alignment of the Region of Interest (ROI) between two hyperspectral images.

[0231] 2) The 81-120 bands of the visible light image after ROI spatial alignment overlap with the 1-40 bands of the shortwave infrared image. A weighted average method is used to process the 81-120 bands of the visible light image and the 1-40 bands of the shortwave infrared image to obtain a full-spectrum hyperspectral image. The specific process is as follows:

[0232] 21) Calculate the weighted value of a pixel. ; indicates as:

[0233]

[0234] in, It is the signal-to-noise ratio of the sensor for visible light images; It is the signal-to-noise ratio of the sensor for shortwave infrared images; It is a pixel in a visible light image; It is a pixel in a shortwave infrared image; It is the weighted value of a single pixel;

[0235] 22) A five-point Gaussian filter is used to smooth and eliminate spectral abrupt changes at the seams between overlapping and non-overlapping regions, resulting in a full-spectrum hyperspectral image; the specific process is as follows:

[0236] The kernel window size for a five-point Gaussian filter (one-dimensional filtering along bands 1-200) is 5, and the corresponding Gaussian kernel weights are... ;

[0237] The visible light image (bands 81-120) after ROI spatial alignment overlaps with the shortwave infrared image (bands 1-40). The visible light image (band 80) and the shortwave infrared image (band 41) are two seams. Two bands before and after each seam (a total of five bands) are selected as filtering regions. Each pixel in the filtering region is weighted and averaged using a five-point Gaussian kernel to obtain the smoothed visible light image (band 80) and the shortwave infrared image (band 41) after seam processing.

[0238] The weighted values ​​of all pixels in the 1-80 band of the visible light image after seam treatment are obtained. The shortwave infrared images in bands 41-120 after seam processing are stitched together to obtain a complete full-spectrum hyperspectral image in bands 1-200. ;

[0239] 3) Extract the target corresponding to hyperspectral image 1 from the two hyperspectral images. The target corresponding to the average spectrum and hyperspectral image 1 The target corresponding to the average spectrum and hyperspectral image 2 The target corresponding to the average spectrum and hyperspectral image 2 The average spectrum;

[0240] The target corresponding to hyperspectral image 1 The target corresponding to the average spectrum and hyperspectral image 2 The average spectrum is processed in step 2) to obtain a pixel with a band of 1-200.

[0241] The target corresponding to hyperspectral image 1 The target corresponding to the average spectrum and hyperspectral image 2 The average spectrum is processed in step 2) to obtain a pixel with a band of 1-200.

[0242] Insert the target of the corresponding category from the pixels in the two obtained 1-200 bands into The full-spectrum hyperspectral image is obtained by taking the corresponding mask positions in the image. .

[0243] The other steps and parameters are the same as those in any of the specific implementation methods one to nine.

[0244] This invention may have other embodiments. Without departing from the spirit and essence of this invention, those skilled in the art can make various corresponding changes and modifications according to this invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A Transformer-based unsupervised cross-spectral identification and pairing method for full-spectral synthesis, characterized in that: The specific process of the method is as follows: Step 1: Acquire the hyperspectral image to be tested (image 1), and construct the original spectral matrix based on hyperspectral image 1. ; Obtain the hyperspectral image 2 to be tested, and construct the original spectral matrix based on the hyperspectral image 2. ; The hyperspectral image to be tested, image 1, is a visible light image; The second hyperspectral image to be tested is a shortwave infrared image; There are two types of targets to be detected in the hyperspectral image 1. and ; There are two types of targets to be detected in the hyperspectral image 2. and ; Step 2: Process the original spectral matrix For each pixel spectrum, construct a spatial-spectral attention weight matrix based on neighborhood spectral relationships; The fused spectral matrix is ​​obtained based on the spatial-spectral attention weight matrix. ; For the original spectral matrix For each pixel spectrum, construct a spatial-spectral attention weight matrix based on neighborhood spectral relationships; The fused spectral matrix is ​​obtained based on the spatial-spectral attention weight matrix. ; Step 3: Analyze the fused spectral matrix Each spectrum is divided into overlapping spectral segments to obtain overlapping patches for each spectrum; each spectrum after overlapping spectral segmentation is mapped to a feature vector through a linear embedding layer; the spectral matrix is ​​then fused. The feature vectors corresponding to all spectra in the sequence are combined, and a learnable global embedding is concatenated at the beginning of the sequence. , to obtain the sequence ;Change the sequence With learnable location embedding By adding elements one by one, we obtain the sequence. ; For the fused spectral matrix Each spectrum is divided into overlapping spectral segments to obtain overlapping patches for each spectrum; each spectrum after overlapping spectral segmentation is mapped to a feature vector through a linear embedding layer; the spectral matrix is ​​then fused. The feature vectors corresponding to all spectra in the sequence are combined, and a learnable global embedding is concatenated at the beginning of the sequence. , to obtain the sequence ;Change the sequence With learnable location embedding By adding elements one by one, we obtain the sequence. ; Step 4: From the sequence Random sampling Each pixel spectrum is used as a training sample. For random sampling Gaussian blur is applied to each pixel in the pixel spectrum, and each pixel with Gaussian blur is used as an enhancement sample. ; train samples Input Online Encoder Online encoder Output features; enhance samples Input momentum encoder Momentum encoder Output features; Online encoder The output features are mapped to embedding vectors through the projection head. Momentum encoder The output features are mapped to embedding vectors through the projection head. ; From sequence Random sampling Each pixel spectrum is used as a training sample. For random sampling Gaussian blur is applied to each pixel in the pixel spectrum, and each pixel with Gaussian blur is used as an enhancement sample. ; train samples Input Online Encoder Online encoder Output features; enhance samples Input momentum encoder Momentum encoder Output features; Online encoder The output features are mapped to embedding vectors through the projection head. Momentum encoder The output features are mapped to embedding vectors through the projection head. ; The projection head consists of a first MLP and a second MLP in sequence; Step 5: Based on embedding vectors and embedding vector Calculate the InfoNCE loss to obtain the trained online encoder. Based on embedding vectors and embedding vector Calculate the InfoNCE loss to obtain the trained online encoder. ; Step 6: Fuse the spectral matrix Input the trained online encoder The trained online encoder Output features ; fused spectral matrix Input the trained online encoder The trained online encoder Output features ; Step 7: Classify the target category of the hyperspectral image 1 to be tested. Prior spectrum and categories Prior spectrum Mapped to embedding vector and ; The target category of the hyperspectral image 2 to be tested Prior spectrum and categories Prior spectrum Mapped to embedding vector and ; Step 8: Calculate the features The spectrum of each pixel in and cosine similarity and A target response map is generated based on the cosine similarity of the spectra of all pixels; each pixel in the target response map is normalized to obtain the response value of each pixel. Set threshold ; If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. and ; If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. and ; detect the target The corresponding pixels are marked with rectangles; the detected target The corresponding pixels are marked with a rectangle; Calculation features The spectrum of each pixel in and cosine similarity and A target response map is generated based on the cosine similarity of the spectra of all pixels; each pixel in the target response map is normalized to obtain the response value of each pixel. Set threshold ; If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. and ; If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. and ; detect the target The corresponding pixels are marked with rectangles; the detected target The corresponding pixels are marked with a rectangle; Step 9: Target corresponding to the hyperspectral image 1 to be tested and The group characteristics are calculated from the pixel spectral matrix within the rectangular frame; Z-score normalization is applied to the group features to obtain the target corresponding to the normalized hyperspectral image 1 of the test. Population spectral eigenvectors and target The population spectral eigenvectors; Target corresponding to hyperspectral image 2 under test and The group characteristics are calculated from the pixel spectral matrix within the rectangular frame; Z-score normalization is applied to the group features to obtain the target corresponding to the normalized hyperspectral image 2 of the test. Population spectral eigenvectors and target The population spectral eigenvectors; Calculate the cosine distance between the standardized population spectral feature vectors corresponding to hyperspectral image 1 and hyperspectral image 2, and form the cost matrix based on the cosine distance. ; The cost matrix is ​​processed using the Hungarian algorithm. The optimal match is obtained by solving the problem. The optimal match includes the type correspondence, spectral distance and matching confidence. Step 10: Obtain the full-spectrum hyperspectral image based on the optimal match obtained in Step 9 and Step 10.

2. The method for unsupervised cross-spectral identification and pairing and full-spectral synthesis based on Transformer according to claim 1, characterized in that: In step one, the hyperspectral image to be tested, 1, is acquired, and an original spectral matrix is ​​constructed based on hyperspectral image 1. ; Obtain the hyperspectral image 2 to be tested, and construct the original spectral matrix based on the hyperspectral image 2. ; The hyperspectral image to be tested, image 1, is a visible light image; The hyperspectral image 2 to be tested is a short-wave infrared image; there are two types of targets to be detected in the hyperspectral image 1 to be tested. and ; There are two types of targets to be detected in the hyperspectral image 2. and The specific process is as follows: Step 11: Acquire the hyperspectral image to be tested , ; There are two types of targets to be detected in the hyperspectral image 1. and ; in, It is a real number; Image length; Image width; Number of bands; The hyperspectral image to be tested Included Each pixel is a row, and the spectrum contained in each pixel is... Each band is used as a column, and the dimension is constructed as follows: The original spectral matrix , ; Steps 1 and 2: Acquire the hyperspectral image to be tested , ; There are two types of targets to be detected in the hyperspectral image 2. and ; in, It is a real number; Image length; Image width; Number of bands; The hyperspectral image to be tested Included Each pixel is a row, and the spectrum contained in each pixel is... Each band is used as a column, and the dimension is constructed as follows: The original spectral matrix , .

3. The Transformer-based unsupervised cross-spectral identification and pairing and full-spectral synthesis method according to claim 2, characterized in that: In step two, the original spectral matrix For each pixel spectrum, construct a spatial-spectral attention weight matrix based on neighborhood spectral relationships; The fused spectral matrix is ​​obtained based on the spatial-spectral attention weight matrix. For the original spectral matrix For each pixel spectrum, construct a spatial-spectral attention weight matrix based on neighborhood spectral relationships; The fused spectral matrix is ​​obtained based on the spatial-spectral attention weight matrix. The specific process is as follows: Step 2.1: Process the original spectral matrix For each pixel spectrum, construct a spatial-spectral attention weight matrix based on neighborhood spectral relationships; The fused spectral matrix is ​​obtained based on the spatial-spectral attention weight matrix. The specific process is as follows: Step 211: Extract the original spectral matrix Each pixel in Nine spectral vectors within a 3×3 neighborhood , include ; Represents pixels The 3×3 neighborhood of the first Spectral vectors; ； ； Step 2.12: Calculation With central spectrum The cosine similarity is expressed as: ， in, Represents the 2-norm; express With central spectrum Cosine similarity; Steps 2-13, based on With central spectrum cosine similarity Construct a spatial-spectral attention weight matrix based on neighborhood spectral relationships. ; ; Step 214: Obtain the fused spectral matrix based on the spatial-spectral attention weight matrix. ; indicates as: in, Represents the 2-norm; express With central spectrum Cosine similarity; Represents pixels The fused spectral vector; superscript This indicates the transpose; Represents the fused spectral matrix; Step 22: Analyze the original spectral matrix. For each pixel spectrum, a spatial-spectral attention weight matrix based on neighborhood spectral relationships is constructed; the fused spectral matrix is ​​then obtained based on the spatial-spectral attention weight matrix. The specific process is as follows: Step 221: Extract the original spectral matrix Each pixel in Nine spectral vectors within a 3×3 neighborhood , include ; Represents pixels The 3×3 neighborhood of the first Spectral vectors; ； ； Step 222, Calculation With central spectrum The cosine similarity is expressed as: ， in, Represents the 2-norm; express With central spectrum Cosine similarity; Steps two through three, based on With central spectrum cosine similarity Construct a spatial-spectral attention weight matrix based on neighborhood spectral relationships. ; ; Step 224: Obtain the fused spectral matrix based on the spatial-spectral attention weight matrix. ; indicates as: in, Represents the 2-norm; express With central spectrum Cosine similarity; Represents each pixel The fused spectral vector; superscript This indicates the transpose; This represents the fused spectral matrix.

4. The Transformer-based unsupervised cross-spectral identification and pairing and full-spectral synthesis method according to claim 3, characterized in that: In step three, the fusion spectral matrix Each spectrum is divided into overlapping spectral segments to obtain overlapping patches for each spectrum; each spectrum after overlapping spectral segmentation is mapped to a feature vector through a linear embedding layer; the spectral matrix is ​​then fused. The feature vectors corresponding to all spectra in the sequence are combined, and a learnable global embedding is concatenated at the beginning of the sequence. , to obtain the sequence ;Change the sequence With learnable location embedding By adding elements one by one, we obtain the sequence. ; For the fused spectral matrix Each spectrum is divided into overlapping spectral segments to obtain overlapping patches for each spectrum; each spectrum after overlapping spectral segmentation is mapped to a feature vector through a linear embedding layer; the spectral matrix is ​​then fused. The feature vectors corresponding to all spectra in the sequence are combined, and a learnable global embedding is concatenated at the beginning of the sequence. , to obtain the sequence ;Change the sequence With learnable location embedding By adding elements one by one, we obtain the sequence. ; The specific process is as follows: Step 31: Analyze the fused spectral matrix Each spectrum is divided into overlapping spectral segments to obtain overlapping patches for each spectrum; each spectrum after overlapping spectral segmentation is mapped to a feature vector through a linear embedding layer; the spectral matrix is ​​then fused. The feature vectors corresponding to all spectra in the sequence are combined, and a learnable global embedding is concatenated at the beginning of the sequence. , to obtain the sequence ;Change the sequence With learnable location embedding By adding elements one by one, we obtain the sequence. ; The specific process is as follows: Step 3.11: Analyze the fused spectral matrix. Each spectrum is divided into overlapping spectra to obtain overlapping patches for each spectrum. The specific process is as follows: Fusion spectral matrix After convolution kernel size stride Patch length A 1D deep convolutional layer generates Each spectral embedding label, i.e., a single spectrum is divided into One overlapping spectral patch; Step 3.12: Map each spectrum after overlapping spectral segmentation into a feature vector through a linear embedding layer; The specific process is as follows: Each length is The spectral patch, after passing through the fully connected layer, is mapped to... eigenvectors; Representing dimension, ; Each spectrum, after passing through the fully connected layer, is mapped to... eigenvectors; Step 3.13: Fusion of the Spectral Matrix All spectra After passing through the fully connected layer, the resulting feature vectors are arranged into a sequence, and a learnable global embedding is concatenated at the beginning of the sequence. , to obtain the sequence ; will sequence With learnable location embedding By adding elements one by one, we obtain the sequence. ; ， ； Step 3.2: Analyze the fused spectral matrix. Each spectrum is divided into overlapping spectral segments to obtain overlapping patches for each spectrum; each spectrum after overlapping spectral segmentation is mapped to a feature vector through a linear embedding layer; the spectral matrix is ​​then fused. The feature vectors corresponding to all spectra in the sequence are combined, and a learnable global embedding is concatenated at the beginning of the sequence. , to obtain the sequence ;Change the sequence With learnable location embedding By adding elements one by one, we obtain the sequence. The specific process is as follows: Step 321: Analyze the fused spectral matrix Each spectrum is divided into overlapping spectral segments to obtain overlapping patches for each spectrum; the specific process is as follows: Fusion spectral matrix After convolution kernel size stride Patch length A 1D deep convolutional layer generates Each spectral embedding label, i.e., a single spectrum is divided into One overlapping spectral patch; Step 3.22: Map each spectrum after overlapping spectral segmentation into a feature vector through a linear embedding layer; the specific process is as follows: Each length is The spectral patch, after passing through the fully connected layer, is mapped to... eigenvectors; Representing dimension, ; Each spectrum, after passing through the fully connected layer, is mapped to... eigenvectors; Step 3: Fusion of the spectral matrix All spectra After passing through the fully connected layer, the resulting feature vectors are arranged into a sequence, and a learnable global embedding is concatenated at the beginning of the sequence. , to obtain the sequence ; will sequence With learnable location embedding By adding elements one by one, we obtain the sequence. ; ， 。 5. The Transformer-based unsupervised cross-spectral identification and pairing and full-spectral synthesis method according to claim 4, characterized in that: In step four, from the sequence Random sampling Each pixel spectrum is used as a training sample. For random sampling Gaussian blur is applied to each pixel in the pixel spectrum, and each pixel with Gaussian blur is used as an enhancement sample. ; train samples Input Online Encoder Online encoder Output features; enhance samples Input momentum encoder Momentum encoder Output features; Online encoder The output features are mapped to embedding vectors through the projection head. Momentum encoder The output features are mapped to embedding vectors through the projection head. ; From sequence Random sampling Each pixel spectrum is used as a training sample. For random sampling Gaussian blur is applied to each pixel in the pixel spectrum, and each pixel with Gaussian blur is used as an enhancement sample. ; train samples Input Online Encoder Online encoder Output features; enhance samples Input momentum encoder Momentum encoder Output features; Online encoder The output features are mapped to embedding vectors through the projection head. Momentum encoder The output features are mapped to embedding vectors through the projection head. The projection head consists of a first MLP and a second MLP. The specific process is as follows: Step 41: From the sequence Random sampling Each pixel spectrum is used as a training sample. For random sampling Gaussian blur is applied to each pixel in the pixel spectrum, and each pixel with Gaussian blur is used as an enhancement sample. ; train samples Input Online Encoder Online encoder Output features; enhance samples Input momentum encoder Momentum encoder Output features; Online encoder The output features are mapped to embedding vectors through the projection head. Momentum encoder The output features are mapped to embedding vectors through the projection head. ; The specific process is as follows: Step 4.11: From the sequence Random sampling Each pixel spectrum is used as a training sample. For random sampling Gaussian blur is applied to each pixel in the pixel spectrum, and each pixel with Gaussian blur is used as an enhancement sample. ; Step 4.12: Transfer the training samples Input Online Encoder Online encoder Output features; the specific process is as follows: Training samples Features are obtained after the normalization layer. ;feature Features were obtained through a multi-head attention mechanism. ; feature and Features are obtained by adding elements one by one. ;feature Features are obtained after the normalization layer. ;feature Features are obtained after a fully connected layer ;feature Features are obtained through 1D depthwise convolution. ;feature Features are obtained after a fully connected layer ; Features Features are obtained after the normalization layer. ;feature Features were obtained through a multi-head attention mechanism. ;feature and characteristics Features are obtained by adding elements one by one. ;feature Features are obtained after the normalization layer. ;feature Features are obtained after a fully connected layer ;feature Features are obtained through 1D depthwise convolution. ;feature Features are obtained after a fully connected layer ; Step 413: Enhance the sample Input momentum encoder Momentum encoder Output features; the specific process is as follows: Enhanced Samples Features are obtained after the normalization layer. ;feature Features were obtained through a multi-head attention mechanism. ; feature and Features are obtained by adding elements one by one. ;feature Features are obtained after the normalization layer. ;feature Features are obtained after a fully connected layer ;feature Features are obtained through 1D depthwise convolution. ;feature Features are obtained after a fully connected layer ; Features Features are obtained after the normalization layer. ;feature Features were obtained through a multi-head attention mechanism. ;feature and Features are obtained by adding elements one by one. ;feature Features are obtained after the normalization layer. ;feature Features are obtained after a fully connected layer ;feature Features are obtained through 1D depthwise convolution. ;feature Features are obtained after a fully connected layer ; Step 414: Online Encoder The output features are mapped to embedding vectors through the projection head. ; Momentum encoder The output features are mapped to embedding vectors through the projection head. ; The projection head consists of a first MLP and a second MLP in sequence; Step 42, from the sequence Random sampling Each pixel spectrum is used as a training sample. For random sampling Gaussian blur is applied to each pixel in the pixel spectrum, and each pixel with Gaussian blur is used as an enhancement sample. ; train samples Input Online Encoder Online encoder Output features; enhance samples Input momentum encoder Momentum encoder Output features; Online encoder The output features are mapped to embedding vectors through the projection head. Momentum encoder The output features are mapped to embedding vectors through the projection head. ; The specific process is as follows: Step 421, from the sequence Random sampling Each pixel spectrum is used as a training sample. ; Random sampling Gaussian blur is applied to each pixel in the pixel spectrum, and each pixel with Gaussian blur is used as an enhancement sample. ; Step 422: Transfer the training samples Input Online Encoder Online encoder Output features; the specific process is as follows: Training samples Features are obtained after the normalization layer. ;feature Features were obtained through a multi-head attention mechanism. ;feature and Features are obtained by adding elements one by one. ;feature Features are obtained after the normalization layer. ;feature Features are obtained after a fully connected layer ;feature Features are obtained through 1D depthwise convolution. ;feature Features are obtained after a fully connected layer ; Features Features are obtained after the normalization layer. ;feature Features were obtained through a multi-head attention mechanism. ;feature and characteristics Features are obtained by adding elements one by one. ;feature Features are obtained after the normalization layer. ;feature Features are obtained after a fully connected layer ;feature Features are obtained through 1D depthwise convolution. ;feature Features are obtained after a fully connected layer ; Steps 4, 2, and 3: Enhance the sample Input momentum encoder Momentum encoder Output features; the specific process is as follows: Enhanced Samples Features are obtained after the normalization layer. ;feature Features were obtained through a multi-head attention mechanism. ;feature and Features are obtained by adding elements one by one. ;feature Features are obtained after the normalization layer. ;feature Features are obtained after a fully connected layer ;feature Features are obtained through 1D depthwise convolution. ; feature Features are obtained after a fully connected layer ; Features Features are obtained after the normalization layer. ;feature Features were obtained through a multi-head attention mechanism. ;feature and Features are obtained by adding elements one by one. ;feature Features are obtained after the normalization layer. ;feature Features are obtained after a fully connected layer ;feature Features are obtained through 1D depthwise convolution. ;feature Features are obtained after a fully connected layer ; Step 424, Online Encoder The output features are mapped to embedding vectors through the projection head. ; Momentum encoder The output features are mapped to embedding vectors through the projection head. ; The projection head consists of a first MLP and a second MLP.

6. The Transformer-based unsupervised cross-spectral identification and pairing and full-spectral synthesis method according to claim 5, characterized in that: Step five is based on embedding vectors and embedding vector Calculate the InfoNCE loss to obtain the trained online encoder. Based on embedding vectors and embedding vector Calculate the InfoNCE loss to obtain the trained online encoder. ; The specific process is as follows: Step 51: Based on embedding vectors and embedding vector Calculate the InfoNCE loss to obtain the trained online encoder. The specific process is as follows: Step 5.11: Embed the vector The negative samples are sequentially stored in a FIFO queue Q of length K=1000 to form a negative sample set. ;in, Indicates the first One negative sample; Step 512: Based on embedding vectors and embedding vector Calculate the InfoNCE loss; expressed as: in, Indicates the first InfoNCE loss of the spectrum per pixel, Indicates the temperature coefficient. Indicates the first One negative sample; Optimization of online encoder parameters based on InfoNCE loss ; Momentum encoder parameters Instead of performing independent gradient updates, the momentum encoder parameters are updated solely through a weighted average of the parameters from the synchronous online encoder. ; indicates as: in, Indicates the momentum coefficient. ; Step 513: Repeat steps 511 and 512 until all training samples are available. corresponding embedding vector After all operations are completed, the trained online encoder is obtained. ; Step 52: Based on embedding vectors and embedding vector Calculate the InfoNCE loss to obtain the trained online encoder. The specific process is as follows: Step 521: Embed the vector The negative samples are sequentially stored in a FIFO queue Q of length K=1000 to form a negative sample set. ; in, Indicates the first One negative sample; Step 522: Based on embedding vectors and embedding vector Calculate the InfoNCE loss; expressed as: in, Indicates the first InfoNCE loss of the spectrum per pixel, Indicates the temperature coefficient. Indicates the first One negative sample; Optimization of online encoder parameters based on InfoNCE loss ; Momentum encoder parameters Instead of performing independent gradient updates, the momentum encoder parameters are updated solely through a weighted average of the parameters from the synchronous online encoder. ; indicates as: in, Indicates the momentum coefficient. ; Step 523: Repeat steps 521 and 522 until all training samples are available. corresponding embedding vector After all operations are completed, the trained online encoder is obtained. .

7. The Transformer-based unsupervised cross-spectral identification and pairing and full-spectral synthesis method according to claim 6, characterized in that: In step seven, the target category of the hyperspectral image 1 to be tested is determined. Prior spectrum and categories Prior spectrum Mapped to embedding vector and ; The target category of the hyperspectral image 2 to be tested Prior spectrum and categories Prior spectrum Mapped to embedding vector and ; The specific process is as follows: Step 71: Classify the target category of the hyperspectral image 1 to be tested. Prior spectrum Mapped to embedding vector ; The specific process is as follows: Construct the original spectral matrix based on the hyperspectral image to be tested; Each spectrum in the fused spectral matrix is ​​divided into overlapping spectral patches to obtain overlapping patches for each spectrum. Each spectrum after overlapping spectral partitioning is mapped to a feature vector through a linear embedding layer. The feature vectors corresponding to all spectra in the fused spectral matrix are combined into a sequence, and a learnable global embedding is concatenated at the beginning of the sequence to obtain the next sequence. The next sequence is then added element-wise to the learnable positional embedding to obtain the next sequence. ; will sequence Input the online encoder trained in step 51 Obtain the embedding vector ; Step 72: Classify the target category of the hyperspectral image 1 to be tested. Prior spectrum Mapped to embedding vector ; The specific process is as follows: Construct the original spectral matrix based on the hyperspectral image to be tested; Each spectrum in the fused spectral matrix is ​​divided into overlapping spectral patches to obtain overlapping patches for each spectrum. Each spectrum after overlapping spectral partitioning is mapped to a feature vector through a linear embedding layer. The feature vectors corresponding to all spectra in the fused spectral matrix are combined into a sequence, and a learnable global embedding is concatenated at the beginning of the sequence to obtain the next sequence. The next sequence is then added element-wise to the learnable positional embedding to obtain the next sequence. ; will sequence Input the online encoder trained in step 51 Obtain the embedding vector ; Step 73: Classify the target category of the hyperspectral image 2 to be tested. Prior spectrum Mapped to embedding vector ; The specific process is as follows: The original spectral matrix is ​​constructed based on the hyperspectral image to be tested. Each spectrum in the fused spectral matrix is ​​divided into overlapping spectral patches to obtain overlapping patches for each spectrum. Each spectrum after overlapping spectral partitioning is mapped to a feature vector through a linear embedding layer. The feature vectors corresponding to all spectra in the fused spectral matrix are combined into a sequence, and a learnable global embedding is concatenated at the beginning of the sequence to obtain the next sequence. The next sequence is then added element-wise to the learnable positional embedding to obtain the next sequence. ; Will Input the online encoder trained in step 52 Obtain the embedding vector ; Step 74: Classify the target category of the hyperspectral image 2 to be tested. Prior spectrum Mapped to embedding vector ; The specific process is as follows: The original spectral matrix is ​​constructed based on the hyperspectral image to be tested. Each spectrum in the fused spectral matrix is ​​divided into overlapping spectral patches to obtain overlapping patches for each spectrum. Each spectrum after overlapping spectral partitioning is mapped to a feature vector through a linear embedding layer. The feature vectors corresponding to all spectra in the fused spectral matrix are combined into a sequence, and a learnable global embedding is concatenated at the beginning of the sequence to obtain the next sequence. The next sequence is then added element-wise to the learnable positional embedding to obtain the next sequence. ; Will Input the online encoder trained in step 52 Obtain the embedding vector .

8. The Transformer-based unsupervised cross-spectral identification and pairing and full-spectral synthesis method according to claim 7, characterized in that: The calculation of features in step eight The spectrum of each pixel in and cosine similarity and A target response map is generated based on the cosine similarity of the spectra of all pixels; each pixel in the target response map is normalized to obtain the response value of each pixel. Set threshold ; If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. and ; If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. and ; detect the target The corresponding pixels are marked with rectangles; the detected target The corresponding pixels are marked with a rectangle; Calculation features The spectrum of each pixel in and cosine similarity and A target response map is generated based on the cosine similarity of the spectra of all pixels; each pixel in the target response map is normalized to obtain the response value of each pixel. Set threshold ; If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. and ; If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. and ; detect the target The corresponding pixels are marked with rectangles; the detected target The corresponding pixels are marked with a rectangle; The specific process is as follows: Step 81: Calculate the characteristics The spectrum of each pixel in and cosine similarity and A target response map is generated based on the cosine similarity of the spectra of all pixels; each pixel in the target response map is normalized to obtain the response value of each pixel. Set threshold ; If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. and ; If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. and ; detect the target The corresponding pixels are marked with rectangles; the detected target The corresponding pixels are marked with a rectangle; The specific process is as follows: Step 811: Calculate the characteristics Spectrum of each pixel and cosine similarity , is represented as: ， Features All pixel spectra and cosine similarity Rearrange in descending order, and calculate the cosine similarity of each row. satisfy Move to the next row and rearrange in descending order until all cosine similarities are reached. After sorting, a two-dimensional target response map is obtained; Normalize each pixel in the target response map to obtain the response value of each pixel; Set threshold ; If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. or ; If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. or ; Step 812: Calculate the characteristics Spectrum of each pixel and cosine similarity , is represented as: ， Features All pixel spectra and cosine similarity Rearrange in descending order, and calculate the cosine similarity of each row. satisfy Move to the next row and rearrange in descending order until all cosine similarities are reached. After sorting, a two-dimensional target response map is obtained; Normalize each pixel in the target response map to obtain the response value of each pixel; Set threshold ; If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. or ; If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. or ; Step 813: The targets detected in Steps 811 and 812... The corresponding pixels are marked with a rectangle; The targets detected in steps 811 and 812 The corresponding pixels are marked with a rectangle; Step 82: Calculate the characteristics The spectrum of each pixel in and cosine similarity and A target response map is generated based on the cosine similarity of the spectra of all pixels; each pixel in the target response map is normalized to obtain the response value of each pixel. Set threshold ; If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. and ; If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. and ; detect the target The corresponding pixels are marked with rectangles; the detected target The corresponding pixels are marked with a rectangle; The specific process is as follows: Step 821: Calculate the characteristics Spectrum of each pixel and cosine similarity , is represented as: ， Features All pixel spectra and cosine similarity Rearrange in descending order, and calculate the cosine similarity of each row. satisfy Move to the next row and rearrange in descending order until all cosine similarities are reached. After sorting, a two-dimensional target response map is obtained; Normalize each pixel in the target response map to obtain the response value of each pixel; Set threshold ; If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. or ; If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. or ; Step 822: Calculate the characteristics Spectrum of each pixel and cosine similarity , is represented as: ， Features All pixel spectra and cosine similarity Rearrange in descending order, and calculate the cosine similarity of each row. satisfy Move to the next row and rearrange in descending order until all cosine similarities are reached. After sorting, a two-dimensional target response map is obtained; Normalize each pixel in the target response map to obtain the response value of each pixel; Set threshold ; If features If the response value of a pixel is greater than or equal to a threshold, then the corresponding pixel is the target. or ; If features If the response value of a pixel is less than the threshold, then the corresponding pixel is not the target. or ; Step 823: The targets detected in Steps 821 and 822... The corresponding pixels are marked with a rectangle; The targets detected in steps 821 and 822 The corresponding pixels are marked with a rectangle.

9. The Transformer-based unsupervised cross-spectral identification and pairing and full-spectral synthesis method according to claim 8, characterized in that: In step nine, the target corresponding to the hyperspectral image 1 to be tested and The group characteristics are calculated from the pixel spectral matrix within the rectangular frame; Z-score normalization is applied to the group features to obtain the target corresponding to the normalized hyperspectral image 1 of the test. Population spectral eigenvectors and target The population spectral eigenvectors; Target corresponding to hyperspectral image 2 under test and The group characteristics are calculated from the pixel spectral matrix within the rectangular frame; Z-score normalization is applied to the group features to obtain the target corresponding to the normalized hyperspectral image 2 of the test. Population spectral eigenvectors and target The population spectral eigenvectors; Calculate the cosine distance between the standardized population spectral feature vectors corresponding to hyperspectral image 1 and hyperspectral image 2, and form the cost matrix based on the cosine distance. The Hungarian algorithm is used to analyze the cost matrix. The optimal match is obtained by solving the problem. The optimal match includes the type correspondence, spectral distance and matching confidence. The specific process is as follows: Step 91: Target corresponding to the hyperspectral image 1 to be tested and Pixel spectral matrix within the rectangular box and Calculate the population features separately; perform Z-score normalization on the population features to obtain the normalized population spectral feature vector corresponding to the measured hyperspectral image 1; the specific process is as follows: 1) Population characteristics include mean spectrum, standard deviation spectrum, coefficient of variation, and mean of first derivative; Mean spectrum : ; in, This represents a single pixel within the bounding boxes of all objects of the same type in an image. This represents the total number of pixels of the same type in an image; Standard deviation spectrum : ; coefficient of variation : ; First derivative mean : ; in, Represents the first of the same type of target in a graph. The average value of each band, Represents the first of the same type of target in a graph. The average value of each band; The mean spectrum, standard deviation spectrum, coefficient of variation, and mean of the first derivative are combined into a vector to form the population characteristics; 2) Perform Z-score normalization on the population features to obtain the normalized population spectral feature vector corresponding to the measured hyperspectral image 1; the specific process is as follows: Z-score normalization is performed on the population features to obtain the normalized population spectral feature vector corresponding to the hyperspectral image 1 under test. Each class of hyperspectral image 1 obtains a feature vector. Step 92: Target corresponding to the hyperspectral image 2 under test and The group characteristics are calculated from the pixel spectral matrix within the rectangular frame; Z-score normalization is performed on the population features to obtain the normalized population spectral feature vector corresponding to the hyperspectral image 2 under test. Each class of hyperspectral image 2 obtains a feature vector. Step 93: Calculate the cosine distance between the standardized population spectral feature vector corresponding to the hyperspectral image 1 and the population spectral feature vector corresponding to the hyperspectral image 2, and form the cost matrix based on the cosine distance. The Hungarian algorithm is used to analyze the cost matrix. The solution is performed to obtain the optimal match, which includes type correspondence, spectral distance, and matching confidence. The specific process is as follows: Step 931: Calculate the cosine distance between the standardized population spectral feature vector corresponding to the hyperspectral image 1 and the population spectral feature vector corresponding to the hyperspectral image 2, and form the cost matrix based on the cosine distance. The specific process is as follows: Cost Matrix It is The matrix, ; in, The cosine distance between the first group feature vector of the standardized hyperspectral image 1 and the first group feature vector of the standardized hyperspectral image 2 is represented. The cosine distance between the first group feature vector of the standardized hyperspectral image 1 and the second group feature vector of the standardized hyperspectral image 2 is represented. The cosine distance between the second group feature vector of the standardized hyperspectral image 1 and the first group feature vector of the standardized hyperspectral image 2 is represented. The cosine distance between the second group feature vector of the standardized hyperspectral image 1 and the second group feature vector of the standardized hyperspectral image 2 is represented. Step 932: Use the Hungarian algorithm to process the cost matrix. The solution is performed to obtain the optimal match, which includes type correspondence, spectral distance, and matching confidence. The specific process is as follows: 1) Cost matrix Each element in the matrix is ​​row-normalized to obtain the row-normalized cost matrix. ; 2) The cost matrix after row normalization Each element in the matrix is ​​column-normalized to obtain the column-normalized cost matrix. ; 3) The cost matrix after column normalization is normalized using the method of finding the minimum coverage line. The minimum number of coverage lines is obtained through processing. If the minimum number of coverage lines is equal to the column-normalized cost matrix If the order is 2, then the column-normalized cost matrix is... The position corresponding to the current zero element is the optimal match; Otherwise, repeat steps 1)-3) until the optimal match is obtained.

10. The Transformer-based unsupervised cross-spectral identification and pairing and full-spectral synthesis method according to claim 9, characterized in that: In step ten, a full-spectrum hyperspectral image is obtained based on the optimal matching obtained in step nine. The specific process is as follows: 1) Target in two hyperspectral images and The rectangular area is covered by a mask; Canny edge detection was performed on one of the two hyperspectral images to obtain a binarized edge image. ; Canny edge detection was performed on one of the two hyperspectral images to obtain a binarized edge image. ; Binarized edge image Shi-Tomasi corner detection is performed inside to obtain the feature point set P1; Binarized edge image Shi-Tomasi corner detection is performed inside to obtain the feature point set P2; The FLANN method is used to filter initial matching pairs from feature point sets P1 and P2, removing pairs with ambiguous matches to obtain the initial matching pair set. ; Use the RANSAC method to remove the initial set of matching pairs. The anomalies were obtained. and the corresponding affine matrix ; Using affine matrices Affine transformation is performed on one of the two hyperspectral images to obtain... This enables spatial alignment of the Region of Interest (ROI) between two hyperspectral images. 2) The 81-120 bands of the visible light image after ROI spatial alignment overlap with the 1-40 bands of the shortwave infrared image. A weighted average method is used to process the 81-120 bands of the visible light image and the 1-40 bands of the shortwave infrared image to obtain a full-spectrum hyperspectral image. The specific process is as follows: 21) Calculate the weighted value of a pixel. ; indicates as: in, It is the signal-to-noise ratio of the sensor for visible light images; It is the signal-to-noise ratio of the sensor for shortwave infrared images; It is a pixel in a visible light image; It is a pixel in a shortwave infrared image; It is the weighted value of a single pixel; 22) A five-point Gaussian filter is used to smooth and eliminate spectral abrupt changes at the seams between overlapping and non-overlapping regions, resulting in a full-spectrum hyperspectral image; the specific process is as follows: The kernel window size of the five-point Gaussian filter is 5, and the corresponding Gaussian kernel weights are... ; The visible light image (bands 81-120) after ROI spatial alignment overlaps with the shortwave infrared image (bands 1-40). The visible light image (band 80) and the shortwave infrared image (band 41) are two seams. Two bands before and after each seam are selected as filtering regions. Each pixel in the filtering region is weighted and averaged using a five-point Gaussian kernel to obtain the smoothed visible light image (band 80) and the shortwave infrared image (band 41) after seam processing. The weighted values ​​of all pixels in the 1-80 band of the visible light image after seam treatment are obtained. The shortwave infrared images in bands 41-120 after seam processing are stitched together to obtain a complete full-spectrum hyperspectral image in bands 1-200. ; 3) Extract the target corresponding to hyperspectral image 1 from the two hyperspectral images. The target corresponding to the average spectrum and hyperspectral image 1 The target corresponding to the average spectrum and hyperspectral image 2 The target corresponding to the average spectrum and hyperspectral image 2 The average spectrum; The target corresponding to hyperspectral image 1 The target corresponding to the average spectrum and hyperspectral image 2 The average spectrum is processed in step 2) to obtain a pixel with a band of 1-200. The target corresponding to hyperspectral image 1 The target corresponding to the average spectrum and hyperspectral image 2 The average spectrum is processed in step 2) to obtain a pixel with a band of 1-200. Insert the target of the corresponding category from the pixels in the two obtained 1-200 bands into The full-spectrum hyperspectral image is obtained by taking the corresponding mask positions in the image. .

Images