A hyperspectral image airplane target recognition method based on self-supervised learning

Abstract

The application provides a hyperspectral image airplane target recognition method based on self-supervised learning, and belongs to the field of hyperspectral image airplane target recognition. The method combines a semantic segmentation framework with a mask auto-encoding pre-training paradigm, excavates shared potential features in large-scale unlabeled hyperspectral images, utilizes multi-scale spatial information and multi-section spectral long-range dependence to perform deep modeling on airplane targets, and exhibits good robustness and generalization capability in cross-scene airplane target recognition applications, and has high practical value.

Description

Technical Field

[0001] This invention belongs to the field of hyperspectral image aircraft target recognition, specifically relating to a hyperspectral image aircraft target recognition method based on self-supervised learning. Background Technology

[0002] Hyperspectral remote sensing imaging can record the fine spectral features of aircraft targets and their backgrounds over a continuous narrow band while preserving rich spatial texture information, making it an important data source for aircraft target identification in complex scenes. Compared with traditional multispectral or RGB images, hyperspectral images have significant advantages in material differentiation, structural difference capture, and target saliency representation, thus holding significant application potential in tasks such as airport monitoring and airspace security.

[0003] However, hyperspectral images are characterized by high dimensionality, multiple bands, and large data volume, making manual annotation extremely costly and highly dependent on specialized knowledge. Furthermore, hyperspectral images struggle to simultaneously visualize spectral and spatial features, making it easy for annotators to be influenced by subjective factors when judging target boundaries, structural contours, or spectral differences, resulting in prevalent noisy labels. Once these noisy labels enter the model training process, they interfere with feature distribution learning, reduce model reliability, and hinder the construction and optimization of hyperspectral aircraft target recognition models. In addition, hyperspectral remote sensing applications operate in complex and diverse environments. Data collected across regions, times, and sensors often exhibit significant differences: varying surface backgrounds, changes in solar altitude angles, fluctuations in imaging atmospheric conditions, and inconsistencies in sensor spectral responses can all cause shifts in spectral distribution and spatial structure. The spectral curves of the same aircraft model may differ more under different acquisition conditions than they differ from other categories, making the model prone to overfitting to the training scene features and unable to adapt to new remote sensing areas or imaging conditions, thus leading to a significant decline in cross-scene recognition performance.

[0004] In recent years, self-supervised learning has gradually become an effective technical approach to address the challenges of label scarcity and cross-domain transfer difficulties due to its ability to learn general features from large-scale unlabeled hyperspectral images. By mining structural information from data during the pre-training stage, self-supervised models can obtain more robust spectral spatial representations, providing a more generalizable feature foundation for downstream aircraft target recognition. However, most existing hyperspectral self-supervised methods rely on single-scale structures such as Vision Transformers, which struggle to balance high-resolution spatial details and global structure in dense prediction tasks, resulting in insufficient representation capabilities for aircraft fuselage edges, small structural components, and multi-scale contours.

[0005] On the other hand, aircraft target identification is highly dependent on the sequential structure and long-range dependencies of the spectral dimension. Simple global spectral modeling is insufficient to capture the local differences in the spectral curves of different parts of the aircraft, while spatial modeling alone cannot utilize the multi-segment spectral information unique to hyperspectral imaging, thus failing to fully explore spectral spatial correlations. The lack of multi-scale spatial dimension and the lack of hierarchical representation in the spectral dimension have become structural limitations that hinder the improvement of the generalization performance of hyperspectral aircraft target recognition models.

[0006] Therefore, there is an urgent need for a model base that can be effectively pre-trained on a large number of unlabeled hyperspectral remote sensing images, has multi-scale feature modeling capabilities in space, long-range dependency learning capabilities in the spectrum, and can maintain robustness under cross-scene conditions, so as to improve the accuracy and generalization ability of hyperspectral aircraft target recognition. Summary of the Invention

[0007] To address the aforementioned issues, this invention proposes a hyperspectral image aircraft target recognition method based on self-supervised learning. This method comprises a dual-stream mask autoencoder base. In the spatial branch, a uniform mask is used to preserve the data structure, combined with a spatial shift window, to achieve Swin-based pre-training to mine multi-scale spatial features. The Swin base achieves layer-by-layer modeling of multi-scale features through a hierarchical structure and sliding window mechanism, maintaining high spatial resolution for lower-level features to capture local structural information, while higher-level features gradually aggregate global context information to form a more robust representation. In the spectral branch, multi-segmented spatial spectral tokens and high-mask-rate random masks are used to model long-range spectral dependencies within the Video-ViT pre-training framework, supplementing the Swin base's insufficient spectral modeling capabilities and demonstrating high practical value.

[0008] To achieve the above objectives, the present invention proposes the following technical solution:

[0009] A hyperspectral image aircraft target recognition method based on self-supervised learning, the method includes a dual-stream base pre-training stage and an aircraft target fine recognition task fine-tuning stage;

[0010] The dual-stream base pre-training stage includes: performing band dimensionality reduction on hyperspectral data to obtain a three-band image and a hyperspectral subset; the spatial Swing branch performs grid masking on the three-band image in units of blocks, extracts multi-scale spatial features through a multi-scale Swing encoder, reconstructs the three-channel image through a MAE decoder, and calculates the multi-scale spatial reconstruction loss between the masked region prediction result and the original ground truth; the spectral ViT branch combines adjacent bands of the hyperspectral subset into segments and divides them into tokens, performs random masking, extracts fixed-scale features through a ViT encoder, and the decoder inserts learnable mask tokens and visualization sequences to reconstruct the hyperspectral subset, and calculates the spectral token reconstruction loss and spectral angle consistency loss; the dual-stream pre-training is completed by combining the multi-scale spatial reconstruction loss, spectral token reconstruction loss, and spectral angle consistency loss.

[0011] The fine-tuning stage of the aircraft target recognition task includes: sampling and stitching the multi-scale spatial features and fixed-scale features, wherein the fixed-scale features are downsampled to the same spatial resolution as the multi-scale spatial features, and then stitched layer by layer to obtain multi-scale spatial spectral fusion features; the fusion features are input into the semantic segmentation head to output pixel-by-pixel category predictions, and the consistency between the predictions and the real labels is constrained by cross-entropy loss to achieve aircraft target recognition.

[0012] A hyperspectral image aircraft target recognition device based on self-supervised learning, the device comprising a dual-stream base pre-training module and an aircraft target fine recognition task fine-tuning module;

[0013] The dual-stream base pre-training module performs the following steps: Band dimensionality reduction of the hyperspectral data yields a three-band image and a hyperspectral subset; the spatial Swing branch performs grid masking on the three-band image in units of blocks, extracts multi-scale spatial features through a multi-scale Swing encoder, reconstructs the three-channel image via a MAE decoder, and calculates the multi-scale spatial reconstruction loss between the masked region prediction result and the original ground truth; the spectral ViT branch combines adjacent bands of the hyperspectral subset into segments and divides them into tokens, performs random masking, extracts fixed-scale features through a ViT encoder, and the decoder inserts learnable mask tokens along with the visualization sequence to reconstruct the hyperspectral subset, calculating the spectral token reconstruction loss and spectral angle consistency loss; the dual-stream pre-training is completed by combining the multi-scale spatial reconstruction loss, spectral token reconstruction loss, and spectral angle consistency loss.

[0014] The fine-tuning module for aircraft target identification performs the following steps: sampling and stitching the multi-scale spatial features and fixed-scale features, wherein the fixed-scale features are downsampled to the same spatial resolution as the multi-scale spatial features, and then stitched layer by layer to obtain multi-scale spatial spectral fusion features; the fusion features are input into the semantic segmentation head to output pixel-by-pixel category predictions, and the consistency between the predictions and the true labels is constrained by cross-entropy loss to achieve aircraft target identification.

[0015] An electronic device includes: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method.

[0016] A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to implement the method described thereon.

[0017] The beneficial effects of this invention are as follows:

[0018] Without requiring large-scale annotation training, this method relies on the semantic segmentation task framework and the mask autoencoder pre-training paradigm, enabling effective encoder training in unlabeled hyperspectral image scenarios. It has the ability to fuse multi-scale spatial and multi-segment spectral features. The spatial branch adopts the Swin-Transformer structure, and the spectral branch adopts the ViT structure. By fusing the features of both, it achieves in-depth mining of the spectral diversity of aircraft targets and has significant cross-scenario aircraft target classification capabilities. Attached Figure Description

[0019] Figure 1 This is a flowchart of a hyperspectral image aircraft target recognition method based on self-supervised learning according to the present invention. Detailed Implementation

[0020] The present invention will be further described in conjunction with the accompanying drawings and embodiments.

[0021] like Figure 1 As shown, this invention proposes a hyperspectral image aircraft target recognition method based on self-supervised learning, which consists of two steps: dual-stream base pre-training and fine-tuning for aircraft target recognition tasks. These two steps allow for training of the encoding model on unlabeled hyperspectral data and subsequent fine-tuning for downstream hyperspectral image aircraft target recognition tasks. The specific steps include:

[0022] 1) Dual-stream base pre-training phase:

[0023] Let the hyperspectral data be , For hyperspectral images, For the real number field, This represents the number of bands in the hyperspectral image. For the height of the hyperspectral image, To define the width of the hyperspectral image, extract three bands of the image and perform a grid masking. Let the three band images be denoted as... Assume the image is according to Divide the grid into blocks of different sizes, then there are a total of OK, Cut into chunks, This represents the total number of blocks in the image's height direction. This represents the total number of blocks in the image width direction. The length of the slice is the side length of the slice, and the slice is the smallest unit of the mask. All slices are then divided into slices. If we divide them into groups, then the group in row a and column b can be represented by the following formula:

[0024] ;

[0025] in, , For coordinates within the group, , for Grouped coordinates. The x-axis is the x-axis within the group. The ordinate within the group, For the a-th row and b-th column Grouping, where 'a' is the horizontal sequence number of the group and 'b' is the vertical sequence number of the group.

[0026] To ensure that the data remains rectangular after masking, only one slice is retained within each group, and the remaining three slices are used as a mask, i.e.:

[0027] ;

[0028] in, , for The mask state of the u-th row and v-th column of the group, where 1 indicates masking and 0 indicates visibility. Therefore, the final input sequence length is only a fraction of the original. Let the sequence be... It can be represented as:

[0029] ;

[0030] in, The input is the number of tokens (tokens, often used in this field to refer to slice features). For feature dimension, The third in the three-band image Line 1 The segmentation feature of the column. This is a subset of the visible sequence of the three-band image.

[0031] The features acquired by the multi-scale encoder Swin are denoted as follows: , This is the Swing feature for the first spatial scale; the other three parameters are similar. Given a multi-scale spatial feature set, the multi-scale encoder processing can be expressed as follows:

[0032] ;

[0033] in, This indicates a multi-scale encoder.

[0034] Subsequently, as Figure 1 As shown in the upper spatial branch of stage one, the MAE (Masked Autoencoder) decoder predicts the complete three-band image by performing deconvolution and layer-by-layer reconstruction operations on the encoded features. , The three-band image reconstructed by the decoder can be expressed as follows:

[0035] ;

[0036] in, This represents the MAE decoder. By minimizing the reconstruction error between the predicted result and the original input, the spatial branch is able to learn the multi-scale spatial structure features of the aircraft target under unsupervised conditions.

[0037] To ensure that the multi-scale encoder can learn the multi-scale spatial structure features of the aircraft target under unsupervised conditions, reconstruction loss is used as the optimization objective of the spatial branch. The model is trained by minimizing the difference between the predicted results of the masked region and the original ground truth. The expression is as follows:

[0038] ;

[0039] for Predicted value, the original ground truth value of the masked region The expression is as follows:

[0040] ;

[0041] Let the loss function be Multi-Scale Spatial Reconstruction (MSR), and the loss be... The calculation method is as follows:

[0042] ;

[0043] in, The mask rate is used to realize the pre-training process of the multi-scale encoder on hyperspectral images.

[0044] The native encoder uses band dimensionality reduction to obtain a hyperspectral subset. , will adjacent Each band is combined into a segment, and with Divide into chunk sizes, then the first... Duan Di Line 1 hyperspectral subset chunks of the column It can be expressed by the following formula:

[0045] ;

[0046] in, This indicates taking a subset of hyperspectral images. The part, Indicates taking a subset of hyperspectral data The part, Indicates taking a subset of hyperspectral data The part, For a hyperspectral subset containing K bands, for The number of bands in the hyperspectral subset For the first Duan Di Line 1 The hyperspectral subset of the column is cut into blocks.

[0047] Subsequently, a random mask is applied, and the visualized sequence is denoted as... The mask sequence is Then the two can be expressed by the following formula:

[0048] ;

[0049] in, 0 indicates no mask, and 1 indicates a mask. This is a subset of hyperspectral image visualizations, referred to as the visualization sequence; A subset of hyperspectral image masks, or simply a mask sequence; For the first Duan Di Line 1 Mask state of hyperspectral subset blocks.

[0050] The native encoder Vit will then visualize the sequence. Feature extraction is performed to obtain fixed-scale features. The process is shown in the following formula:

[0051] ;

[0052] ViTEnc stands for native encoder, where the decoder will insert learnable mask tokens. With visual sequences Reconstructing a subset of hyperspectral images together The process is shown in the following formula:

[0053] ;

[0054] in, The result is the prediction of the masked region. The prediction results for the hyperspectral subset are then used, followed by Spectral Token Reconstruction (STR) loss applied to the reconstructed results. Loss of Spectral Angle Consistency (SAC) The optimization and calculation formula is shown below:

[0055] ;

[0056] ;

[0057] in, For the mask rate, It is a function of spectral angle. For the hyperspectral subset Line 1 The spectral vector of the column, For the prediction result Line 1 The spectral vectors of the column. The final multi-segment self-supervised spectral loss is then expressed as follows:

[0058] ;

[0059] in, For hyperparameters. The spectral branch effectively improves the robustness and generalization of spectral representation in dense prediction tasks by modeling the local context of adjacent bands, capturing long-range dependencies across bands in the native encoder, and employing a highly random mask reconstruction strategy.

[0060] 2) Fine-tuning phase of aircraft target identification mission:

[0061] The multi-scale spatial features extracted by the multi-scale encoder are ,in This represents the extracted shallow, high-resolution basic features. This represents the extracted high-level, low-resolution semantic features; features from other layers follow the same pattern. The fixed-scale features obtained from the native encoder Vit are... The two features will be spliced ​​together by the downsampling, at which point the features extracted by the spectral branch will be... Belongs to high-resolution features, and If they have the same spatial resolution, align them first. Then Downsampling is performed as shown in the following formula:

[0062] ;

[0063] in, This indicates a 2x downsampling. Subsequently, the features from each level of the spatial and spectral branches are concatenated to obtain the final multi-scale spatial-spectral fusion (multi-scale encoder and native encoder fusion) features. The process is shown in the following formula:

[0064] ;

[0065] In the downstream hyperspectral aircraft target fine recognition task, the category prediction output by the semantic segmentation head is as follows: The true value is The loss function for this task is as follows:

[0066] ;

[0067] in, This is the set of all valid foreground pixels. For category prediction results; This is the true value for the category.

[0068] In summary, this invention employs supervised fine-tuning based on a dual-stream pre-trained model. The spatial branch utilizes SwinTransformer to extract multi-scale spatial features, while the spectral branch employs ViT to capture spectral context and long-range dependencies. Furthermore, multi-scale spatial-spectral fusion yields a more discriminative representation. Finally, pixel-by-pixel classification is achieved via an FPN decoder, and cross-entropy loss constrains the consistency between the predicted and true labels. After fine-tuning training with the dual-stream pre-trained model frozen, the model achieves high-precision target-level recognition in complex scenes and exhibits excellent cross-scene generalization capabilities.

[0069] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for aircraft target recognition based on self-supervised learning in hyperspectral images, characterized in that, The method includes a dual-stream base pre-training phase and an aircraft target fine-tuning phase for target identification. The dual-stream base pre-training stage includes: performing band dimensionality reduction on the hyperspectral data to obtain a three-band image and a subset of the hyperspectral image; the spatial Swing branch performs grid masking on the three-band image in units of blocks, extracts multi-scale spatial features through a multi-scale Swing encoder, reconstructs the three-channel image through a MAE decoder, and calculates the multi-scale spatial reconstruction loss between the predicted mask region and the original ground truth; the spectral ViT branch combines adjacent bands of the hyperspectral image subset into segments and divides them into tokens, performs random masking, extracts fixed-scale features through a ViT encoder, and the decoder inserts learnable mask tokens and visualization sequences to reconstruct the hyperspectral image subset, and calculates the spectral token reconstruction loss and spectral angle consistency loss; the dual-stream pre-training is completed by combining the multi-scale spatial reconstruction loss, spectral token reconstruction loss, and spectral angle consistency loss. The fine-tuning stage of the aircraft target recognition task includes: sampling and stitching the multi-scale spatial features and fixed-scale features, wherein the fixed-scale features are downsampled to the same spatial resolution as the multi-scale spatial features, and then stitched layer by layer to obtain multi-scale spatial spectral fusion features; the fusion features are input into the semantic segmentation head to output pixel-by-pixel category predictions, and the consistency between the predictions and the real labels is constrained by cross-entropy loss to achieve aircraft target recognition.

2. The method for aircraft target recognition based on self-supervised learning in hyperspectral images according to claim 1, characterized in that, The dual-stream base pre-training phase includes: Let the hyperspectral data be , For hyperspectral images, For the real number field, This represents the number of bands in the hyperspectral image. For the height of the hyperspectral image, To define the width of the hyperspectral image, extract three bands of the image and perform a grid masking. Let the three band images be denoted as... Assume the image is according to Divide the grid into blocks of different sizes, then there are a total of OK, Cut into chunks, This represents the total number of blocks in the image's height direction. This represents the total number of blocks in the image width direction. The length of the slice is the side length of the slice, and the slice is the smallest unit of the mask. All slices are then divided into slices. If we divide them into groups, then the group in row a and column b is represented by the following formula: ； in, , For coordinates within the group, , for Grouped coordinates; The x-axis is the x-axis within the group. The ordinate within the group, For the a-th row and b-th column Grouping, where 'a' is the horizontal number of the group and 'b' is the vertical number of the group; Only one slice is kept in each group, and the remaining three slices are set as masks, that is: ； in, , for Given the mask state of the u-th row and v-th column of the group, where 1 represents a mask and 0 represents visibility, the final input sequence is: Represented as: ； in, The input is the number of tokens, where each token represents a segmentation feature. For feature dimension, The third in the three-band image Line 1 The segmentation features of the column, This is a subset of the visible sequence of the three-band image.

3. The method for aircraft target recognition based on self-supervised learning in hyperspectral images according to claim 2, characterized in that, The features obtained by the spatial Swing branch are denoted as , This is the Swing feature for the first spatial scale; the other three parameters are similar. Given a multi-scale spatial feature set, the multi-scale encoder processing procedure is expressed as follows: ； in, This indicates a multi-scale encoder; The MAE decoder uses deconvolution and layer-by-layer reconstruction operations to predict the complete three-band image by encoding the features. , The three-band image reconstructed by the decoder is expressed as follows: ； in, This indicates the MAE decoder.

4. The method for aircraft target recognition based on self-supervised learning in hyperspectral images according to claim 3, characterized in that, The reconstruction loss, as the optimization objective of the spatial Swin branch, is used to train the model by minimizing the difference between the predicted results of the masked region and the original ground truth. The expression is as follows: ； in, for The predicted value, the original ground truth value of the masked region The expression is as follows: ； Let the loss function be and the multi-scale spatial reconstruction loss be . The calculation method is as follows: ； in, This is the mask rate.

5. The method for aircraft target recognition based on self-supervised learning in hyperspectral images according to claim 4, characterized in that, Spectral ViT branch utilizes hyperspectral subsets , will adjacent Each band is combined into a segment, and with Divide into chunk sizes, then the first... Duan Di Line 1 hyperspectral subset chunks of the column It can be expressed by the following formula, For a hyperspectral subset containing K bands, for The number of bands in the hyperspectral subset For the first Duan Di Line 1 Hyperspectral subset chunks of the column: ； in, This indicates taking a subset of hyperspectral images. The part; Subsequently, a random mask is applied, and the visualized sequence is denoted as... The mask sequence is Then the two can be expressed by the following formula: ； in, 0 indicates no mask, 1 indicates a mask; This is a subset of hyperspectral image visualizations, referred to as the visualization sequence; A subset of hyperspectral image masks, or simply a mask sequence; For the first Duan Di Line 1 The mask state of the hyperspectral subset chunks is then used; the spectral ViT branch will then visualize the sequence. Feature extraction is performed to obtain fixed-scale features. The process is shown in the following formula: ； ViTEnc stands for native encoder, where the decoder will insert a learnable mask token. With visual sequences The hyperspectral image subset is reconstructed together, as shown in the following equation: ； in, The result is the prediction of the masked region. This is the prediction result for the hyperspectral subset.

6. The method for aircraft target recognition based on self-supervised learning in hyperspectral images according to claim 5, characterized in that, Apply spectral token reconstruction loss to the reconstruction results. loss of spectral angular consistency The optimization and calculation formula is shown below: ； ； in, For the mask rate, It is a function of spectral angle. For the hyperspectral subset Line 1 The spectral vector of the column, For the prediction result Line 1 The spectral vector of the column; then the final spectral multi-segment self-supervised spectral loss is as follows: ； in, This is a hyperparameter.

7. The method for aircraft target recognition based on self-supervised learning in hyperspectral images according to claim 6, characterized in that, Step 2 includes: Multi-scale spatial features and fixed scale features Perform sampling and splicing, features and If they have the same spatial resolution, align them first. Then Downsampling is performed as shown in the following formula: ； in, This indicates a 2x downsampling; subsequently, the features from each level are stitched together to obtain the final multi-scale spatial spectral fusion feature. The process is shown in the following formula: ； In the downstream hyperspectral aircraft target fine recognition task, the category prediction output by the semantic segmentation head is as follows: The true value is The loss function for this task is as follows: ； in, For the set of all valid foreground pixels, For category prediction results; This is the true value for the category.

8. A hyperspectral image aircraft target recognition device based on self-supervised learning, characterized in that, The device includes a dual-stream base pre-training module and an aircraft target fine-tuning module for target identification. The dual-stream base pre-training module performs the following steps: Band dimensionality reduction of the hyperspectral data yields a three-band image and a hyperspectral subset; the spatial Swing branch performs grid masking on the three-band image in units of blocks, extracts multi-scale spatial features through a multi-scale Swing encoder, reconstructs the three-channel image via a MAE decoder, and calculates the multi-scale spatial reconstruction loss between the masked region prediction result and the original ground truth; the spectral ViT branch combines adjacent bands of the hyperspectral subset into segments and divides them into tokens, performs random masking, extracts fixed-scale features through a ViT encoder, and the decoder inserts learnable mask tokens along with the visualization sequence to reconstruct the hyperspectral subset, calculating the spectral token reconstruction loss and spectral angle consistency loss; the dual-stream pre-training is completed by combining the multi-scale spatial reconstruction loss, spectral token reconstruction loss, and spectral angle consistency loss. The fine-tuning module for aircraft target identification performs the following steps: sampling and stitching the multi-scale spatial features and fixed-scale features, wherein the fixed-scale features are downsampled to the same spatial resolution as the multi-scale spatial features, and then stitched layer by layer to obtain multi-scale spatial spectral fusion features; the fusion features are input into the semantic segmentation head to output pixel-by-pixel category predictions, and the consistency between the predictions and the true labels is constrained by cross-entropy loss to achieve aircraft target identification.

9. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the hyperspectral image aircraft target recognition method based on self-supervised learning as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, cause the processor to implement the hyperspectral image aircraft target recognition method based on self-supervised learning as described in any one of claims 1 to 7.

Images