Hyperspectral target tracking method based on multi-stream vision-spectrum-space adapter

Abstract

The invention provides a hyperspectral target tracking method based on a multi-stream vision-spectrum-space adapter. The method mainly solves the problem of poor tracking performance caused by insufficient information interaction among different wavebands in the prior art. Comprising the following steps of: 1) grouping and preprocessing hyperspectral image data to obtain a hyperspectral image group and a pseudo-color image; 2) forming a single-stream vision block by a normalization layer, a multi-head attention layer and a full connection layer, and constructing a multi-stream hyperspectral block by N single-stream vision blocks in parallel; 3) respectively sending the hyperspectral picture group and the pseudo-color picture into two blocks, and extracting spectrum-space features and visual features; 4) constructing a bidirectional vision-spectrum-space adapter to process the features, and respectively feeding back the features to corresponding blocks; and 5) respectively carrying out feature fusion in the two blocks, fusing output features again, and sending the fused output features to a prediction head to obtain a tracking result. According to the invention, information from different wavebands can be effectively fused, the calculation cost is reduced, and the tracking performance is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology and further relates to target tracking technology, specifically a hyperspectral target tracking method based on a multi-stream vision-spectral-spatial adapter, which can be used for unmanned aerial vehicles (UAVs) to track targets in complex backgrounds. Background Art

[0002] Over the past few decades, object tracking has been a hot topic in academia. Researchers have devoted significant effort to RGB-based object tracking, proposing numerous advanced methods, ranging from those based on handcrafted features to those using deep learning. However, due to the imaging characteristics of RGB cameras, RGB images have limited ability to describe the inherent properties of objects, and object tracking in complex backgrounds remains challenging.

[0003] Unlike RGB cameras, hyperspectral cameras combine imaging and spectroscopy techniques, enabling simultaneous detection of both the two-dimensional geometric space and one-dimensional spectral information of a target. They capture continuous narrowband image data at high spectral resolution, preserving the imaging information of an object across different spectrums. Different materials exhibit distinct reflection, absorption, or emission characteristics at different wavelengths, making hyperspectral imagery more effective for material identification. This capability helps accurately distinguish between targets and background information in complex environments, providing enhanced robustness for target tracking.

[0004] In recent years, handcrafted feature-based hyperspectral object tracking methods, such as MHT, TASSCF, and TSCFW, have established target appearance features through material abundance scores, band selection, and tensor sparse correlation filters to separate targets from backgrounds. However, handcrafted feature-based trackers have limited processing capabilities and weak generalization capabilities when handling complex scenes with similar backgrounds and a large number of similar targets, making them ineffective in addressing real-world challenges. Deep learning-based hyperspectral object tracking methods, such as the spectral-spatial histogram of oriented gradients and the band autoencoder attention mechanism, track targets by extracting features. However, deep learning methods also face three major challenges. First, data is limited. Given the available data, training a deep neural network that can effectively extract target features and distinguish between targets and background is a challenging task. Second, image differences. Due to the different imaging mechanisms of hyperspectral cameras and RGB cameras, using pre-trained models requires addressing the differences in semantic information and the number of bands in the image, making current pre-trained models unsuitable for direct application to hyperspectral trackers. Third, the inherent low-rank, sparsity, spectral self-expression, and spectral correlation of hyperspectral images leads to the problem of band information interaction. How to effectively integrate and supplement visual-spectral-spatial information from different bands in highly sparse and strongly correlated data while bridging the spectral differences between bands has become a new challenge facing this field. Summary of the Invention

[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a hyperspectral target tracking method based on a multi-stream visual-spectral-spatial adapter. It is used to solve the problem of poor information interaction between different bands and unsatisfactory tracking effects caused by spectral differences. First, the spectral-spatial features and visual features in the hyperspectral image are extracted respectively through the stream spectral-spatial extraction module and the single-stream visual feature extraction module. Then, the spectral-spatial features and the visual features are fused layer by layer through the visual-spectral-spatial adapter to construct the intrinsic connection between the visual-spectral-spatial features, so that the model has stronger robustness. The present invention can effectively integrate and supplement the visual-spectral-spatial information of different bands in highly sparse and strongly correlated data, while bridging the spectral differences between the bands, thereby achieving effective tracking of targets in complex background environments.

[0006] To achieve the above objectives, the technical solutions of the present invention include the following:

[0007] (1) Decompose the input original hyperspectral image to obtain hyperspectral cube data, and then group and preprocess the data to obtain hyperspectral image groups and pseudo-color images;

[0008] (2) Construct two types of Transformer blocks, including a multi-stream hyperspectral block and a single-stream vision block. The single-stream vision block consists of a normalization layer, a multi-head attention layer, and a fully connected layer; the multi-stream hyperspectral block consists of N single-stream vision blocks in parallel. Let the number of hyperspectral data channels be C, then N = C / 3, and parameters are shared between multiple streams.

[0009] (3) The hyperspectral image group and the pseudo-color image are fed into the multi-stream hyperspectral and single-stream visual blocks loaded with pre-trained model parameters to extract spectral-spatial features and visual features;

[0010] (4) constructing a bidirectional visual-spectral-spatial adapter by using multiple sub-adapter modules with the same structure, so as to realize dynamic complementarity and interaction between visual, spectral, and spatial features of different bands; the sub-adapter module is composed of three linear layers, wherein the first linear layer is a downward projection layer, the second linear layer is a hidden layer, and the third linear layer is an upward projection layer;

[0011] (5) The two Transformer blocks respectively send their respective extractions to the bidirectional visual-spectral-spatial adapter constructed in step (4) for adapter;

[0012] (6) The bidirectional visual-spectral-spatial adapter adapts the received features and feeds the processed spectral-spatial features and visual features back to their corresponding Transformer blocks respectively;

[0013] (7) In the multi-stream hyperspectral block and the single-stream visual block, the features extracted by themselves are fused with the features fed back by the adapter and output;

[0014] (8) Fuse the features output by the two Transformer blocks and then feed the fused features into the prediction head;

[0015] (9) The predicted target coordinates are given by the prediction head, and the target tracking position is corrected in real time using the loss function to obtain the target tracking result.

[0016] Compared with the prior art, the present invention has the following advantages:

[0017] First, since the design of this invention introduces a multi-stream hyperspectral Transformer architecture, it can extract and fuse rich spectral and spatial information in different bands, thereby effectively improving the ability to distinguish between targets and backgrounds, especially for complex environments and weakly textured targets.

[0018] Second, in response to the semantic and distribution differences between different bands, the present invention proposes a bidirectional visual-spectral-spatial adapter to achieve dynamic complementarity and interaction between the visual, spectral, and spatial features of different bands, thereby enhancing the robustness of cross-modal information. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the overall implementation process of the method of the present invention;

[0020] Figure 2 Schematic diagram of the structure of the visual-spectral-spatial adapter constructed in the present invention;

[0021] Figure 3 This is a visual effect diagram of the model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The present invention will be further described below with reference to the accompanying drawings.

[0023] Example 1: Refer to the attached Figure 1 The present invention discloses a hyperspectral target tracking method based on a multi-stream vision-spectral-spatial adapter, comprising the following steps:

[0024] Step 1) Decomposing the input original hyperspectral image to obtain hyperspectral cube data, and then grouping and preprocessing the data to obtain a hyperspectral picture group and a pseudo-color picture; the grouping operation includes grouping the hyperspectral cube data into groups of three according to channel order to obtain a hyperspectral picture group; and finding the three channels closest to the RGB spectrum in the hyperspectral cube data to form a pseudo-color picture; the preprocessing operation includes randomly flipping, resizing, and normalizing the hyperspectral picture group and the pseudo-color picture.

[0025] In the grouping operation of this embodiment, the hyperspectral cube data is grouped into groups of three according to the channel order. A specific example is as follows: for hyperspectral cube data with C channels, the grouping method is: channels 1, 2, and 3 are grouped together, 4, 5, and 6 are grouped together, 7, 8, and 9 are grouped together, and so on. C-2, C-1, and C are grouped together. The above pictures are recorded as hyperspectral picture 1, hyperspectral picture 2, hyperspectral picture 3, and so on. Hyperspectral picture C / 3;

[0026] Step 2) Construct two Transformer blocks, including a multi-stream hyperspectral block and a single-stream vision block. The single-stream vision block consists of a normalization layer, a multi-head attention layer, and a fully connected layer. The multi-stream hyperspectral block is composed of N single-stream vision blocks in parallel. Let the number of hyperspectral data channels be C, then N = C / 3, and parameters are shared between the multiple streams. In this embodiment, the single-stream vision block constructed in this step and all single-stream vision blocks that make up the multi-stream hyperspectral block use the following hyperparameters: the number of attention heads is 8, and the input and output dimensions are both 768.

[0027] Step 3) The hyperspectral image set and the pseudo-color image are fed into the multi-stream hyperspectral and single-stream visual blocks, respectively, which have been loaded with pre-trained model parameters, to extract spectral-spatial and visual features. Specifically, in this embodiment, the images in the hyperspectral image set are sequentially fed into the various streams of the multi-stream hyperspectral block to obtain spectral-spatial features of different spectra. The pseudo-color image is fed into the single-stream visual block, where independent weight parameters are set to extract visual features only from the pseudo-color image. The pre-trained model parameters used in this embodiment are preferably derived from the OSTrack model.

[0028] Step 4) A bidirectional visual-spectral-spatial adapter is constructed using multiple structurally identical sub-adapter modules to achieve dynamic complementarity and interaction between visual, spectral, and spatial features in different bands. The sub-adapter module consists of three linear layers, where the first linear layer is a downward projection layer, the second linear layer is a hidden layer, and the third linear layer is an upward projection layer. In this embodiment, the input and output dimensions of the three linear layers are preferably 768, 8, 8, 8, 8, and 768, respectively.

[0029] Step 5) The two Transformer blocks respectively send their respective extractions to the bidirectional visual-spectral-spatial adapter constructed in step 4) for adaptation;

[0030] Step 6) The bidirectional visual-spectral-spatial adapter adapts the received features and feeds the processed spectral-spatial features and visual features back to their corresponding Transformer blocks respectively;

[0031] Step 7) In the multi-stream hyperspectral block and the single-stream visual block, the features extracted by themselves are fused with the features fed back by the adapter and output;

[0032] Step 8) Fuse the features output by the two Transformer blocks and then feed the fused features into the prediction head;

[0033] In this embodiment, the fusion involved in steps 7) and 8) is an addition operation of features.

[0034] Step 9) The prediction head gives the predicted target coordinates, and the target tracking position is corrected in real time using the loss function to obtain the target tracking result. In this embodiment, the model performance is optimized by using the loss function training based on the target coordinates given in the prediction head. When the model performance reaches the expected level, the training is stopped to maintain the model weight. The above loss function is calculated as follows:

[0035]

[0036] Among them L total represents the overall classification loss, L iou is the generalized model loss, L1 is the mean absolute error loss, and L cls is the weighted model loss, where They represent the generalized model loss weight parameter and the mean absolute error loss weight parameter respectively.

[0037] The effects of the present invention will be further described below in conjunction with simulation experiments.

[0038] 1. Simulation conditions:

[0039] The simulation experiment of the present invention is carried out in the hardware environment of CPU 12th Gen Intel(R)Core(TM)i9-12900K, memory 64GB, graphics card NVIDIA GeForce RTX 3090 and the software environment of Ubuntu22.04.4LTS and Python3.8.

[0040] 2. Simulation content:

[0041] The present invention is compared with various mainstream methods on the HOTC2020 dataset. As shown in Table 1, the tracking effect of the present invention is quantitatively compared with the five existing mainstream methods and shows the best tracking effect. The evaluation indicators used are precision Pre and success rate Suc. The higher the precision, the closer the center point of the predicted box is to the center point of the true box, that is, the better the performance of the predicted center point. The higher the success rate, the greater the intersection-over-union ratio between the predicted box and the true box, indicating that the predicted box is closer to the true box.

[0042] 3. Simulation results:

[0043] As shown in Table 1, the method of the present invention can achieve the best accuracy and success rate.

[0044] Table 1 Suc and Pre indicators of the present invention and the existing method

[0045] The present invention OSTrack SiamOHOT SiamBAG TSCFW BAE-Net Suc 0.677 0.645 0.634 0.641 0.604 0.606 Pre 0.914 0.890 0.883 0.904 0.887 0.879

[0046] like Figure 3 As shown, the blue box is the true box and the black box is the model prediction box. From this visual effect diagram, we can intuitively see that the black box can accurately frame the truck, and the difference with the true box of the model is small.

[0047] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0048] The above simulation analysis proves the correctness and effectiveness of the method proposed in the present invention.

[0049] Parts of the present invention that are not described in detail belong to common knowledge among those skilled in the art.

[0050] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Obviously, for professionals in this field, after understanding the content and principles of the present invention, they may make various modifications and changes in form and details without departing from the principles and structure of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

Claims

1. A multi-stream adaptive target tracking method based on hyperspectral video, characterized in that: The steps include: (1) Decompose the input original hyperspectral image to obtain hyperspectral cube data, and then group and preprocess the data to obtain hyperspectral image groups and pseudo-color images; (2) Construct two types of Transformer blocks, including a multi-stream hyperspectral block and a single-stream vision block. The single-stream vision block consists of a normalization layer, a multi-head attention layer, and a fully connected layer; the multi-stream hyperspectral block consists of N single-stream vision blocks in parallel. Let the number of hyperspectral data channels be C, then N = C / 3, and parameters are shared between multiple streams. (3) The hyperspectral image group and the pseudo-color image are fed into the multi-stream hyperspectral and single-stream visual blocks loaded with pre-trained model parameters to extract spectral-spatial features and visual features; (4) A bidirectional visual-spectral-spatial adapter is constructed by multiple sub-adapter modules with the same structure to achieve dynamic complementarity and interaction between visual, spectral, and spatial features of different bands; The sub-adapter module consists of three linear layers, wherein the first linear layer is a downward projection layer, the second linear layer is a hidden layer, and the third linear layer is an upward projection layer; (5) The two Transformer blocks respectively send their respective extractions to the bidirectional visual-spectral-spatial adapter constructed in step (4) for adapter; (6) The bidirectional visual-spectral-spatial adapter adapts the received features and feeds the processed spectral-spatial features and visual features back to their corresponding Transformer blocks respectively; (7) In the multi-stream hyperspectral block and the single-stream visual block, the features extracted by themselves are fused with the features fed back by the adapter and output; (8) Fuse the features output by the two Transformer blocks and then feed the fused features into the prediction head; (9) The predicted target coordinates are given by the prediction head, and the target tracking position is corrected in real time using the loss function to obtain the target tracking result.

2. The method according to claim 1, wherein: The grouping in step (1) includes grouping the hyperspectral cube data into groups of three according to the channel order to obtain a hyperspectral image group; And find the three channels closest to the RGB spectrum from the hyperspectral cube data to form a pseudo-color image.

3. The method according to claim 2, wherein: The preprocessing operation is to perform random flipping, size scaling and normalization operations on the hyperspectral image group and the pseudo-color image.

4. The method according to claim 1, wherein: The hyperparameters used for each single-stream vision block in step (2) are: the number of attention heads is 8, and the input and output dimensions are both 768.

5. The method according to claim 1, wherein: The pre-trained model parameters in step (3) are taken from the OSTrack model.

6. The method according to claim 1, wherein: The step (3) of extracting the spectral-spatial features and visual features is as follows: the pictures in the hyperspectral picture group are sequentially sent to the respective streams in the multi-stream hyperspectral block to obtain the spectral-spatial features of different spectra; The pseudo-color image is fed into the single-stream vision block, independent weight parameters are set, only pseudo-color images are accepted, and visual features are extracted.

7. The method according to claim 1, wherein: In step (4), the input and output dimensions of the first linear layer are 768 and 8 respectively; the input and output dimensions of the second linear layer are 8 and 8 respectively; the input and output dimensions of the third linear layer are 8 and 768 respectively.

8. The method according to claim 1, wherein: The fusion described in steps (7) and (8) is feature addition.

9. The method according to claim 1, wherein: Step (9) uses the loss function to correct the target tracking position in real time. Specifically, the loss function is used to train the model to optimize the performance based on the target coordinates given in the prediction head. When the model performance reaches the expected level, the training is stopped to maintain the model weight.

10. The method according to claim 9, characterized in that: The loss function is calculated as follows: Among them L total represents the overall classification loss, L iou is the generalized model loss, L1 is the mean absolute error loss, and L cls is the weighted model loss, where They represent the generalized model loss weight parameter and the mean absolute error loss weight parameter respectively.

Images