A small target tracking method for hyperspectral video

Abstract

The application belongs to the field of target tracking, and discloses a small target tracking method for hyperspectral video, which comprises the following steps: respectively calculating the contribution degree scores of each waveband of a hyperspectral frame and an initial target template with consistent waveband information; forming multiple false-color hyperspectral images and multiple false-color initial target images; performing distortion to obtain a distorted hyperspectral image and a distorted initial target image, and obtaining a region of interest; respectively extracting features of the distorted hyperspectral image and the distorted initial target image and fusing the features to obtain a feature map, and then using a prediction head to regress a rectangular frame in which a small target is located in the region of interest in the feature map, if the rectangular frame in which the small target is located is not obtained in the region of interest, then performing global search on the hyperspectral frame to obtain the rectangular frame in which the small target is located, which is the tracking result of the small target; and the application solves the problems of difficult feature extraction of small targets and serious influence of scenes such as occlusion through waveband reorganization and template updating.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of target tracking, and particularly relates to a small target tracking method for hyperspectral video. BACKGROUND

[0002] Target tracking is a basic task in computer vision, and small target tracking is a challenging and widely applied research direction in the field of target tracking, and is applied in fields such as remote sensing image processing and automatic driving. In many actual scenes, due to the small size of small targets or the interference of complex backgrounds and other factors, the performance of many excellent trackers on small target tracking is still very low.

[0003] With the rapid development of hyperspectral sensor technology, the hyperspectral video (HSV, Hyperspectral Video) generated by using hyperspectral imaging technology can highlight the material information of the target in a specific spectral channel. The spatial, spectral and temporal information recorded by HSV will be more helpful for tracking small targets. However, due to the limited resolution of small targets and the interference of complex backgrounds, it is necessary and urgent to research and develop hyperspectral small target trackers. Although there are a large number of hyperspectral object trackers (HOT, Hyperspectral Object Tracker), most of the mainstream HOTs focus on improving tracking accuracy or speed to meet the demand of quickly and accurately locating and tracking targets. However, these HOTs often focus on regular size targets in the design and optimization process, such as large objects or objects with obvious features. These regular targets usually occupy a larger area in the image or video frame, and their features are more prominent, so they are easier to be captured and recognized by tracking algorithms. Small targets in complex scenes may be affected by factors such as rapid movement and occlusion, resulting in their appearance changing and limited features, which brings greater challenges to the tracking task. In addition, due to the limited size information of small targets, existing HOTs often cannot obtain enough effective information when extracting features, and it is difficult to effectively deal with challenges such as small target occlusion, resulting in the inability to accurately capture the motion process of small targets. SUMMARY

[0004] The purpose of the present application is to provide a small target tracking method for hyperspectral video, to solve the problem that the prior art cannot accurately track small targets.

[0005] The application adopts the following technical scheme: a small target tracking method for hyperspectral video, comprising:

[0006] Step 1: respectively calculate the contribution score of each waveband of the hyperspectral frame and the initial target template with consistent waveband information; the initial target template is a pre-defined template containing typical features of the small target to be tracked in the hyperspectral frame;

[0007] Step 2: Sort according to the contribution score of each band of the hyperspectral frame, and reorganize the adjacent three bands in the hyperspectral frame to form multiple false color hyperspectral images; similarly, sort according to the contribution score of each band of the initial target template, and reorganize the adjacent three bands in the initial target template to form multiple false color initial target images;

[0008] Step 3: Distort the small target area of the false color hyperspectral image and the small target area of the false color initial target image to obtain a distorted hyperspectral image and a distorted initial target image; and mark the circumscribed rectangle of the highlight area in the saliency map of the distorted hyperspectral image as the region of interest;

[0009] Step 4: Extract the features of the distorted hyperspectral image and the distorted initial target image using the backbone network of the twin network tracker, fuse the features to obtain a feature map, and then regress the rectangular frame where the small target is located in the region of interest in the feature map using the prediction head, which is the tracking result of the small target.

[0010] The beneficial effects of the present application are:

[0011] When tracking the small target in the HSV, the present application can effectively improve the tracking accuracy of the small target by distorting the small target area. Since the small target has a small size, it only occupies a small number of pixels in the image, and its feature information is relatively limited and easy to be submerged in the background noise. Fully considering the spatial-spectral aliasing correlation, the resolution of the small target area is adaptively adjusted to enhance the perception ability of the small target, and the target template is updated in time to improve the tracking accuracy of the small target.

[0012] By band reorganization and template updating, the present application solves the problems of difficult feature extraction of small targets and serious influence of occlusion scenes. When tracking the small target, the band reorganization of the hyperspectral frame can filter out the bands with significant differences between the small target and the background, which helps the feature extraction network to extract sufficient and efficient discriminative feature information. The template updating judges whether the existing template is occluded to reduce the problems of tracking accuracy reduction and tracking trajectory drift caused by occlusion. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 The comparison results of the precision map and the success rate map of the tracking method of the present application and the existing hyperspectral target tracker on the MSSOT dataset;

[0014] Figure 2 The comparison results of the precision map and the success rate map of the tracking method of the present application and the visible light tracker on the MSSOT dataset;

[0015] Figure 3The trade-off between tracking accuracy and tracking speed of the tracking method of the present application and the existing hyperspectral target tracker on the MSSOT dataset;

[0016] Figure 3 (a) is the trade-off between tracking accuracy and tracking speed;

[0017] Figure 3 (b) is the trade-off between tracking success rate and tracking speed;

[0018] Figure 4 The visual tracking results of the tracking method of the present application and the existing hyperspectral target tracker on four video sequences of the MSSOT dataset;

[0019] Figure 4 (a) is the tracking result of the video sequence with the video name doublecar-9 in the MSSOT dataset;

[0020] Figure 4 (b) is the tracking result of the video sequence with the video name human-1 in the MSSOT dataset;

[0021] Figure 4 (c) is the tracking result of the video sequence with the video name motorcycle-6 in the MSSOT dataset;

[0022] Figure 4 (d) is the tracking result of the video sequence with the video name double-9 in the MSSOT dataset;

[0023] Figure 5 The visual tracking results of the tracking method of the present application and the visible light tracker on four video sequences;

[0024] Figure 5 (a) is the tracking result of the video sequence with the video name airplane-2 in the MSSOT dataset;

[0025] Figure 5 (b) is the tracking result of the video sequence with the video name basketball-0 in the MSSOT dataset;

[0026] Figure 5 (c) is the tracking result of the video sequence with the video name electriccar-6 in the MSSOT dataset;

[0027] Figure 5 (d) is the tracking result of the video sequence with the video name boat-13 in the MSSOT dataset. DETAILED DESCRIPTION

[0028] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0029] This invention discloses a small target tracking method for hyperspectral video, comprising four steps.

[0030] Step 1: Calculate the contribution scores of each band of the hyperspectral frame and the initial target template with consistent band information; the initial target template is a predefined template that contains typical features of the small targets tracked in the hyperspectral frame.

[0031] Among them, a small target refers to a target whose pixel area is much smaller than the size of a video frame (image). Based on the definition of absolute scale, it is generally considered that when the pixel area of ​​a target is less than 32×32 pixels, the target is a small target.

[0032] Assuming the hyperspectral frame has B bands, the hyperspectral frame input at time t can be: , The i-th band in is , , Each band consists of W row pixels and H column pixels, that is... .

[0033] Given the initial target template The band information is consistent with the hyperspectral frame. The i-th band in is , Each band consists of V rows of pixels and U columns of pixels, that is... 1 <V<W,1<U<H。

[0034] With hyperspectral frames For example, calculate respectively The contribution score of each band can be obtained by adding and averaging the importance normalized score of each band and the target-background difference normalized score of each band.

[0035] The method for obtaining the importance normalization score for each band is as follows: the initial spectral features of the hyperspectral frame and the initial target template are extracted sequentially through convolution, batch normalization and ReLU activation function. Then, the initial spectral features are used as input to extract the depth spectral features. Next, global average pooling, fully connected layer and softmax are used to calculate the attention weights of each band. Then, the depth spectral features are combined with the corresponding attention weights to obtain the importance score vector and the importance ranking matrix, and the importance normalization score is obtained.

[0036] The band attention mechanism is used to obtain depth spectral features. And by using the depth spectral characteristics of each band Multiply by the attention weight corresponding to that band The enhanced spectral features were obtained. This leads to the importance score vector of the band. This process generates an importance ranking matrix, which in turn yields the normalized importance score for each band.

[0037] First, the hyperspectral frames are processed using formulas (1) and (2) respectively. and initial target template Extracting initial spectral features, specifically:

[0038] (1)

[0039] (2)

[0040] in, For hyperspectral frames The initial spectral characteristics, For the initial target template The initial spectral characteristics, and Let be the initial spectral features of the i-th band in the hyperspectral frame. The initial spectral features of the i-th band in the initial target template; It is the ReLU activation function. For batch normalization operations, Let f(x) be a convolution operation, representing a convolution operation with kernel size a×a and stride b, where the kernel is... .

[0041] To extract depth spectral features, the initial spectral features obtained in the previous step are calculated using formulas (3) and (4) to complete the effective extraction of depth spectral features, specifically as follows:

[0042] (3)

[0043] (4)

[0044] in, For hyperspectral frames Extracted depth spectral features, For the initial target template Extracted depth spectral features.

[0045] The attention weights are calculated using the depth spectral characteristics of each band. Specifically:

[0046] (5)

[0047] (6)

[0048] wherein, is the attention weight of the i-th band in the hyperspectral frame ; and is the initial target template is the attention weight of the i-th band in the initial target template ; is calculated using the function, is a fully connected layer (FC), is a global average pooling operation (GAP), is the deep spectral feature of the i-th band in the hyperspectral frame is the deep spectral feature of the i-th band in the initial target template.

[0049] The importance score vector of the hyperspectral frame is calculated, and the deep spectral features of each band in the hyperspectral frame are multiplied by the corresponding attention weight according to formula (7) to obtain enhanced spectral features, denoted as ; the residual network and hierarchical feature extraction are used according to formula (8) to obtain the final feature map through the fully connected layers FC1 and FC2, and the global average pooling GAP operation is performed on the final feature map to obtain the global feature of each band in the hyperspectral frame , and finally the function is used to normalize the output to a probability distribution to obtain the importance score vector of each band, denoted as .

[0050] (7)

[0051] (8)

[0052] wherein, is the enhanced spectral feature of the i-th band in the hyperspectral frame , and is the importance score vector of the i-th band in the hyperspectral frame, and the importance score vectors of each band are arranged in order to form the importance score matrix of all bands .

[0053] The initial target template ​importance score vector, steps and methods and computing hyperspectral frames importance score vector.

[0054] (9)

[0055] (10)

[0056] wherein, is the initial target template is the enhanced spectral feature of the i-th band, is the importance score vector of the i-th band in the initial target template, and the importance score vectors of all bands are arranged in order to form an importance score matrix of all bands .

[0057] wherein, and are two fully connected layers, reduces the feature dimension from B to , wherein is a scale factor, maps the feature dimension back to B, is a residual connection network, is a matrix multiplication operation.

[0058] According to formula (11), the importance score matrix is multiplied by its transpose to obtain an importance ranking matrix , wherein the main diagonal values of the matrix are the importance normalized scores of the hyperspectral frame , and the specific formula is:

[0059] (11)

[0060] wherein, is the importance ranking matrix of all bands of the hyperspectral frame .

[0061] According to formula (12), the importance score matrix is multiplied by its transpose to obtain an importance ranking matrix , wherein the main diagonal values of the matrix are the importance normalized scores of the initial target template , and the specific formula is:

[0062] (12)

[0063] wherein, is the initial target template​​ Importance ranking matrix for all bands.

[0064] The method for obtaining the target-to-background difference normalized score is as follows:

[0065] First, windmill convolution is used to extract the initial spectral features of the hyperspectral frame and the initial target template, and batch normalization is performed. Second, complementary stitching of feature maps is achieved through multi-scale feature extraction branches. Then, the channel dimension is compressed and restored using a fully connected layer, and global average pooling is used to obtain the normalized score of the difference between the target and the background.

[0066] The target-background separation mechanism is mainly used to measure the difference between small targets and the background in each band of the hyperspectral frame, and to assign higher scores to bands that can highlight small target areas. Specifically, the target-background separation mechanism extracts hyperspectral frames using formulas (13) and (14). and initial target template Initial spectral characteristics:

[0067] (13)

[0068] (14)

[0069] in, For hyperspectral frames The initial spectral characteristics, For the initial target template The initial spectral characteristics, then The initial spectral features of the i-th band of the hyperspectral frame are given. The initial spectral characteristics of the i-th band of the initial target template. For windmill convolution.

[0070] The obtained initial spectral features are then combined using a multi-scale feature extraction branch, where the hyperspectral frames are concatenated complementaryly. The initial spectral characteristics are calculated using formula (15) for the initial target template. The initial spectral characteristics are calculated using formula (16):

[0071] (15)

[0072] (16)

[0073] in, For hyperspectral frames The stitched feature map For the initial target template The stitched feature map This is for splicing operations.

[0074] The target and background difference normalized score of the hyperspectral frame is calculated by formula (17) The target and background difference normalized score of the initial target template is calculated by formula (18) Firstly, the channel number is reduced to 1 / B of the original by the first full connection layer, denoted as , then after the and ReLU activation function, the channel number is restored to the original size by the second full connection layer , denoted as , and finally the target and background difference normalized score is obtained after the function:

[0075] (17)

[0076] (18)

[0077] wherein, is the target and background difference normalized score of the hyperspectral frame , and is the target and background difference normalized score of the initial target template .

[0078] Step 2: Sort according to the contribution score of each band of the hyperspectral frame, and recombine the adjacent three bands in the hyperspectral frame to form multiple false color hyperspectral images; similarly, sort according to the contribution score of each band of the initial target template, and recombine the adjacent three bands in the initial target template to form multiple false color initial target images.

[0079] The band contribution score is calculated, taking the hyperspectral frame for example, the importance normalized score of each band is added to the target and background difference normalized score , and then normalized by to obtain the final band contribution score , which is calculated according to formula (19), and for the initial target template , it is calculated according to formula (20):

[0080] (19)

[0081] (20)

[0082] wherein, is the hyperspectral frame ​​​the contribution score of the i-th band of the hyperspectral frame, the importance normalized score of the i-th band of the hyperspectral frame, the target-to-background difference normalized score of the i-th band of the hyperspectral frame, is an addition operation, using The function outputs a value normalized to the range [0, 1].

[0083] wherein, is the initial target template the contribution score of the i-th band of the hyperspectral frame, the importance normalized score of the i-th band of the initial target template, the target-to-background difference normalized score of the i-th band of the initial target template.

[0084] Take the hyperspectral frame for example, all bands in the frame are re-ranked according to the band contribution score from high to low, and the order is denoted as , and the new-ordered bands are used to synthesize false-color hyperspectral images according to every three adjacent bands, and finally N false-color hyperspectral images are obtained, then for the initial target template N false-color initial target images are obtained.

[0085] The bands with higher contribution scores are key bands, which are considered to have a beneficial effect on subsequent tracking, while the bands with lower contribution scores are considered to have an adverse effect on subsequent tracking, all bands in the frame are re-ranked according to the band contribution score from high to low, and a new band order is obtained, and N false-color hyperspectral images are obtained according to the new band order by reorganizing every three adjacent bands; the initial target template is reorganized in the same way as the hyperspectral frame .

[0086] Since the band information in the hyperspectral frame and the initial target template is consistent, the band contribution score ranking of the hyperspectral frame and the band contribution score ranking of the initial target template are consistent, i.e., the band contribution score rankings of the two are the same, therefore, in the N false-color hyperspectral images after band reorganization of the hyperspectral frame and the N false-color initial target images after reorganization of the initial target template, when the N false-color hyperspectral images and the N false-color initial target images are arranged in the order from 1 to N, the three band information contained in each false-color hyperspectral image and false-color initial target image is consistent.

[0087] Each three adjacent wave bands can form a false color hyperspectral image , Each three adjacent wave bands can form a false color initial target image , so that a total of N false color hyperspectral images and N false color initial target images are obtained, wherein . Wherein, when B is an integer multiple of 3, then N is the quotient of B divided by 3; otherwise, N is the result of rounding down the quotient of B divided by 3, that is .

[0088] Step 3: respectively warp the small target area of the false color hyperspectral image and the small target area in the false color initial target image to obtain a warped hyperspectral image and a warped initial target image; and the circumscribed rectangle of the highlight area in the saliency map of the warped hyperspectral image is recorded as the region of interest.

[0089] Specifically: warping is to elastically expand or shrink the small target area of the input false color hyperspectral image and false color initial target image while keeping the resolution unchanged. The initial target template is for the small target tracked in the hyperspectral frame. The small target area is obtained by calculating the area in the hyperspectral frame that is highly similar to the target in the initial target template. The small target area is an area that is highly likely to contain a small target.

[0090] The saliency map is used in combination with forward mapping to warp the input image, and the output is the warped image, specifically:

[0091] (21)

[0092] Wherein, is the mapping relationship of image warping, and are the input and output coordinates of reading the image.

[0093] The false color hyperspectral image and the false color initial target image are warped according to the transformation Resample the pixel values and output the warped hyperspectral image and the warped initial target image , specifically:

[0094] (22)

[0095] Image warping is realized by backward mapping. At each output pixel grid position, the inverse mapping is calculated to find its corresponding input pixel coordinates. The color depth of adjacent input pixel grid points is calculated using the bilinear interpolation method, and the calculated color depth is assigned to the output pixel, specifically:

[0096] (twenty three)

[0097] The complete process of zooming in resolution on a small target area It can be written as:

[0098] (twenty four)

[0099] in, It is a non-linear function that returns the coordinates of the predicted bounding box. The information will be used to regress to the target location in the original spatial label.

[0100] Step 4: The backbone network of the twin network tracker extracts features from the distorted hyperspectral image and the distorted initial target image respectively and fuses them to obtain a feature map. Then, the prediction head regresses within the region of interest in the feature map to obtain the bounding box where the small target is located. If the bounding box where the small target is located is not obtained within the region of interest, a global search is performed on the hyperspectral frame to obtain the bounding box where the small target is located, which is the tracking result of the small target.

[0101] This invention employs a multi-stream Siamese network structure, in which feature extraction, feature fusion, and target location regression prediction heads all originate from a Siamese network tracker pre-trained on a large-scale visible light dataset.

[0102] This invention uses multiple identical backbone networks to perform feature extraction and fusion operations on the output N distorted initial target images and N distorted hyperspectral images, respectively, to obtain N feature maps. Each set of feature maps is ,use The operation concatenates these feature maps and feeds them into the prediction head. The prediction head then reads the location information of the region of interest and performs foreground-background regression, centrality regression, and bounding box regression within the region of interest to obtain the input hyperspectral frame at time t. Tracking results This result will be used for subsequent template updates.

[0103] Template updating comprises an occlusion detection phase and a template updating phase. In the occlusion detection phase, target prior (π) and contextual information of the region of interest are used to generate the input hyperspectral frame at time t. Tracking results The saliency map is calculated using aggregated signatures to obtain the saliency map of the initial target template without occlusion, and an aggregated signature feature saliency map is generated. The saliency map is compared with the aggregate signature feature saliency map to generate a response map, and a threshold judgment method is used to detect whether occlusion has occurred.

[0104] In the template updating stage, if the occlusion is detected, the previous template is continued to be used; if no occlusion occurs, the new template is used The previous template is replaced by the new dynamic template.

[0105] The method for calculating the saliency map of is as follows:

[0106] (25)

[0107] wherein, , , is the integrated context information of three different channels of N twisted hyperspectral images, , , is a weight coefficient, and π is a two-dimensional prior related to the tracked target.

[0108] For the calculation of the aggregated signature saliency map, the saliency map of the target in the initial target template is mainly obtained when no occlusion occurs, and the target is further located, and the aggregated signature is gradually smoothed by a Gaussian kernel to obtain the aggregated signature saliency map.

[0109] In the occlusion judgment, the present application uses a threshold-based method to trigger template updating, as shown in formula (26):

[0110] (26)

[0111] wherein, is the response of the aggregated signature saliency map using the initial target template, is the response of the saliency map, is a threshold value, is the tracking result of the hyperspectral frame at time t.

[0112] The response map is generated by the response of and the response of the initial target template, and the response map shows the difference between the saliency maps. If the response of is far away from the response of the initial target template, it is considered that the target is occluded or not in the field of view, and when the response difference exceeds the threshold value, the template updating mechanism is triggered.

[0113] Embodiment: This embodiment uses the MSSOT dataset to illustrate the implementation process. The MSSOT dataset is constructed by a snapshot spectral imaging system with a 5x5 SFA mode, the sampling spectral band is 680-960 nm, the frame rate is 25 frames / s, the band number B = 25, the row pixel number W = 2045 and the column pixel number H = 1080 of the hyperspectral frame in the dataset, the adaptive scaling method is used to reduce the row pixels and column pixels of each band of the input hyperspectral frame to 320 pixels, that is, W = H = 320, and the row pixels and column pixels of each band of the input initial target template are reduced to 128 pixels, that is, V = U = 128.

[0114] The MSSOT dataset contains 185 training videos and 40 test videos, and contains nine challenge attributes, including small target challenge SO, illumination variation challenge IV, on-chip rotation challenge OR, shape variation challenge SV, occlusion challenge OCC, deformation challenge DEF, motion blur challenge MB, camera motion challenge CM, and background clutter challenge BC.

[0115] The MSSOT dataset is derived from L. Chen, Y. Zhao, and S. G. Kong. "SFA-guided mosaic transformer for tracking small objects in snapshot spectral imaging." ISPRS Journal of Photogrammetry and Remote Sensing, vol. 204, pp. 223-236, 2023.

[0116] Hyperspectral frame with the initial target template After input, the importance normalization score of each band and the target and background difference normalization score of each band are obtained through the band attention mechanism and the target and background separation mechanism respectively, the contribution degree score of each band is obtained by adding and averaging the importance normalization score of each band and the target and background difference normalization score of each band, and then the ranking of the contribution degree score is obtained, and the bands in the hyperspectral frame with the initial target template are reordered according to the ranking of the contribution degree score, the bands with higher ranking of the contribution degree score are sorted in front, otherwise, the bands are sorted in the back.

[0117] The convolution operation used in formulas (1), (2), (7) and (9) In formula (1), a and b are both set to 1, and the used is a convolution kernel with a size of 1x1. The convolution operation used in formulas (3), (4), (15) and (16) In the middle, a is set to 3, b is set to 1, and the used is a convolution kernel with a size of 3x3 .

[0118] The hyperspectral frame in the MSSOT dataset HSV is calculated to be divided into 8 false-color hyperspectral images, while the initial target template is divided into 8 false-color initial target images.

[0119] The small target regions in the 8 false-color hyperspectral images and the small target regions in the 8 false-color initial target images are warped respectively, and the warping operation of the small target region is performed according to formula (24), the prior knowledge of which comes from the small target position information in the initial target template, to obtain the warped image and regress to the target position in the original space label through the inverse operation of the warping operation , and the saliency map of the warped hyperspectral image after resolution magnification of the small target region is calculated by the Gaussian kernel smoothing filtering method, which contains the position information of the small target region with high confidence in the saliency map. According to the length and width of the saliency map, a rectangular region of interest is divided, and the region of interest information will be transmitted to the target position regression prediction head, and the position regression of the small target will focus on this area.

[0120] This embodiment uses a large-scale visible light dataset to train a twin feature extraction fusion backbone network and a target position regression prediction head to perform feature extraction and fusion and small target position information regression operation. The feature extraction and fusion use a ResNet-50 network, and the target position regression prediction head reads the region of interest position information and performs foreground-background regression, center degree regression, and target frame regression operation in the region of interest to obtain the tracking result of the small target.

[0121] According to formula (25), the saliency map of the tracking result of the input hyperspectral frame at time t is obtained , the weight coefficients , , are all set to 0.25, the saliency map of the tracking result is compared with the aggregated signature feature saliency map to generate a response map, and formula (26) is used to determine whether the target in is occluded, and the threshold is set to 1.6. When the response difference exceeds the threshold, it is considered that occlusion has occurred, triggering the template update mechanism, and the previous template is continued to be used, otherwise is used to replace the previous template.

[0122] Eleven existing HOTs and seven visible light trackers are selected to verify the effectiveness of the embodiment, wherein the eleven existing HOTs are MHT, MFI-HVT, DeepHKCF, BAE-Net, SiamBAG, SiamOHOT, Trans-DAT, MMF-Net, PHTrack, SSTtrack and HDSP, MHT uses hand-crafted features for tracking, while other trackers use deep features for tracking, the hand-crafted feature-based tracker is run on a machine equipped with an Intel Core i9-14900HX CPU@2.20 GHz and 32 GB RAM; while the deep feature-based tracker is run on another machine equipped with the same CPU but with an added NVIDIA RTX 4060 GPU.

[0123] The seven visible light trackers selected by the embodiment are SiamCAR, SiamBAN, Stark, TransT, OTrack, SeqTrack and AQATrack. Precision plots and success plots are selected to evaluate tracking accuracy, precision plots show the precision plots with center location error less than a specified 20 pixels, which are used to rank the trackers, success plots describe the number of successful frames, the overlap rate between the predicted bounding box and the ground truth value, the ground truth value exceeds a given threshold, ranging from 0 to 1, the embodiment also uses FPS and FLOPs to compare the real-time performance and computational complexity of the trackers, and the specific results are shown in Table 1 and Table 2.

[0124] Table 1 Comparison results of AUC of the embodiment and HOTs under different attributes of the MSSOT dataset

[0125]

[0126] Table 2 Comparison results of AUC of the embodiment and visible light trackers under different attributes of the MSSOT dataset

[0127]

[0128] SO in Table 1 and Table 2 is a small target challenge; IV is an illumination variation challenge; OR is an on-chip rotation challenge; SV is a shape variation challenge; OCC is an occlusion challenge; DEF is a deformation challenge; MB is a motion blur challenge; CM is a camera motion challenge; and BC is a background clutter challenge.

[0129] As shown in Figure 1 and Figure 2 Compared with the existing eleven HOTs and seven visible light trackers, the tracking accuracy of the embodiment for small targets is in the leading position, as shown in Figure 3As shown, the embodiment provides a good balance between tracking accuracy and tracking speed, and according to the results of Table 1 and Table 2, the embodiment performs best in dealing with challenging attributes such as occlusion, verifying the effectiveness of the tracking method of the embodiment.

[0130] As shown in Figure 4 and Figure 5 shown, the interference of occlusion and target deformation brings great challenges to the tracker, resulting in most trackers failing to successfully track the target, while the embodiment can effectively exclude the interference such as occlusion, and even if the target resolution is very low, it can still effectively regress the position of the small target. Overall, the method of the embodiment achieves good tracking effect on the MSSOT dataset, verifying the effectiveness of the present application.

[0131] Among them, the 11 kinds of HOT information are:

[0132] MMF-Net: Z. Li, F. Xiong, J. Zhou, J. Lu, Z. Zhao, and Y. Qian. "Material-guided multiview fusion network for hyperspectral object tracking". IEEE Transactions on Geoscience and Remote Sensing., vol. 62, pp. 1-15, 2024, Art no. 5509415.

[0133] SSTtrack: Y. Chen, Q. Yuan, Y. Tang, Y. Xiao, J. He, T. Han, Z. Liu, and L. Zhang. "SSTtrack: A unified hyperspectral video tracking framework via modeling spectral-spatial-temporal conditions". Information Fusion., vol. 114, p. 102658, 2025.

[0134] PHTrack: Y. Chen, Y. Tang, X. Su, J. Li, Y. Xiao, J. He, and Q. Yuan. "PHTrack: Prompting for hyperspectral video tracking". IEEE Transactions on Geoscience and Remote Sensing., vol. 62, pp. 1-18, 2024, Art no. 5533918.

[0135] Trans-DAT: Y. Wu, L. Jiao, X. Liu, F. Liu, S. Yang, and L. Li. "Domain adaptation-aware transformer for hyperspectral object tracking". IEEE Transactions on Circuits and Systems for Video Technology., vol. 34, pp.8041-8052, 2024.

[0136] SiamOHOT: C. Sun, X. Wang, Z. Liu, Y. Wan, L. Zhang, and Y. Zhong. "SiamOHOT: A lightweight dual Siamese network for onboard hyperspectral object tracking via joint spatial-spectral knowledge distillation". IEEE Transactions on Geoscience and Remote Sensing., vol. 61, pp. 1-12, 2023, Art no. 5521112.

[0137] HDSP: R. Yao, L. Zhang, Y. Zhou, H. Zhu, J. Zhao, and Z. Shao. "Hyperspectral object tracking with dual-stream prompt". IEEE Transactions on Geoscience and Remote Sensing., vol. 63, pp. 1-12, 2025, Art no. 5500612.

[0138] SiamBAG: W. Li, Z. F. Hou, J. Zhou, and R. Tao. "SiamBAG: Bandattention grouping-based Siamese object tracking network for hyperspectralvideos". IEEE Transactions on Geoscience and Remote Sensing., vol. 61, 2023,Art no. 5514712.

[0139] BAE-Net: Z. Li, F. Xiong, J. Zhou, J. Wang, J. Lu, and Y. Qian. "BAE-Net: A band attention aware ensemble network for hyperspectral objecttracking". in Proceedings of the IEEE International Conference on ImageProcessing. (ICIP), 2020, pp. 2106-2110.

[0140] MHT: F. Xiong, J. Zhou, and Y. Qian. "Material based object trackingin hyperspectral videos". IEEE Transactions on Image Processing., vol. 29,pp. 3719-3733, 2020.

[0141] MFI-HVT: Z. Zhang, K. Qian, J. Du, and H. Zhou. "Multi-features integration based hyperspectral videos tracker". in Proceedings of the 11th Workshop on Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing (WHISPERS), 2021, pp. 1-5.

[0142] DeepHKCF: B. Uzkent, A. Rangnekar, and M. J. Hoffman. "Tracking in aerial hyperspectral videos using deep kernelized correlation filters". IEEE Transactions on Geoscience and Remote Sensing., vol. 57, no. 1, pp. 449-461, 2019.

[0143] where the information of the 7 visible light trackers are respectively:

[0144] AQATrack: J. Xie, B. Zhong Z. Mo, S. Zhang, L. Shi, S. Song, and R. Ji. "Autoregressive queries for adaptive tracking with spatio-temporal transformers". in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (CVPR), 2024, pp. 14572-14581.

[0145] Stark: B. Yan, H. Peng, J. Fu, D. Wang, and H. Lu. "Learning spatio-temporal transformer for visual tracking". in Proceedings of the International Conference on Computer Vision. (ICCV), 2021, pp. 10428-10437.

[0146] OSTrack: B. Ye, H. Chang, B. Ma, S. Shan, and X. Chen. "Joint feature learning and relation modeling for tracking: A one-stream framework". in Proceedings of the European Conference on Computer Vision. (ECCV), 2022, pp.341-357.

[0147] SeqTrack: X. Chen, H. Peng, D. Wang, H. Lu, and H. Hu. "SeqTrack: Sequence to sequence learning for visual object tracking". in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (CVPR), 2023,pp. 14572-14581.

[0148] TransT: X. Chen, B. Yan, J. Zhu, D. Wang, X. Yang, and H. Lu. "Transformer tracking". in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (CVPR), 2021, pp. 8126-8135.

[0149] SiamCAR: D. Guo, J. Wang, Y. Cui, Z. Wang, and S. Chen. "SiamCAR: Siamese fully convolutional classification and regression for visual tracking". in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (CVPR), 2020, pp. 6268-6276.

[0150] SiamBAN: Z. Chen, B. Zhong, G. Li, S. Zhang, R. Ji, and Jeee. "Siamese box adaptive network for visual tracking". in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. (CVPR), 2020, pp.6667-6676.

Claims

1. A small target tracking method for hyperspectral video, characterized in that, The method comprises the following steps: Step 1: respectively calculate the contribution score of each band of the hyperspectral frame and the initial target template which have consistent band information; The initial target template is a pre-defined template containing typical characteristics of the small target tracked in the hyperspectral frame; Step 2: sort the contribution score of each band of the hyperspectral frame, and recombine the three adjacent bands in the hyperspectral frame to form a plurality of false-color hyperspectral images; similarly, sort the contribution score of each band of the initial target template, and recombine the three adjacent bands in the initial target template to form a plurality of false-color initial target images; Step 3: respectively warp the small target region of the false-color hyperspectral image and the small target region of the false-color initial target image to obtain a warped hyperspectral image and a warped initial target image; and record the circumscribed rectangle of the highlight region in the saliency map of the warped hyperspectral image as a region of interest; Step 4: extract the features of the warped hyperspectral image and the warped initial target image using the backbone network of the twin network tracker, fuse the features to obtain a feature map, and then regress the rectangular box where the small target is located in the region of interest in the feature map using the prediction head to obtain the tracking result of the small target; In step 3, the warping is to elastically expand or shrink the small target region of the input false-color hyperspectral image and false-color initial target image while keeping the resolution unchanged. The initial target template is for the small target tracked in the hyperspectral frame. The small target region is obtained by calculating the region with high similarity between the hyperspectral frame and the initial target template. The small target region is a region with high probability of containing a small target. 2.The small target tracking method for hyperspectral video of claim 1, wherein, In step 1, the contribution score is calculated by adding the importance normalized score and the target-background difference normalized score of each band, and then using softmax normalization to obtain the contribution score of each band.

3. The small target tracking method for hyperspectral video according to claim 2, wherein, The importance normalized score is obtained by: First, extract the initial spectral features of the hyperspectral frame and the initial target template through convolution, batch normalization and ReLU activation function in turn, then extract the deep spectral features by inputting the initial spectral features, and then calculate the attention weight of each band by using global average pooling, fully connected layer and softmax. Then, the deep spectral features and the corresponding attention weight are combined to obtain the importance score vector and the importance ranking matrix, and the importance normalized score is obtained.

4. The small target tracking method for hyperspectral video according to claim 2, characterized in that, The target-background difference normalized score is obtained by: First, extract the initial spectral features of the hyperspectral frame and the initial target template using windmill convolution and perform batch normalization processing; second, realize complementary splicing of feature maps through multi-scale feature extraction branch; then, compress and restore the channel dimension by using fully connected layer, and obtain the target-background difference normalized score by combining global average pooling.

5. The small target tracking method for hyperspectral video according to claim 1, wherein, In step 4, the twin network tracker uses a threshold method to trigger template update, and the threshold method is: wherein, is a saliency map response of the aggregated signature features using the initial target template, is a saliency map response of the aggregated signature features using the initial target template, is a threshold value, is a tracking result of the hyperspectral frame at time t.

Images