A target tracking method for hyperspectral video

Abstract

A target tracking method for a hyperspectral video, which initializes weight parameters in a hyperspectral target tracking algorithm; reads an initial frame of a tracking video as an initial template and an online update template image for target tracking, and simultaneously reads the position information of the target in the initial frame; reads the next frame of the tracking video and performs cropping processing to use it as a search image for target tracking; uses the initial template, the online update template image and the search image as inputs of a hyperspectral target tracking algorithm, reduces the dimensions to three-channel images through a dimensionality reduction module, and then inputs the image into a feature extraction network after low-rank adaptive learning, inputs the obtained feature information into a target position decoder, outputs the predicted position of the target in the current frame and its confidence, and determines whether to update the online update template image according to the confidence; repeats each frame of the tracking video and records the predicted position of the target in each frame of the video; the invention has the advantages of high target tracking accuracy, strong extracted feature representation, low training cost, and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking, and in particular to a target tracking method for hyperspectral video. Background Art

[0002] The target tracking task is a basic and important task in the field of computer vision, and is often used in fields such as traffic monitoring, self-driving cars, and video surveillance. Its main goal is to accurately predict the position of a given target object in a video sequence. Traditional target tracking algorithms are designed based on RGB videos, but this tracking algorithm will encounter difficulties when facing challenging scenes such as background interference and target deformation, resulting in a decrease in the tracking accuracy of the tracking algorithm. Unlike RGB tracking algorithms, hyperspectral tracking algorithms designed for hyperspectral videos still have good robustness in the face of these challenging scenes and can accurately track targets. This is because compared with RGB videos, hyperspectral videos contain a wider range of light wavelengths and have richer spectral information, which enables hyperspectral tracking algorithms to accurately analyze and identify different materials. This unique spectral feature enables the target tracking algorithm to better track the target and provide stronger resolution, thereby improving the accuracy of target tracking in challenging scenes.

[0003] At present, the hyperspectral tracking algorithms designed for hyperspectral videos have attracted more and more attention from researchers. Many hyperspectral target tracking algorithms have been proposed, but the existing hyperspectral tracking algorithms have problems such as low accuracy and slow speed. In addition, due to the high difficulty in acquiring hyperspectral videos, the amount of data in the current hyperspectral tracking dataset is small, which makes it easy to cause overfitting when training high-complexity tracking algorithms, thereby reducing the performance and generalization of the tracking algorithm. In order to solve these problems, the current mainstream methods of tracking algorithms are as follows:

[0004] 1) Use fine-tuning method: Use the RGB tracking algorithm as a pre-training algorithm, and input the used hyperspectral target tracking algorithm into the pre-training algorithm for fine-tuning training. Only a small learning rate and number of training times are needed to adapt the pre-training algorithm to the hyperspectral dataset, thereby obtaining a hyperspectral tracking algorithm. This method can not only inherit the resolution and generalization of the RGB tracking algorithm learned from a large amount of RGB training data, but also adapt it to the hyperspectral tracking task.

[0005] 2) Band reorganization method: Since the number of bands in hyperspectral video is much greater than that in RGB video, and in order to adapt hyperspectral video to RGB tracking algorithm, band reorganization method is proposed. Multi-band hyperspectral video is reorganized into multiple false color videos, and used as input to fine-tune the RGB tracking algorithm, so that the algorithm can fully learn the hyperspectral feature information in the hyperspectral video and improve the accuracy of the tracking algorithm.

[0006] 3) Band selection method: In order to solve the problem of real-time tracking being unable to be performed due to the reduced tracking speed caused by the band recombination method, a band selection method is proposed. The multi-band hyperspectral video is selected according to the three most important bands to generate a single false color video. The false color video is used as the input of the RGB tracking algorithm, so that the algorithm can learn the important feature information in the hyperspectral video. This allows the fine-tuned algorithm to accelerate the speed of target tracking while maintaining tracking accuracy.

[0007] However, existing technologies still cannot be adaptively applied to every frame of the video; the number of spectral channels in the hyperspectral data is not reduced, which leads to a decrease in the tracking speed of the entire network; the extracted features have weak representation and resolution capabilities, and are prone to overfitting, which leads to a decrease in the performance of the entire tracking method.

[0008] The above information disclosed in this Background section is only for enhancement of understanding of the background of the invention and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art.

[0009] The patent application document with the publication number CN113838088A discloses a method for tracking hyperspectral video targets based on deep tensors. The method includes randomly sampling a tensor sample set from the initial frame of the hyperspectral video to be tracked; performing a tensor principal component analysis based on tensor representation on the collected tensor sample set to obtain a projection matrix; using the projection matrix to map the tensor block X corresponding to the hyperspectral video frame to be tracked to the tensor subspace; inputting the candidate area of ​​the projected video frame into the VGG-M[2] network to extract spatial features, wherein the candidate area is obtained by proportionally enlarging the target area of ​​the previous frame; selecting the shallow and deep outputs of the network as feature maps, wherein the shallow layer is the first convolution layer and the deep layer is the fifth convolution layer; integrating the feature map into the target tracking model based on correlation filtering through kernel function mapping to achieve tracking. This method randomly samples the hyperspectral video and performs principal component analysis, which is a universal method for reducing information redundancy, but cannot be adaptively applied to every frame of the video.

[0010] The School of Information Science and Engineering of Hunan University of Technology and the Hunan Engineering Research Center for 3D Reconstruction and Intelligent Application Technology jointly published a paper titled "Hyperspectral Target Tracking Method Based on Channel Attention Mechanism" (Journal of Hunan University of Technology: Natural Science Edition, Vol. 36133-38) [3], which discloses a hyperspectral target tracking method. The implementation steps of the method include using a channel attention mechanism to selectively enhance useful and informative channels and suppress useless channel spectral information. Subsequently, the enhanced hyperspectral data is subjected to feature extraction through five convolutional units and further processed through two fully connected layers to obtain the position information of the tracked target. The accuracy and robustness of the target tracking algorithm are achieved in difficult scenarios such as illumination changes, target occlusion, target deformation, and motion blur. The method optimizes the hyperspectral data, but does not reduce the number of spectral channels in the hyperspectral data, which results in a decrease in the tracking speed of the entire network.

[0011] Moreover, the feature extraction methods in the above two prior arts both use convolutional layers for feature extraction, which will result in weak representation and resolution capabilities of the extracted features and prone to overfitting, thereby causing a degradation in the performance of the entire tracking method. Summary of the invention

[0012] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a target tracking method for hyperspectral videos, which can adaptively obtain the main information of the hyperspectral image and improve the representation and resolution capabilities of the extracted features by adopting a dimensionality reduction module based on a convolutional layer to reduce the number of spectral bands of the hyperspectral image and a Transformer network based on low-rank adaptive learning as a feature extraction method, and has the advantages of higher training efficiency, better tracking performance and faster tracking speed. The present invention solves the three problems of low tracking accuracy, high implementation complexity and difficulty in ensuring real-time performance of the current tracking method.

[0013] In order to achieve the above object, the present invention provides the following technical solutions:

[0014] A target tracking method for hyperspectral video includes the following steps:

[0015] Step 1, initializing the weight parameters in the hyperspectral target tracking algorithm, and loading the parameters fine-tuned on the hyperspectral target tracking data set into the hyperspectral target tracking algorithm; the hyperspectral target tracking algorithm is a hyperspectral target tracking algorithm based on low-rank adaptive learning;

[0016] Step 2: read the initial frame of the tracking video as the initial template for target tracking and the online updated template image, and read the location information of the target in the initial frame (G x ,G y ,G w,G h ), as the initial position of the target in tracking;

[0017] Step 3, read the next frame of the tracking video and crop it to serve as the search image for target tracking;

[0018] Step 4, the initial template of the target tracking in step 2 and the online updated template image and the search image in step 3 are used as the input of the hyperspectral target tracking algorithm with the fine-tuned parameters loaded in step 1, and the hyperspectral image is reduced to a three-channel image through a dimensionality reduction module;

[0019] Step 5, inputting the three-channel image of step 4 into the feature extraction network after low-rank adaptive learning to obtain feature information adapted to the hyperspectral image;

[0020] Step 6: Input the feature information of the adapted hyperspectral image obtained in step 5 into the target position decoder, output the predicted position of the target in the current frame and its confidence, and determine whether to update the online update template image according to the confidence;

[0021] Step 7: Repeat the process of predicting the target position from step 3 to step 6 for each frame of the tracking video, and record the predicted position of the target in each frame of the video to achieve tracking of the target in the hyperspectral video.

[0022] The method further includes:

[0023] Step 8: Input the predicted position in each frame of the video in step 7 and the actual position information of the target in the video into the evaluation function to calculate the accuracy of the tracking algorithm.

[0024] In the step 1, the parameters fine-tuned on the data set of hyperspectral target tracking are obtained by learning on the dimension reduction module and the low-rank adaptive learning module, wherein the dimension reduction module is composed of three convolutional layers, the convolution kernels used are all 1*1 in size, and the step size is set to 1; the low-rank adaptive learning module is composed of two linear layers, namely, a dimension reduction linear layer that reduces the high-rank information to low-rank information and a dimension increase linear layer that restores the low-rank information to high-rank information. The parameters used by the dimension reduction linear layer and the dimension increase linear layer are W up and W down .

[0025] In step 2, the initial frame of the tracking video and the position information of the search target in the initial frame are read (G x ,G y ,G w ,G h ), according to the bounding box width G of the target in the position information w , height G h and the template image cropping factor T f, calculate the side length C of the required cropping area t , and the center position information G of the target box in the position information x and G y As the center, the initial frame image is cropped with a side length of C t The calculation method is as follows:

[0026] Side Length

[0027] Among them, T f is a hyperparameter, and then the Resize(·) function is used to convert the cropped initial frame image into a size of 16×256×256, which is used as the initial template and online update template image for target tracking.

[0028] In step 3, the next frame of the tracking video is read as the search frame, and the bounding box width B of the target in the previous frame is calculated. w , height B h and search for image crop factor S f , calculate the side length C of the required cropping area s , search for image cropping factor S f is 4; according to the calculated side length C s The center position information of the bounding box of the target in the previous frame B x and B y , crop the search frame image and convert it to 16×256×256 size; use the processed search frame as the search image for target tracking.

[0029] In the step 4, the initial template and online updated template image of the target tracking in step 2 and the search image in step 3 are input into the dimension reduction module, and the hyperspectral image with a resolution of 16×256×256 is used as input to reduce the number of image channels through three convolution layers to obtain a three-channel image with a resolution of 3×256×256, so that the resolution size of the initial template, online updated template image and search image of the target tracking becomes 3×256×256.

[0030] In the step 5, the feature extraction network includes a ViT network, which is composed of a block coding module and a multi-layer Transformer layer. Before using the Transformer layer for feature extraction, the three-channel images of the input step 4 are respectively passed through a block coding module composed of convolution to convert the three-dimensional image information into two-dimensional token information. The size of the initial template, the online update template and the search token after encoding by the block coding module are all 768×256; the initial template token, the online update template token and the search token after the block encoding are connected in the spatial dimension to obtain a joint token as the input of the Transformer layer, and the size of the joint token is 768×768; each Transformer layer uses a different linear layer parameter W before calculating the attention. q , W k and W v , the input generates the corresponding query (Q), key (K) and value (V) tokens, so as to calculate the attention; a low-rank adaptive learning module is added next to the linear layer for generating query and key tokens; the low-rank adaptive learning module reduces the high-rank information, i.e. 768×768, to low-rank information, i.e. 8×768, through a dimensionality reduction linear layer, and then restores the low-rank information to high-rank information through a dimensionality increase linear layer; the joint token is passed through multiple layers of the Transformer layer after adding the low-rank adaptive learning module to obtain the feature information adapted to the hyperspectral image.

[0031] In step 6, the features output by the last layer of the feature extraction network in step 5 are used as visualization features and the sequence tokens self-learned by the model as feature information to input into the target position decoder, and the predicted position of the target in the current frame and its confidence are output. The target position decoder is composed of multiple attention modules and a pair of encoding and decoding dictionaries; wherein the attention module includes a masked self-attention layer for enhancing the query (Q) token and a mutual attention layer for performing attention calculation on the query (Q), key (K) token and value (V) token; the query (Q) token is generated by the sequence token through the encoding dictionary, and the output of the attention module is converted into a value through the decoding dictionary corresponding to the encoding dictionary, and the predicted position of the target in the current frame and its confidence are output; the parameters of the encoding dictionary and the decoding dictionary are obtained during the training process; the query input of the target position decoder is [start, x, y, w, h], and the output is [x, y, w, h, end], wherein start and end are start and end marks, x, y, w, h are normalized target prediction positions and sizes, respectively, and the confidence corresponding to x, y, w, h is arg x ,arg y ,arg w ,argh ; In the process of target tracking reasoning, the position and size information of the target are output one by one. In the process of obtaining the position information of the target, the confidence arg x ,arg y ,arg w ,arg h are saved, and these confidences are processed as the final judgment confidence confi, that is:

[0032] confi=(arg x +arg y +arg w +arg h )*10

[0033] Whether to update the image of the online update template is determined based on the confidence confi and the preset update threshold and template update cycle settings; when the set template update cycle is reached, if the confidence confi is greater than the preset update threshold, the template is updated. At this time, the current frame is cropped according to the currently predicted position information to obtain a new image as the updated template image; if the confidence confi is less than the preset update threshold, the original template image is maintained.

[0034] In step 8, the accuracy of the tracking algorithm is described using a precision map, DP@20P, a success map, and an area under the curve evaluation function, wherein the precision map records the fraction of frames whose predicted positions are within a given distance threshold, DP@20P is the average distance accuracy under a threshold of 20 pixels, the success map shows the percentage of successful frames whose overlap ratio between the predicted bounding box and the target true bounding box is greater than a certain threshold, ranging from 0 to 1, and AUC is the size of the area under the curve of the success map.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. The present invention adopts a dimensionality reduction module constructed by multi-layer convolution, which can adaptively reduce the channel dimension of hyperspectral video data. This method can perform personalized processing on each frame of hyperspectral video, effectively retain key spectral information, thereby improving the accuracy and efficiency of data processing.

[0037] 2. The present invention introduces a multi-layer Transformer layer as the core of feature extraction, which can perform in-depth feature extraction of input data on a global scale. This global feature extraction mechanism significantly enhances the representation and resolution capabilities of the model, making the extracted features richer and more accurate.

[0038] 3. The present invention integrates a low-rank adaptive learning module into the feature extraction network, and only trains the dimension reduction module and the low-rank adaptive learning module during the model training phase. This strategy significantly reduces the number of parameters required for model training, thereby reducing the demand for training data. More importantly, it significantly reduces the risk of overfitting during model training, thereby improving tracking performance.

[0039] In summary, the present invention processes hyperspectral data using multi-layer convolution technology through an adaptive dimensionality reduction module, and adopts a multi-layer Transformer layer combined with a low-rank adaptive learning module to achieve efficient extraction of global features of hyperspectral data. The present invention has the advantages of improving target tracking accuracy, enhancing the ability of extracted feature representation and resolution, and reducing training costs and overfitting risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 The present invention provides a flow chart of the method.

[0041] Figure 2 This is a schematic diagram of the dimensionality reduction module provided by the present invention.

[0042] Figure 3 Schematic diagram of the feature extraction network provided by the present invention.

[0043] Figure 4 Schematic diagram of the Transformer layer after low-rank adaptive learning provided by the present invention.

[0044] Figure 5 Schematic diagram of the target position decoder provided by the present invention.

[0045] Figure 6 It is the tracking effect diagram of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] like Figure 1-5 As shown, the present disclosure provides a target tracking method for a hyperspectral video, comprising the following steps:

[0049] Step 1, initializing the weight parameters in the hyperspectral target tracking algorithm, and loading the parameters fine-tuned on the hyperspectral target tracking data set into the hyperspectral target tracking algorithm; the hyperspectral target tracking algorithm is a hyperspectral target tracking algorithm based on low-rank adaptive learning;

[0050] Step 2: read the initial frame of the tracking video as the initial template for target tracking and the online updated template image, and read the location information of the target in the initial frame (G x ,G y ,G w ,G h ), as the initial position of the target in tracking;

[0051] Step 3, read the next frame of the tracking video and crop it to serve as the search image for target tracking;

[0052] Step 4, the initial template of the target tracking in step 2 and the online updated template image and the search image in step 3 are used as the input of the hyperspectral target tracking algorithm with the fine-tuned parameters loaded in step 1, and the hyperspectral image is reduced to a three-channel image through a dimensionality reduction module;

[0053] Step 5, inputting the three-channel image of step 4 into the feature extraction network after low-rank adaptive learning to obtain feature information adapted to the hyperspectral image;

[0054] Step 6: Input the feature information of the adapted hyperspectral image obtained in step 5 into the target position decoder, output the predicted position of the target in the current frame and its confidence, and determine whether to update the online update template image according to the confidence;

[0055] Step 7: Repeat the process of predicting the target position from step 3 to step 6 for each frame of the tracking video, and record the predicted position of the target in each frame of the video to achieve tracking of the target in the hyperspectral video.

[0056] In a preferred embodiment of the method, it also includes:

[0057] Step 8: Input the predicted position in each frame of the video in step 7 and the actual position information of the target in the video into the evaluation function to calculate the accuracy of the tracking algorithm.

[0058] In the step 1, the parameters fine-tuned on the data set of hyperspectral target tracking are obtained by learning on the dimension reduction module and the low-rank adaptive learning module, wherein the dimension reduction module is composed of three convolutional layers, the convolution kernels used are all 1*1 in size, and the step size is set to 1; the low-rank adaptive learning module is composed of two linear layers, namely, a dimension reduction linear layer that reduces the high-rank information to low-rank information and a dimension increase linear layer that restores the low-rank information to high-rank information. The parameters used by the dimension reduction linear layer and the dimension increase linear layer are W up and W down .

[0059] In a preferred embodiment of the method, in step 2, the initial frame of the tracking video and the position information of the search target in the initial frame are read (G x ,G y ,G w ,G h ), according to the bounding box width G of the target in the position information w , height G h and the template image cropping factor T f , calculate the side length C of the required cropping area t , and the center position information G of the target box in the position information x and G y As the center, the initial frame image is cropped with a side length of C t The calculation method is as follows:

[0060] Side Length

[0061] Among them, T f is a hyperparameter, and then the Resize(·) function is used to convert the cropped initial frame image into a size of 16×256×256, which is used as the initial template and online update template image for target tracking.

[0062] In a preferred embodiment of the method, the hyperparameter T f is 4.

[0063] In a preferred embodiment of the method, in step 3, the next frame of the tracking video is read as the search frame, and the bounding box width B of the target in the previous frame is w , height B h and search for image crop factor S f , calculate the side length C of the required cropping area s , search for image cropping factor S f is 4; according to the calculated side length C s The center position information of the bounding box of the target in the previous frame B x and B y, crop the search frame image and convert it to 16×256×256 size; use the processed search frame as the search image for target tracking.

[0064] In a preferred implementation of the method, in step 4, the initial template and online update template image of the target tracking in step 2 and the search image in step 3 are input into a dimension reduction module, and a hyperspectral image with a resolution of 16×256×256 is used as input and the number of image channels is reduced through three convolutional layers to obtain a three-channel image with a resolution of 3×256×256, so that the resolution size of the initial template, online update template image and search image of the target tracking becomes 3×256×256.

[0065] The dimension reduction module includes a three-layer convolutional network, the convolution kernels are all 1*1 in size, and the step size is also set to 1.

[0066] In a preferred embodiment of the method, in step 5, the feature extraction network includes a ViT network, and the ViT network is composed of a block coding module and a multi-layer Transformer layer. Before using the Transformer layer for feature extraction, the three-channel image of step 4 is respectively passed through a block coding module composed of convolution to convert the three-dimensional image information into two-dimensional token information for subsequent calculation of the attention map. The size of the initial template, the online update template and the search token after encoding by the block coding module are all 768×256; the initial template token, the online update template token and the search token after the block encoding are connected in the spatial dimension to obtain a joint token as the input of the Transformer layer, and the size of the joint token is 768×768; each Transformer layer uses a different linear layer parameter W before calculating the attention. q , W k and W v , the corresponding query (Q), key (K) and value (V) tokens are generated from the input to calculate the attention; a low-rank adaptive learning module is added next to the linear layer that generates the query and key tokens; the low-rank adaptive learning module consists of two linear layers, namely, a dimensionality reduction linear layer that reduces the high-rank information, i.e., 768×768, to low-rank information, i.e., 8×768, and a dimensionality increase linear layer that restores the low-rank information to high-rank information. The parameters used by the dimensionality reduction linear layer and the dimensionality increase linear layer are W up and W down The joint token is passed through multiple layers of the Transformer layer after adding the low-rank adaptive learning module to obtain the feature information adapted to the hyperspectral image.

[0067] In a preferred embodiment of the method, in step 6, the features output by the last layer of the feature extraction network in step 5 are used as visualization features and the sequence tokens self-learned by the model are used as feature information to input into the target position decoder, and the predicted position of the target in the current frame and its confidence are output, and the target position decoder is composed of multiple attention modules and a pair of encoding and decoding dictionaries; wherein the attention module includes a masked self-attention layer for enhancing the query (Q) token and a mutual attention layer for performing attention calculation on the query (Q), key (K) token and value (V) token; the query (Q )token is generated by the sequence token through the encoding dictionary. The output of the attention module is converted into a numerical value through the decoding dictionary corresponding to the encoding dictionary, and the predicted position of the target in the current frame and its confidence are output; the parameters of the encoding dictionary and the decoding dictionary are obtained during the training process; the query input of the target position decoder is [start, x, y, w, h], and the output is [x, y, w, h, end], where start and end are the start and end marks, x, y, w, h are the normalized target prediction position and size respectively, and the confidence corresponding to x, y, w, h is arg x ,arg y ,arg w ,arg h ; In the process of target tracking reasoning, the position and size information of the target are output one by one. In the process of obtaining the position information of the target, the confidence arg x ,arg y ,arg w ,arg h are saved, and these confidences are processed as the final judgment confidence confi, that is:

[0068] confi=(arg x +arg y +arg w +arg h )*10

[0069] Whether to update the image of the online update template is determined based on the judgment of the size of the confidence confi and the preset update threshold and the setting of the template update cycle; when the set template update cycle is reached, if the confidence confi is greater than the preset update threshold, the template is updated. At this time, the current frame is cropped according to the currently predicted position information to obtain a new image as the updated template image; if the confidence confi is less than the preset update threshold, the original template image is maintained.

[0070] In a preferred embodiment of the method, the three convolutional layers used for dimension reduction can be replaced by multiple linear layers. The feature extraction network used can replace ViT with other feature extraction networks, such as ResNet-50.

[0071] In a preferred embodiment of the method, in step 8, the accuracy of the tracking algorithm is described using a precision map, DP@20P, a success map, and an area under the curve evaluation function, wherein the precision map records the fraction of frames whose predicted positions are within a given distance threshold, DP@20P is the average distance accuracy at a threshold of 20 pixels, the success map displays the percentage of successful frames whose overlap ratio between the predicted bounding box and the target true bounding box is greater than a certain threshold, ranging from 0 to 1, and AUC is the size of the area under the curve of the success map.

[0072] like Figure 1 As shown, in one embodiment, a target tracking method for a hyperspectral video includes the following steps:

[0073] Step 1, initialize the weight parameters in the hyperspectral target tracking algorithm, and load the parameters fine-tuned on the hyperspectral target tracking dataset into the proposed hyperspectral target tracking algorithm; use the commonly used RGB tracking datasets COCO, LaSOT, GOT-10k and TrackingNet to pre-train the sequence generation target tracking network, learn the generalization and resolution of various tracking objects in a large amount of RGB data, add modules specially designed for hyperspectral tracking on the basis of the pre-trained sequence generation target tracking network, namely the dimensionality reduction module and the low-rank adaptive learning module, and fine-tune the tracking network with the new modules in the hyperspectral tracking dataset, and only train the added dimensionality reduction module and low-rank adaptive learning module during the fine-tuning period, load the parameters obtained after fine-tuning into the proposed hyperspectral target tracking algorithm, and complete the initialization of the algorithm.

[0074] Step 2: read the initial frame of the tracking video as the initial template for target tracking and the online updated template image, and read the location information of the target in the initial frame (G x ,G y ,G w ,G h ), as the initial position of the target being tracked; read the initial frame of the test hyperspectral video and the real position information of the search target, and calculate the bounding box width G of the target in the position information. w , height G h and the template image cropping factor T f , calculate the side length C of the required cropping area t , and the center position information G of the target box in the position information x and G yAs the center, the initial frame image is cropped with a side length of C t The calculation method is as follows:

[0075]

[0076] Among them, T f It is a hyperparameter, which is set to 4 here. Then, the cropped initial frame image is converted into a size of 16×256×256 using the Resize(·) function, and the image is used as the initial template image and online update template image for target tracking to complete the initialization of the template information.

[0077] Step 3: read the next frame of the tracking video and crop it to use as the search image for target tracking; read the next frame of the tracking video as the search frame, and calculate the search frame based on the bounding box width B of the target in the previous frame. w , height B h and search for image crop factor S f , calculate the side length C of the required cropping area s ; At this time, search for image cropping factor S f Set to 4; based on the calculated side length C s The center position information of the bounding box of the target in the previous frame B x and B y , the search frame image is cropped and converted to 16×256×256 size; the image is used as the search image for target tracking, so as to obtain all the inputs of the hyperspectral tracking network: two template images, namely the initial template and the online updated template image and a search image.

[0078] Step 4, the initial template image of the target tracking obtained in step 2 and the online updated template image and the search image of the target tracking obtained in step 3 are used as the input of the hyperspectral target tracking algorithm with the fine-tuned parameters loaded in step 1, and the input 16-channel hyperspectral image is reduced to a three-channel image through the proposed dimensionality reduction module to obtain a three-channel image;

[0079] The specific method of step 4 is:

[0080] The initial template image and online updated template image of target tracking obtained in step 2 and the search image of target tracking obtained in step 3 are respectively input into the dimension reduction module, and the number of channels of the 16-channel hyperspectral image is reduced from 16 to 3, so that the resolution size of the two template images, the initial template and online updated template image of target tracking and one search image becomes 3×256×256. The dimension reduction module used here consists of three convolution layers, the convolution kernel is 1*1 in size, and the step size is also set to 1; the specific implementation method of this module is as follows Figure 2This module takes a hyperspectral image with a resolution of 16×256×256 as input and reduces the number of image channels through three convolutional layers to obtain a three-channel image with a resolution of 3×256×256.

[0081] Step 5: Input the three-channel image after dimension reduction in step 4 into the feature extraction network after low-rank adaptive learning to obtain feature information adapted to the hyperspectral image; input the template image and the search image after dimension reduction into the feature extraction network after low-rank adaptive learning to obtain feature information adapted to the hyperspectral image. The basic structure of the feature extraction network is composed of a ViT network, which is composed of a block encoding module and a multi-layer Transformer layer. Before using the Transformer layer for feature extraction, the three-channel image of step 4 is respectively passed through a block encoding module composed of convolution to convert the three-dimensional image information into two-dimensional token information for subsequent calculation of the attention map. The size of the initial template, online updated template and search token after encoding by the block encoding module is 768×256; in order to enable the template and search information to interact when extracting the features of the template and the search image, the initial template token, online updated template token and search token after block encoding are connected in the spatial dimension to obtain the input connection token of the Transformer layer, which is 768×768 in size; in order to enable the feature extraction network to adapt to the extraction of hyperspectral image features, the Transformer layer is modified and a low-rank adaptive learning module is added; each Transformer layer uses a different linear layer parameter W before calculating the attention q , W k and W v , the input generates the corresponding query (Q), key (K) and value (V) tokens, thereby calculating the attention. The low-rank adaptive learning module is added next to the linear layer that generates the query and key tokens; the low-rank adaptive learning module consists of two linear layers, namely, a dimensionality reduction linear layer that reduces the high-rank information, i.e., 768×768, to low-rank information, i.e., 8×768, and a dimensionality increase linear layer that restores the low-rank information to high-rank information. The parameters used by the dimensionality reduction and dimensionality increase linear layers are W up and W down ; Connect the tokens through the Transformer layer after adding the low-rank adaptive learning module, and obtain the feature information adapted to the hyperspectral image; and use the features corresponding to the search image as the visualization features, with a size of 256×768. The specific structure of the feature extraction network and the low-rank adaptive learning module after adding the low-rank adaptive learning module is as follows Figure 3 and Figure 4 shown.

[0082] like Figure 5 As shown, in step 6, the feature information of the adapted hyperspectral image obtained in step 5 is input into the target position decoder, the predicted position of the target in the current frame and its confidence are output, and whether to update the online update template image is determined according to the confidence; the specific method of step 6 is: the visualization feature and the sequence token, the visualization feature is the feature output by the last layer of the feature extraction network, and the sequence token is the feature information of the model self-learning, are input into the target position decoder, and the predicted position of the target in the current frame and its confidence are output. The target position decoder consists of multiple attention modules and a pair of encoding and decoding dictionaries; the attention module includes a masked self-attention layer that enhances the query (Q) token and a mutual attention layer that performs attention calculations on the query (Q), key (K) token and value (V) token; the query (Q) token is generated by the sequence token through the encoding dictionary, and the output of the attention module is converted into a numerical value through the decoding dictionary corresponding to the encoding dictionary, and the predicted position of the target in the current frame and its confidence are output; the parameters of the encoding dictionary and the decoding dictionary are obtained during the training process; the query input of the target position decoder is [start, x, y, w, h], and the output is [x, y, w, h, end]. Start and end are the start and end marks, x, y, w, h are the normalized target prediction position and size, respectively, and the confidence corresponding to x, y, w, h is arg x ,arg y ,arg w ,arg h ; During the inference process of target tracking, the position and size information of the target are output one by one. The specific example is as follows: First, input start as the query feature, and the decoder outputs x and the corresponding arg x ; Then connect start and x to form the new input start,x, and get the output y and the corresponding arg y ; The required target position and size are obtained by looping in sequence, i.e., x, y, w, h. After post-processing and restoration, the normalized target position and size information is transformed into the target position and size information on the original image, i.e., B x ,B y ,B w ,B h .

[0083] In the process of obtaining the target's position information, the confidence arg x ,arg y ,arg w ,arg h are saved, and these confidences are processed as the final judgment confidence confi, that is:

[0084] confi=(arg x +arg y +arg w +arg h )*10

[0085] Whether to update the image of the online update template is determined based on the judgment of the size of the confidence confi and the preset update threshold and the setting of the template update cycle; when the set template update cycle is reached, if the confidence confi is greater than the preset update threshold, the template is updated, and the current frame is cropped according to the currently predicted position information to obtain a new image as the updated template image; if the confidence confi is less than the preset update threshold, the original template image is maintained. In the present invention, the update threshold is set to 0.6, and the template update cycle is set according to different situations. Long-term tracking is generally set to 10, and short-term tracking is generally set to 1.

[0086] Step 7: Repeat steps 3 to 6 for each frame of the tracking video to predict the target position and record the position information of the target in each frame of the video, thereby tracking the target in the hyperspectral video.

[0087] Step 8, input the predicted position of step 7 in each frame of the video and the real position information of the target in the video into the evaluation function, and calculate the accuracy of the tracking algorithm. The specific method of step 8 is: input the predicted position of step 7 in each frame of the video and the real position information of the target in the video into the evaluation function to obtain the performance of the tracking algorithm. Four evaluation schemes, namely, accuracy map, DP@20P, success map and area under curve (AUC), are used to describe the performance of the algorithm. The accuracy map records the fraction of frames whose estimated positions are within a given distance threshold. DP@20P is the average distance accuracy at a threshold of 20 pixels. The success map shows the percentage of successful frames whose overlap ratio between the predicted bounding box and the target's real bounding box is greater than a certain threshold, ranging from 0 to 1, and AUC is the size of the area under the curve of the success map.

[0088] The present invention provides a hyperspectral target tracking method, which successfully achieves efficient target tracking by following steps 1 to 8. Figure 6 As shown in Figure 4, the tracking effect of this method is demonstrated. Figure 6 The experimental video in this article comes from the "basketball" video provided by the "HyperspectralObject Tracking Challenge" competition. Figure 6The visualization results of tracking the hyperspectral data in the "basketball" video using the method of the present invention are shown, including screenshots of video frames (#2, #50, #106, #186). It can be seen from the result graph that the basketball in the video, as a small target, faces many challenges in the tracking process, such as fast movement, image blur, and background interference. Nevertheless, the hyperspectral target tracking algorithm based on low-rank adaptive learning proposed in the present invention shows excellent tracking performance, ensuring accurate tracking of the basketball throughout the video.

[0089] The present invention uses a dimensionality reduction module based on a convolutional neural network, which reduces the number of hyperspectral data channels while preserving the main spectral information in the hyperspectral data, reducing redundant information, not only speeding up the target tracking task, reducing tracking delay, but also improving tracking performance. A feature extraction network is obtained using low-rank adaptive learning. The method uses low-rank adaptive learning to fine-tune the existing feature extraction network for color tracking learning to adapt to hyperspectral tracking. The method only needs to train the parameters in the low-rank adaptive learning, while the parameters in the original color tracking network do not need to be trained and remain frozen during the training process. The amount of parameters that need to be trained during the training process is greatly reduced. The reduction in the amount of training parameters does not require a large amount of training data during training, which is a good solution to the problem of insufficient data volume in the current hyperspectral tracking data set. And a small amount of training parameters can reduce the number of training rounds and improve the efficiency of training. The present invention uses ViT as a feature extraction network for color tracking learning, which has greater advantages than the feature extraction network previously formed using a convolutional neural network. Compared with a convolutional network, ViT can learn the global features of the entire image and can better find targets in a global range. Moreover, ViT is a feature extraction network designed based on the Transformer mechanism. Its generalization and robustness are better than those of the feature extraction network composed of convolutional neural networks. It improves the robustness of hyperspectral target tracking in challenging scenarios such as background interference and object deformation. It improves the accuracy of hyperspectral target tracking. It increases the speed of target tracking and reduces the tracking delay. It uses the weights of the RGB tracking algorithm as pre-training parameters and reduces the number of parameters that need to be trained, thereby improving the training efficiency of the tracking algorithm.

[0090] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.

Claims

1. A target tracking method for hyperspectral video, characterized in that: The following steps are involved: Step 1, initializing the weight parameters in the hyperspectral target tracking algorithm, and loading the parameters fine-tuned on the hyperspectral target tracking data set into the hyperspectral target tracking algorithm; the hyperspectral target tracking algorithm is a hyperspectral target tracking algorithm based on low-rank adaptive learning; In the step 1, the parameters fine-tuned on the data set of hyperspectral target tracking are obtained by learning on the dimension reduction module and the low-rank adaptive learning module, wherein the dimension reduction module is composed of three convolutional layers, the convolution kernels used are all 1*1 in size, and the step size is set to 1; the low-rank adaptive learning module is composed of two linear layers, namely, a dimension reduction linear layer that reduces the high-rank information to low-rank information and a dimension increase linear layer that restores the low-rank information to high-rank information. The parameters used by the dimension reduction linear layer and the dimension increase linear layer are W up and W down ; Step 2: read the initial frame of the tracking video as the initial template for target tracking and the online updated template image, and read the location information of the target in the initial frame (G x ,G y ,G w ,G h ), where G x and G y is the center position information of the target box, G w and G h are the width and height of the bounding box, respectively, which serve as the initial position of the target in tracking; Step 3, read the next frame of the tracking video and crop it to serve as the search image for target tracking; Step 4, the initial template of the target tracking in step 2 and the online updated template image and the search image in step 3 are used as the input of the hyperspectral target tracking algorithm with the fine-tuned parameters loaded in step 1, and the hyperspectral image is reduced to a three-channel image through a dimensionality reduction module; Step 5, inputting the three-channel image of step 4 into the feature extraction network after low-rank adaptive learning to obtain feature information adapted to the hyperspectral image; Step 6: Input the feature information of the adapted hyperspectral image obtained in step 5 into the target position decoder, output the predicted position of the target in the current frame and its confidence, and determine whether to update the online update template image according to the confidence; Step 7: Repeat the process of predicting the target position from step 3 to step 6 for each frame of the tracking video, and record the predicted position of the target in each frame of the video to achieve tracking of the target in the hyperspectral video.

2. The target tracking method for hyperspectral video according to claim 1, characterized in that: The method further includes: Step 8: Input the predicted position in each frame of the video in step 7 and the actual position information of the target in the video into the evaluation function to calculate the accuracy of the tracking algorithm.

3. The target tracking method for hyperspectral video according to claim 1, characterized in that: In step 2, the initial frame of the tracking video and the position information of the search target in the initial frame are read (G x ,G y ,G w ,G h ), according to the bounding box width G of the target in the position information w , height G h and the template image cropping factor T f , calculate the side length C of the required cropping area t , and the center position information G of the target box in the position information x and G y As the center, the initial frame image is cropped with a side length of C t The calculation method is as follows: Side Length Among them, T f is a hyperparameter, and then the Resize(·) function is used to convert the cropped initial frame image into a size of 16×256×256, which is used as the initial template and online update template image for target tracking.

4. The target tracking method for hyperspectral video according to claim 1, characterized in that: In step 3, the next frame of the tracking video is read as the search frame, and the bounding box width B of the target in the previous frame is calculated. w , height B h and search for image crop factor S f , calculate the side length C of the required cropping area s , search for image cropping factor S f is 4; according to the calculated side length C s The center position information of the bounding box of the target in the previous frame B x and B y , crop the search frame image and convert it to 16×256×256 size; use the processed search frame as the search image for target tracking.

5. The target tracking method for hyperspectral video according to claim 1, characterized in that: In the step 4, the initial template and online updated template image of the target tracking in step 2 and the search image in step 3 are input into the dimension reduction module, and the hyperspectral image with a resolution of 16×256×256 is used as input to reduce the number of image channels through three convolution layers to obtain a three-channel image with a resolution of 3×256×256, so that the resolution size of the initial template, online updated template image and search image of the target tracking becomes 3×256×256.

6. The target tracking method for hyperspectral video according to claim 1, characterized in that: In the step 5, the feature extraction network includes a ViT network, which is composed of a block coding module and a multi-layer Transformer layer. Before using the Transformer layer for feature extraction, the three-channel images of the input step 4 are respectively passed through a block coding module composed of convolution to convert the three-dimensional image information into two-dimensional token information. The size of the initial template, the online update template and the search token after encoding by the block coding module are all 768×256; the initial template token, the online update template token and the search token after the block encoding are connected in the spatial dimension to obtain a joint token as the input of the Transformer layer, and the size of the joint token is 768×768; each Transformer layer uses a different linear layer parameter W before calculating the attention. q , W k and W v , the input generates the corresponding query Q, key K and value V token, so as to calculate the attention; a low-rank adaptive learning module is added next to the linear layer for generating the query and key token; the low-rank adaptive learning module reduces the high-rank information, i.e. 768×768, to low-rank information, i.e. 8×768, through a dimensionality reduction linear layer, and then restores the low-rank information to high-rank information through a dimensionality increase linear layer; the joint token is passed through multiple layers of the Transformer layer after adding the low-rank adaptive learning module to obtain the feature information adapted to the hyperspectral image.

7. The target tracking method for hyperspectral video according to claim 1, characterized in that: In step 6, the features output by the last layer of the feature extraction network in step 5 are used as visualization features and the sequence tokens self-learned by the model are used as feature information to input into the target position decoder, and the predicted position of the target in the current frame and its confidence are output. The target position decoder is composed of multiple attention modules and a pair of encoding and decoding dictionaries; The attention module includes a masked self-attention layer that enhances the query Q token and a mutual attention layer that performs attention calculation on the query Q, key K token and value V token; the query Q token is generated by the sequence token through the encoding dictionary, and the output of the attention module is converted into a numerical value through the decoding dictionary corresponding to the encoding dictionary, and the predicted position of the target in the current frame and its confidence are output; the parameters of the encoding dictionary and the decoding dictionary are obtained during the training process; the query input of the target position decoder is [start, x, y, w, h], and the output is [x, y, w, h, end], where start and end are the start and end marks, x, y, w, h are the normalized target prediction position and size respectively, and the confidence corresponding to x, y, w, h is arg x ,arg y ,arg w ,arg h ; In the process of target tracking reasoning, the position and size information of the target are output one by one. In the process of obtaining the position information of the target, the confidence arg x ,arg y ,arg w ,arg h are saved, and these confidences are processed as the final judgment confidence confi, that is: confi=(arg x +arg y +arg w +arg h )*10 Whether to update the image of the online update template is determined based on the confidence confi and the preset update threshold and template update cycle settings; when the set template update cycle is reached, if the confidence confi is greater than the preset update threshold, the template is updated. At this time, the current frame is cropped according to the currently predicted position information to obtain a new image as the updated template image; if the confidence confi is less than the preset update threshold, the original template image is maintained.

8. The target tracking method for hyperspectral video according to claim 2, characterized in that: In step 8, the accuracy of the tracking algorithm is described using a precision map, DP@20P, a success map, and an area under the curve evaluation function, wherein the precision map records the fraction of frames whose predicted positions are within a given distance threshold, DP@20P is the average distance accuracy under a threshold of 20 pixels, the success map shows the percentage of successful frames whose overlap ratio between the predicted bounding box and the target true bounding box is greater than a certain threshold, ranging from 0 to 1, and AUC is the size of the area under the curve of the success map.

9. The target tracking method for hyperspectral video according to claim 1, characterized in that: In step 1, the three convolutional layers used in the dimension reduction module are replaced by multiple linear layers.

Images