Hyperspectral target tracking method based on cross-modal fusion and memory prompt

Abstract

The invention discloses a hyperspectral target tracking method based on cross-modal fusion and memory prompt, and the method comprises the steps: determining template regions Zrgb and Zhs of a first frame of RGB modal image and a (t-1) th frame of RGB modal image and a hyperspectral modal image, searching the regions, carrying out Patchedding operation on the regions, carrying out the flattening and splicing, sending an obtained RGB token and a pseudo-color token group into a spectrum complementary generator, and carrying out the detection of the hyperspectral modal image. The method comprises the following steps: determining a complementary fusion token, carrying out feature extraction on the fusion through a Transform encoder, sending an extracted spatial-spectral complementary feature Fcom into a memory prompt network, obtaining a memory key value pair KM and VM of a current t-th frame through the memory prompt encoder, sending the memory key value pair KM and VM into a memory pool, determining a query key value pair KQ and VQ of the current t-th frame Fcom through convolution operation, and in a memory prompt decoder, carrying out memory prompt on the query key value pair KQ and VQ; and querying and retrieving the KQ and the VQ with the KM and the VM containing the first 50 frames in the memory pool, determining a memory feature Fmem with historical frame information of the t-th frame, determining the position of a prediction frame according to the Fmem, and obtaining a tracked target.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a hyperspectral target tracking method based on cross-modal fusion and memory prompting. BACKGROUND

[0002] Hyperspectral video target tracking is a technical means focusing on real-time identification and continuous tracking of targets in hyperspectral video sequences. Compared with traditional videos, hyperspectral videos combine two-dimensional spatial dimensions and one-dimensional spectral dimensions, and can obtain more comprehensive scene information, thereby improving target discrimination and recognition accuracy. Such data is generally collected by hyperspectral imaging devices, has the characteristics of multiple wavebands and large information quantity, and is widely used in many fields such as national defense monitoring, ecological environment assessment, fine agricultural management and the like.

[0003] At present, most target tracking methods are still based on visible light videos, but in complex environments such as close target and background colors or severe light changes, it is often difficult to maintain stable tracking effect, and positioning deviation or target loss is prone to occur. In contrast, hyperspectral video target tracking uses the multi-waveband reflectivity data obtained by hyperspectral imaging devices, and can describe target characteristics from a more detailed spectral dimension, thereby realizing more accurate identification and positioning. Even if the target is difficult to distinguish in color, hyperspectral technology can complete effective distinction with its unique spectral prompt information. This advantage makes it show strong robustness in complex application environments, and gradually attracts widespread attention from the academic and industrial circles. Target tracking mainly relies on two types of mainstream method systems: one is a traditional tracking model constructed on the basis of correlation filtering strategy, and the other is a learning type tracking framework relying on deep neural network for feature extraction and decision making.

[0004] The correlation filtering type tracking method belongs to the classical technology developed earlier, and its basic principle is to construct a matching filter in an image, to find the position with the highest response value by correlation calculation with a candidate region to determine the current position of the target. This method is well-known for high calculation efficiency and fast response speed, and is widely used in scenes with high real-time requirement. However, at the same time, it is sensitive to changes in target shape, partial occlusion and fluctuations in environmental light, and is prone to accuracy decline in complex conditions.

[0005] In recent years, thanks to the rapid development of deep learning, target tracking methods based on deep models have become the dominant direction in this field. Among them, tracking algorithms using the Transformer structure have attracted widespread attention because the self-attention mechanism can effectively model the dependence of the target in the time and spatial dimensions, thereby improving the tracking performance in complex scenarios such as occlusion, violent motion, and background interference. However, most of these methods rely only on spatial and texture information in visible light images and fail to fully utilize the spectral dimension features in the hyperspectral modality, resulting in limited accuracy in identifying and tracking when the target and background have similar appearances. In addition, tracking methods under the Transformer framework have difficulty in fully modeling the temporal dependence of the target between different frames without explicitly introducing temporal information. When the target is temporarily occluded or temporarily lost during long-term tracking, the model often struggles to accurately recover the target position due to the lack of continuous temporal context support, leading to tracking failure or exacerbation of the drift problem. SUMMARY

[0006] Therefore, the main purpose of the present application is to provide a hyperspectral target tracking method based on cross-modal fusion and memory prompting.

[0007] To achieve the above purpose, the technical scheme of the present application is as follows:

[0008] The embodiment of the present application provides a hyperspectral target tracking method based on cross-modal fusion and memory prompting, which is as follows:

[0009] Step one, load the RGB modality image and the hyperspectral modality image, divide the hyperspectral modality image into multiple groups of pseudo-color images according to the channel order, and the number of channels of each group of pseudo-color images is consistent with the RGB modality image; determine the template region Z rgb and the template region Z hs of the first frame of the hyperspectral modality image through the ground truth value labeled in advance, determine the search region and the search region of the t-th frame of the hyperspectral modality image through the size of the target obtained after completing target tracking of the t-1-th frame, wherein t represents the frame number of the RGB modality and the hyperspectral modality image, and t is an integer greater than or equal to 1;

[0010] Step two, perform Patchembedding operation on Z rgb , Z hs , and respectively to determine the flattened vectors and Then, they are spliced according to the channel to determine the RGB token And pseudo-color token group Wherein, n represents the number of pseudo-color image groups;

[0011] Step three, by T rgb And Determine the complementary fusion token f fusion ;

[0012] Step four, f fusion Into the Transformer for feature extraction, obtain the empty spectrum complementary feature F com ;

[0013] Step five, according to F com Determine the memory prompt of the current frame, and combine the history memory pool module in the memory prompt decoder and the subsequent decoding operation to determine the memory feature F mem ;

[0014] Step six, F mem Into the prediction head network, obtain the prediction box position, complete the target tracking of the t-th frame of hyperspectral image, load each frame of hyperspectral image in the hyperspectral image sequence in turn, and complete the target tracking of the entire hyperspectral image sequence.

[0015] Preferably, the step two, specifically comprising: Patchembedding operation divides Z rgb , Z hs , And Into 16x16 pixel size patches respectively, then each patch is flattened into a one-dimensional vector, forming a unified dimension token and used for subsequent input, and the flattened vector is And And And And According to the corresponding pseudo-color image group, the RGB token T rgb Its dimension is 720x768; determine the pseudo-color token group Its dimension is 5x720x768.

[0016] Preferably, the step three, specifically comprising:

[0017] By reconstructing and convolution operation, T rgb And Reshape the two-dimensional feature map H rgb ,

[0018] Determine the similarity L rgb , Of the reshaped two-dimensional feature map H n;

[0019] The similarity L n is normalized to obtain the weight W n ;

[0020] According to the weight W n , H rgb , determine the complementary feature representation H;

[0021] H is reshaped into a complementary fusion token f fusion by convolution and reconstruction operations.

[0022] Preferably, the step three is implemented in particular by the following steps:

[0023] (301) T rgb and are reshaped into two-dimensional feature maps by reconstruction and convolution operations according to the following formula:

[0024]

[0025] wherein reshape(·) represents a reconstruction transformation operation, conv(·) represents a 1×1 convolution operation, H rgb represents a two-dimensional feature map on the RGB branch, represents a two-dimensional feature map of the pseudo-color branch, and n is an interval closed on the left and open on the right from 1 to N, and N represents the total number of feature maps;

[0026] (302) The similarity L n between the two modal branches is determined by L2 distance according to the following formula:

[0027]

[0028] wherein ‖·‖2 represents L2 norm calculation, L n represents the similarity between two features, wherein a higher value indicates that the difference between the two is greater. The subscript n represents the calculation result between the nth pseudo-color feature map and the RGB feature map;

[0029] (303) The L2 distance is normalized by using the softmax function according to the following formula:

[0030]

[0031] wherein W n represents the maximum weight generated by n elements, and ∑(·) represents the summation operation;

[0032] (304) According to the following formula, W is given to W n after Hrgb The complementary feature representation H is determined by adding:

[0033]

[0034] (305) H is reshaped into a complementary fusion token f by a convolution and reconstruction operation according to the following formula: fusion :

[0035] f fusion = reshape(conv(H)) (5).

[0036] Preferably, the step five specifically comprises:

[0037] According to F com , a residual output F1 is determined;

[0038] The F1 is subjected to feature enhancement by a channel space attention module to determine an enhanced feature F2;

[0039] The F2 is subjected to a convolution operation to determine the memory key-value pair K com and V M of the F M input into the memory prompt encoder of the current frame;

[0040] The F com is subjected to an operation to determine the query key-value pair K Q and V Q of the current frame;

[0041] The memory key K M in the historical memory pool is subjected to matrix multiplication with the query key K Q of the current frame in the memory prompt decoder, and a normalization operation is performed to generate a query attention map Map;

[0042] The Map is subjected to matrix multiplication with the reconstructed historical value V M , and a historical prompt value is extracted, which is spliced with the query value V Q of the current frame in the channel dimension to determine the memory prompt feature F mem of the current frame.

[0043] Preferably, the step five is specifically implemented by the following steps:

[0044] (501) F com is input into a first residual block of a memory prompt encoder, the residual block being composed of two 3x3 convolution layers, to determine a residual output F1 according to the following formula:

[0045] F1 = F com + Conv2(ReLU(Conv1(F com ))) (6)

[0046] wherein, Conv1(·) represents the first convolutional layer operation, Conv2(·) represents the second convolutional layer operation, the two convolutional layer structures are the same, the step is 1, and the padding is 1, ReLU(·) represents the ReLU activation function operation;

[0047] (502) The feature is enhanced by using a channel spatial attention module according to the following formula to determine the enhanced feature F2:

[0048]

[0049]

[0050] F2=(σ(W C )⊙F1)⊙σ(W S ) (9)

[0051] wherein, and represent the maximum pooling and average pooling operations in the spatial dimension respectively; and and are the pooling operations in the channel dimension; MLP(·) is a multi-layer perceptron operation composed of two linear layers, which does not change the feature dimension, and the channel attention W C is obtained by adding the outputs of the two MLPs; S Conv(·) is a convolutional layer operation with a convolution kernel size of 7, a padding of 3, and an output channel of 1, and outputs the spatial attention W C ; σ(·) represents the Sigmoid function operation, and ⊙ represents the element-wise multiplication. When σ(W S ) is the channel attention weight, the channel dimension broadcast is performed; when σ(W com ) is the spatial attention map, the spatial dimension broadcast is performed;

[0052] (503) The memory key-value pair K M and V M of F com input into the memory prompt encoder are determined by convolution operation according to the following formula, and are sent into the historical memory pool of the memory prompt decoder for storage:

[0053]

[0054] wherein, and are both 3x3 convolutional layer operations, the step and the padding are both 1, and only the number of output channels is different;

[0055] (504) The query key-value pair K com of the current frame F com is determined by convolution operation according to the following formula:Q and V Q And send it to the memory prompt decoder:

[0056]

[0057] in, and The parameter settings are the same as those of the previous convolutional layer.

[0058] (505) According to the following formula, the memory key K in the history memory pool is entered into the memory prompt decoder. M The query key K of the current frame Q Perform matrix multiplication and normalization to generate a query attention graph (Map):

[0059]

[0060] Where Softmax(·) represents the normalization operation. This represents matrix multiplication. The query attention graph (Map) dynamically assigns weights to each historical frame based on similarity to adaptively adjust the contribution of each frame to the current frame.

[0061] (506) According to the following formula, Map and the reconstructed historical value V are combined. M Perform matrix multiplication to extract historical prompt values, and then multiply these historical prompt values ​​with the current frame query value V. Q The memory cue features F of the current frame are determined by concatenating the data along the channel dimension. mem :

[0062]

[0063] Concat(·) represents the concatenation operation.

[0064] Compared to existing technologies, to achieve spectral complementarity in feature fusion, this invention employs a spectral complementarity generator to generate adaptive weights based on modal differences, thereby enabling cross-modal spectral cue learning and effectively improving the utilization rate of spectral information in hyperspectral modes. Furthermore, this invention introduces a memory cue network, which generates memory cue key-value pairs containing historical information through a memory cue encoder and stores them in a memory pool. Subsequently, a memory cue decoder is used to query and retrieve target features. The memory cue network achieves robust target state modeling based on historical frame information, overcoming problems such as target loss, difficulty in occlusion recovery, and decreased long-term tracking accuracy caused by the lack of temporal modeling capabilities in traditional tracking frameworks. Attached Figure Description

[0065] Figure 1 This is a flowchart of the present invention;

[0066] Figure 2 Fig. 16 is a 16-channel image of the template region Z of the first frame of the forest2 hyperspectral image in an embodiment of the present application; hs Fig. 17 is a single-channel visualization image of the search region of the 74th frame of the forest2 hyperspectral image in an embodiment of the present application;

[0067] Figure 3 Fig. 18 is a visualization result of the memory pool attention map of the 74th frame of the forest2 hyperspectral image in an embodiment of the present application;

[0068] Figure 4 Fig. 19 is a tracking target of the current frame of the hyperspectral image sequence obtained in the 74th frame of the forest2 hyperspectral image in an embodiment of the present application;

[0069] Figure 5 Fig. 20 is a specific implementation of the present application. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0071] The embodiment of the present application provides a hyperspectral target tracking method based on cross-modal fusion and memory prompt, which is:

[0072] Step one, load the RGB modal image and the hyperspectral modal image, divide the hyperspectral modal image into multiple groups of pseudo-color images according to the channel order, and the number of channels of each group of pseudo-color images is consistent with the RGB modal image; through the ground truth labeled in advance, determine the template region Z of the first frame of the RGB modal image rgb and the template region Z of the first frame of the hyperspectral modal image hs , determine the search region of the tth frame of the RGB modal image and the search region of the tth frame of the hyperspectral modal image through the size of the target obtained after completing target tracking of the t-1th frame, wherein t represents the frame number of the RGB modal image and the hyperspectral modal image, and t is an integer greater than or equal to 1;

[0073] ​Specifically, the hyperspectral dataset HOT2020 downloaded from the website (www.hsitracking.com) was selected, and the forest2 image sequence including the RGB image and the hyperspectral image taken in the same scene was selected. The forest2 image sequence has a total of 363 frames, and the hyperspectral image of forest2 includes 16 channels, and the resolution of a single channel is 1084*532 pixels. The RGB image and the hyperspectral image of the 74th frame of the forest2 sequence were loaded at the same time. Since the 12th band has an overexposure problem, the present application discards this band and uses the remaining 15 spectral bands. According to the channel order, it is divided into 5 groups of 3-channel pseudo-color images. Through the pre-labeled ground truth, the template region Z rgb and Z hs , Figure 2 is the 16-channel image of Z hs of the first frame of the forest2 hyperspectral image in the embodiment of the present application. Through the target size obtained after completing the target tracking of 73 frames, the search region and and of the RGB modality and the hyperspectral modality image is 2 times the target size of the previous frame, Figure 3 is the single-channel visualization image of Z of the 74th frame of the forest2 hyperspectral image in the embodiment of the present application, and the image resolution is 384*384 pixels.

[0074] Step two, Z rgb , Z hs , and are respectively subjected to Patchembedding operation to determine the flattened vectors and Then they are spliced according to the channel to determine the RGB token and the pseudo-color token group wherein n represents the number of pseudo-color image groups;

[0075] Specifically, the Patchembedding operation divides Z rgb and Z hs , and into 16*16 pixel patches, respectively, and then flattens each patch into a one-dimensional vector to form tokens of uniform dimensions and used for subsequent input. The flattened vectors are and and and and Perform a stitching operation based on the corresponding pseudo-color image groups to determine the RGB token T. rgb Its dimensions are 720×768; determine the pseudo-colored token group. Its dimensions are 5×720×768.

[0076] Step 3, via T rgb and Determine the complementary fusion token f fusion ;

[0077] This is achieved through the following steps:

[0078] (301) According to the following formula, T is reconstructed and convolutionally processed. rgb and Reconstructed into a two-dimensional feature map:

[0079]

[0080] Where reshape(·) represents the reconstruction transformation operation, conv(·) represents the 1×1 convolution operation, and H rgb Represents a two-dimensional feature map on the RGB branch. The two-dimensional feature map represents the pseudo-color branch, where n is the left-closed and right-closed interval from 1 to N, and N represents the total number of feature maps.

[0081] Specifically, since these linearized tokens cannot be directly processed in a spatial structure, we first reshape them to convert them into two-dimensional square feature maps. To compress the high-dimensional feature representation and reduce computational cost, we then apply a 1×1 convolution operation to reduce the channel dimension from 768 to 256. Furthermore, since five groups of pseudo-color images were divided, at this point... In this context, n is a left- or right-closed interval from 1 to 5, with a dimension of 5×720×256.

[0082] (302) The similarity L between the two modal branches is determined using the L2 distance according to the following formula. n for:

[0083]

[0084] Where, ||·||2 represents the L2 norm calculation. n This represents the similarity between two features, with higher values ​​indicating greater differences between them. The subscript n indicates the calculation result between the nth pseudo-color feature map and the RGB feature map.

[0085] Specifically, H rgb and L2 norm calculation is performed respectively, first point by point difference in channel dimension, then sum of squares of all difference values and square root, get {L1, L2, L3, L4, L5}, used to reflect the overall difference degree of the two in the whole two-dimensional feature map. Exemplarily, after retaining one decimal place, the L n value obtained in the 74th frame of the forest2 sequence is {10.2, 8.6, 11.1, 9.3, 7.8}.

[0086] (303) The L2 distance is normalized by using the softmax function according to the following formula:

[0087]

[0088] where W n represents the maximum weight generated by n elements, and Σ(·) represents the summation operation. It is worth noting that the negative exponential function is used to adjust the weight distribution. When the L2 distance is relatively small, it indicates that the similarity between the two modalities is high. In this case, a larger weight is assigned, thereby increasing the contribution of the high-confidence modality in the feature fusion process. This further effectively compensates for the limitations of the RGB modality in spectral information representation.

[0089] Specifically, in order to increase the spectral complementarity of hyperspectral images, after adjusting the weight distribution by using the negative exponential function, {W1, W2, W3, W4, W5} is obtained. Exemplarily, after retaining four decimal places, the W n value obtained in the 74th frame of the forest2 sequence is {0.1117, 0.2285, 0.0510, 0.1593, 0.4495}, it is worth noting that the value of L5 is the smallest, so the largest weight W5 is assigned.

[0090] (304) The H is assigned to W n and added to H rgb to determine the complementary feature representation H:

[0091]

[0092] Specifically, H sufficiently fuses the complementary spectral information provided by each pseudo-color map branch, while retaining the spatial detail information in the original RGB feature, and its dimension is 720x256.

[0093] (305) H is reshaped into a complementary fusion token f fusion according to the following formula by convolution and reconstruction operation:

[0094] f fusion = reshape(conv(H)) (5)

[0095] Wherein, the convolution and reconstruction operation is to convert the two-dimensional feature map into a token sequence, so as to facilitate the subsequent input of the Transformer encoder.

[0096] Specifically, f fusion The dimension of f

[0097] Step four, f fusion is sent into the Transformer for feature extraction to obtain the spectral complementary feature F com ;

[0098] Specifically, the Transformer encoder is a prior art, and the present application adopts ViT-B / 16 as the feature extraction network, adopts a 12-layer Transformer encoder structure, and the embedding dimension is 768 and the number of attention heads is 12. ViT-B / 16 can fully capture the long-distance dependency relationship and global context information between fused features, thereby improving the expression ability of the semantic complementary features between the two modalities. F com The dimension of F

[0099] Step five, according to F com , the memory prompt of the current frame is determined, and then combined with the historical memory pool module in the memory prompt decoder and the subsequent decoding operation, the memory feature F mem with historical frame information of the t-th frame is determined.

[0100] Specifically, the following steps are implemented:

[0101] (501) According to the following formula, F com is first input into the first residual block of the memory prompt encoder, and the residual block is composed of two 3x3 convolution layers to determine the residual output F1:

[0102] F1=F com +Conv2(ReLU(Conv1(F com ))) (6)

[0103] Wherein, Conv1(·) represents the first convolution layer operation, Conv2(·) represents the second convolution layer operation, the two convolution layer structures are the same, the step is 1, and the padding is 1. ReLU(·) represents the ReLU activation function operation.

[0104] Specifically, neither of the two convolution layers changes the channel of the input, and the dimension of F1 is still 720x768, realizing the residual connection with unchanged channel number.

[0105] (502) According to the following formula, the channel spatial attention module is used to enhance the feature to determine the enhanced feature F2:

[0106]

[0107]

[0108] F2 = (σ(W C ) ⊙ F1) ⊙ σ(W S ) (9)

[0109] wherein, and denote max-pooling and average-pooling operations in spatial dimensions, respectively; and and are pooling operations in channel dimension. MLP(·) is a multi-layer perceptron operation consisting of two linear layers, which does not change the feature dimension, and the channel attention W C is obtained by adding the outputs of the two MLPs. S Conv(·) is a convolutional layer operation with a kernel size of 7, padding of 3, and output channel of 1, which outputs the spatial attention W C . σ(·) denotes a Sigmoid function operation, and ⊙ denotes element-wise multiplication. When σ(W S ) is the channel attention weight, channel dimension broadcasting is performed; when σ(W S ) is the spatial attention map, spatial dimension broadcasting is performed.

[0110] Specifically, the channel-spatial attention module is used to enhance the features, so as to improve the perception ability of the network to key areas and significant features, suppress redundant information interference, and facilitate the generated memory key-value pair K M and V M to more accurately represent historical feature information. The dimension of F2 is still 720x768.

[0111] (503) determines the memory key-value pair K com and V M of F M input into the memory prompt encoder by convolution operation, and sends it into the historical memory pool of the memory prompt decoder for storage:

[0112]

[0113] wherein, and are 3x3 convolutional layer operations with a step size and padding of 1, except that the number of output channels is different. In addition, the memory pool stores the historical memory key-value pairs from the previous 50 frames, which are used for subsequent query and retrieval of historical information by the current frame to generate memory features with historical information.

[0114] Specifically, and The output channels are 64 and 384 respectively, therefore K M and V M The dimensions are 720×64 and 720×384, respectively. It is worth noting that after being sent to the memory pool for storage, K... M and V M The dimensions are T×720×64 and T×720×384 respectively. Here, T represents the historical memory key-value pairs of the first 50 frames stored in the memory pool, so T equals 50. Figure 4 This is a visualization of the memory pool attention map of the 74th frame of the forest2 hyperspectral image in an embodiment of the present invention.

[0115] (504) Determine the current frame F through a convolution operation according to the following formula. com Query key-value pair K Q and V Q And send it to the memory prompt decoder:

[0116]

[0117] in, and The parameter settings are the same as those of the previous convolutional layer.

[0118] Specifically, since the parameter settings for the convolutional layer here are the same as in the previous step, K Q and V Q The dimensions are 64×720 and 384×720, respectively.

[0119] (505) According to the following formula, the memory key K in the history memory pool is entered into the memory prompt decoder. M The query key K of the current frame Q Perform matrix multiplication and normalization to generate a query attention graph (Map):

[0120]

[0121] Where Softmax(·) represents the normalization operation. Representing matrix multiplication, the query attention graph (Map) can dynamically assign weights to each historical frame based on similarity, adaptively adjusting the contribution of each frame to the current frame.

[0122] Specifically, the normalized Map has dimensions of 50×720×720.

[0123] (506) According to the following formula, Map and the reconstructed historical value V are combined. M Perform matrix multiplication to extract historical prompt values, and then multiply these historical prompt values ​​with the current frame query value V. QConcatenate in the channel dimension to determine the memory prompt feature F of the current frame mem :

[0124]

[0125] Wherein, Concat(·) represents the concatenation operation.

[0126] Specifically, Map is multiplied with the reconstructed historical value V M After matrix multiplication, the dimension of the extracted historical prompt value is 384x720, and then the current frame query value V Q After concatenation in the channel dimension, F mem The dimension of is 768x720.

[0127] Step six, F mem is sent into the prediction head network to obtain the predicted bounding box position, and the target tracking of the t-th hyperspectral image is completed. Each frame of hyperspectral image in the hyperspectral image sequence is loaded in turn to complete the target tracking of the entire hyperspectral image sequence.

[0128] Specifically, in the present patent, a center-based prediction head network is used as the prediction head, which produces three outputs. Respectively, a classification score map is used to estimate the probability of target existence; a target center offset map is used to refine the predicted position; a scale change map is used to compensate for the change of target size. All values are normalized to the range of [0, 1], and the final target position is determined by the position with the highest score in the classification score map. In the 74th frame of the forest2 hyperspectral image sequence, the maximum index value is 353, and the coordinates of the predicted bounding box are (270, 42, 20, 49). Among them, 270 represents the horizontal coordinate of the upper left corner of the target box, and 42 represents the vertical coordinate of the upper left corner of the target box. The size of the predicted target box is 20x49, 45 represents the width of the target box, and 67 represents the height of the target box. Figure 5 In the present patent embodiment, the tracking target of the current frame of hyperspectral image obtained in the 74th frame of the forest2 hyperspectral image sequence.

[0129] The application provides a hyperspectral target tracking method based on cross-modal fusion and memory prompt. First, considering that a single mode cannot effectively consider the expression ability of spatial structure and spectral discriminant information, a spectral complementary generator is proposed. Adaptive weights are generated by using the difference between the two modes, not only considering the spatial detail features of the RGB mode, but also fully utilizing the rich spectral prompt of the hyperspectral mode, avoiding the loss of spectral information caused by dimension reduction. Therefore, the utilization efficiency of spectral information is significantly improved, and cross-modal fusion is realized. In addition, the application adopts the recently popular Transformer architecture for tracking, and proposes to combine the memory prompt network to introduce the time clue, generates and retrieves the memory prompt from the historical frame, realizes dynamic reasoning, and solves the problems of ignoring historical information and insufficient modeling of space-time context in the traditional architecture.

[0130] The above merely describes preferred embodiments of the application and is not intended to limit the protection scope of the application.

Claims

1. A hyperspectral target tracking method based on cross-modal fusion and memory cues, characterized in that, The method is as follows: Step 1: Load the RGB modal image and the hyperspectral modal image. Divide the hyperspectral modal image into multiple groups of pseudo-color images according to channel order. The number of channels in each group of pseudo-color images is the same as that in the RGB modal image. Determine the template region Z of the first frame of the RGB modal image using pre-labeled ground truth values. rgb The template region Z of the first frame hyperspectral modal image hs The search area for the RGB mode in frame t is determined by using the target size obtained after target tracking is completed in frame t-1. Search region of hyperspectral modal images Where t represents the number of frames in the RGB mode and hyperspectral mode images, and t is an integer greater than or equal to 1; Step 2, Z rgb Z hs , and Perform the Patchembedding operation separately to determine the flattened vector. and Then concatenate them by channel to determine the RGB token. and pseudo-colored token group Where n represents the number of pseudo-color image groups; Step 3, via T rgb and Determine the complementary fusion token f fusion ; Step 4, for f fusion The data is fed into a Transformer for feature extraction to obtain the spatial-spectral complementary features F. com ; Step 5, according to F com The memory cue of the current frame is determined, and then, in conjunction with the history memory pool module in the memory cue decoder and subsequent decoding operations, the memory feature F of the t-th frame containing historical frame information is determined. mem ; Step Six, F mem The target is fed into the prediction head network to obtain the predicted bounding box position and complete the target tracking of the t-th frame of the hyperspectral image. Then, each frame of the hyperspectral image in the hyperspectral image sequence is loaded sequentially to complete the target tracking of the entire hyperspectral image sequence.

2. The hyperspectral target tracking method based on cross-modal fusion and memory cueing according to claim 1, characterized in that, Step two specifically includes: the Patchembedding operation will convert Z... rgb Z hs , and The data is divided into 16×16 pixel patches. Each patch is then flattened into a one-dimensional vector to form a token of uniform dimension, which is used as subsequent input. The resulting flattened vector is... and And and and Perform a stitching operation based on the corresponding pseudo-color image groups to determine the RGB token T. rgb Its dimensions are 720×768; determine the pseudo-colored token group. Its dimensions are 5×720×768.

3. The hyperspectral target tracking method based on cross-modal fusion and memory cueing according to claim 1 or 2, characterized in that, Step three specifically includes: T is reconstructed and convolutionally processed. rgb and Reconstruct the two-dimensional feature map H respectively rgb , Determine the reshaped two-dimensional feature map H rgb , Similarity L n ; For the similarity L n After normalization, the weight W is obtained. n ; According to the weight W n H rgb , Determine the complementary feature representation H; H is reshaped into a complementary fusion token f through convolution and reconstruction operations. fusion .

4. The hyperspectral target tracking method based on cross-modal fusion and memory cueing according to claim 3, characterized in that, Step three is implemented through the following steps: (301) According to the following formula, T is reconstructed and convolutionally processed. rgb and Reconstructed into a two-dimensional feature map: Where reshape(·) represents the reconstruction transformation operation, conv(·) represents the 1×1 convolution operation, and H rgb Represents a two-dimensional feature map on the RGB branch. The two-dimensional feature map represents the pseudo-color branch, where n is the left-closed and right-closed interval from 1 to N, and N represents the total number of feature maps. (302) The similarity L between the two modal branches is determined using the L2 distance according to the following formula. n for: Where ||·||2 represents the L2 norm calculation, L n This represents the similarity between two features, where a higher value indicates a greater difference between them. The subscript n indicates the calculation result between the nth pseudo-color feature map and the RGB feature map. (303) The L2 distance is normalized using the softmax function according to the following formula: Among them, W n ∑(·) represents the final weight generated from n elements; (304) According to the following formula, Give W n Later with H rgb Add them together to determine the complementary feature representation H: (305) H is reshaped into a complementary fusion token f by convolution and reconstruction operations according to the following formula. fusion : f fusion =reshape(conv(H)) (5)。 5. The hyperspectral target tracking method based on cross-modal fusion and memory cueing according to claim 4, characterized in that, Step five specifically includes: According to F com Determine the residual output F1; F1 is enhanced using the channel space attention module, and the enhanced feature F2 is determined. Perform a convolution operation on F2 to determine the F of the current frame input into the memory cue encoder. com memory key-value pairs K M and V M ; For F com Perform the operation to determine the query key-value pair K of the current frame. Q and V Q ; In the memory prompt decoder, the memory key K in the history memory pool is... M The query key K of the current frame Q Perform matrix multiplication and normalization to generate a query attention graph (Map); Compare the Map with the reconstructed historical value V M Perform matrix multiplication to extract historical prompt values, and then multiply these historical prompt values ​​with the current frame query value V. Q The memory cue features F of the current frame are determined by concatenating the data along the channel dimension. mem .

6. The hyperspectral target tracking method based on cross-modal fusion and memory cueing according to claim 5, characterized in that, Step five is implemented through the following steps: (501) F is calculated according to the following formula. com The input is fed into the first residual block of the memory cue encoder, which consists of two 3×3 convolutional layers, to determine the residual output F1: F1=F com +Conv2(ReLU(Conv1(F com ))) (6) Where Conv1(·) represents the first convolutional layer operation, Conv2(·) represents the second convolutional layer operation, the two convolutional layers have the same structure, stride of 1, padding of 1, and ReLU(·) represents the ReLU activation function operation; (502) The enhanced feature F2 is determined by using the channel space attention module according to the following formula: F2=(σ(W C )⊙F1)σ(W S ) (9) in, and These represent the max pooling and average pooling operations in the spatial dimension, respectively; and and This is a pooling operation on the channel dimension; MLP(·) is a multilayer perceptron operation consisting of two linear layers, which does not change the feature dimension, and obtains the channel attention W by summing the outputs of the two MLPs. C Conv(·) is a convolutional layer operation with a kernel size of 7, padding of 3, and output channel of 1. The output spatial attention W S σ(·) represents the Sigmoid function operation, and ⊙ represents element-wise multiplication. C When σ(W) is the channel attention weight, channel-dimensional broadcasting is performed; when σ(W) is the channel attention weight, channel-dimensional broadcasting is performed. S When the graph is a spatial attention map, spatial dimension broadcasting is performed; (503) The F of the current frame input into the memory cue encoder is determined by the convolution operation according to the following formula. com memory key-value pairs K M and V M And send it to the history memory pool of the memory cue decoder for storage: in, and Both are 3×3 convolutional layer operations, with stride and padding both being 1, only the number of output channels is different; (504) Determine the current frame F through a convolution operation according to the following formula. com Query key-value pair K Q and V Q And send it to the memory prompt decoder: in, and The parameter settings are the same as those of the previous convolutional layer; (505) According to the following formula, the memory key K in the history memory pool is entered into the memory prompt decoder. M The query key K of the current frame Q Perform matrix multiplication and normalization to generate a query attention graph (Map): Where Softmax(·) represents the normalization operation. This represents matrix multiplication. The query attention graph (Map) dynamically assigns weights to each historical frame based on similarity to adaptively adjust the contribution of each frame to the current frame. (506) According to the following formula, Map and the reconstructed historical value V are combined. M Perform matrix multiplication to extract historical prompt values, and then multiply these historical prompt values ​​with the current frame query value V. Q The memory cue features F of the current frame are determined by concatenating the data along the channel dimension. mem : Concat(·) represents the concatenation operation.

Images