Hyperspectral Video Object Tracking Method Based on Dynamic Spectrum Sensing Module

Abstract

The present invention discloses a hyperspectral video target tracking method based on a dynamic spectral perception module. First, the t-th hyperspectral image in the hyperspectral image sequence is read in, and the hyperspectral gradient map G of the t-th hyperspectral image is determined by matrix subtraction. t , according to G t to determine the spectral gradient vector #imgabs0# of the j-th pixel in G t and the spectral gradient vector G t (k) of the k-th clustering center, calculate the spectral angle distance d between #imgabs1# and G t (k). j,k According to d j,k to judge the clustering center to which the j-th pixel belongs, set the number of iterations q and calculate the loss function ε. Repeat the above steps until ε and q meet the conditions. According to the final result, a single-channel grayscale image after dimensionality reduction is obtained. The grayscale image is sent into a Siamese network for feature extraction, and the extracted features are adjusted to features suitable for a specific task through 1×1 convolution, and classification features P cls and regression features P reg are generated by cross-correlation. cls and SPM reg of two dynamic spectral perception modules respectively enhance P cls and P reg into P cls‑en and P reg‑en respectively. P cls‑en and P reg‑en determine the target position through the prediction head network.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a hyperspectral video target tracking method based on a dynamic spectral sensing module. Background Art

[0002] As a research hotspot in the field of computer vision, target tracking is widely applied in industries such as autonomous driving, security monitoring, and sports event live broadcast; for a video or sequence, an object of interest is generally marked in the initial frame using a rectangular bounding box, and the task of a target tracking algorithm is to predict the position and size of the object of interest in subsequent frames.

[0003] Currently, tracking algorithms based on visible light videos occupy the mainstream position; however, when the colors of the target and the background are similar, tracking algorithms based on visible light videos cannot accurately predict the target; while hyperspectral images with hundreds or thousands of narrow bands contain rich spectral information, and spectral information can be used to effectively identify targets with similar colors but different materials. Recently, thanks to the progress of hyperspectral imaging technology, hyperspectral cameras can capture hyperspectral videos at a stable frame rate, which has led to the gradual rise of tracking algorithms developed based on hyperspectral videos.

[0004] In the field of single-object tracking, tracking algorithms can be roughly divided into correlation filtering-based algorithms and deep learning-based algorithms. Although correlation filtering-based algorithms have a relatively fast tracking speed, their tracking accuracy still needs to be improved; with the wide application of deep learning methods, tracking algorithms based on Siamese networks have developed rapidly. Such algorithms not only have high real-time performance but also have satisfactory tracking accuracy. According to the structure of the prediction head, tracking algorithms based on Siamese networks can be roughly divided into two types: anchor-free tracking algorithms and anchor-based tracking algorithms; the concept of anchor boxes was first proposed in Faster R-CNN, and this algorithm classifies and regresses anchor boxes through a region proposal network. Some Siamese network trackers using anchor boxes have also shown great potential in terms of tracking accuracy; however, the use of anchor boxes requires pre-defining the size, aspect ratio, and number of anchor boxes, which may limit the generalization performance of the model; while trackers using anchor-free models no longer need to adjust and design anchor boxes, but directly predict the position and size of the target, so they can more flexibly adapt to the requirements of different scenarios. Summary of the Invention

[0005] In view of this, the main object of the present invention is to provide a hyperspectral video target tracking method based on a dynamic spectral sensing module.

[0006] To achieve the above object, the technical solution of the present invention is implemented as follows:

[0007] An embodiment of the present invention provides a hyperspectral video target tracking method based on a dynamic spectral sensing module, and the method is as follows:

[0008] Step 1: Read the t-th hyperspectral image in the hyperspectral image sequence and determine the search region S of the t-th hyperspectral image t ; where S t represents a three-dimensional matrix with dimensions B×H s ×W s , t represents the serial number of the hyperspectral image frame in the hyperspectral image sequence, t is an integer greater than or equal to 1, B represents the number of bands of S t , H s represents the number of rows of pixels in each band image on S t , W s represents the number of columns of pixels in each band image on S t ;

[0009] Step 2: Determine the hyperspectral gradient map G of S t by matrix subtraction, and determine the spectral gradient vector of the j-th pixel in G t ; t

[0010] Step 3: Establish a coordinate system for S t in the spatial dimension;

[0011] Step 4: Determine the spectral gradient vector G t (k) of the k-th clustering center;

[0012] Step 5: Determine the spectral angle distance d between t and G j,k (k), and obtain the spectral angle distance set d j of the j-th pixel;

[0013] Step 6: Determine b(j) through the smallest spectral angle distance in d j ;

[0014] Step 7: Determine the clustering center to which the j-th pixel belongs through b(j), determine the number of pixels N k contained in the k-th clustering center C k , and average the spectral gradient vectors of all pixels in C k to obtain the central spectral gradient vector of C k ;

[0015] Step 8: Set and calculate the loss function ε;

[0016] Step 9: Set the number of iterations q, and repeat Steps 1 to 8 until the set number of iterations q is reached or ε monotonically decreases within q times to obtain a minimum value, and then stop the iteration to obtain the clustering result after stopping the iteration;​

[0017] Step Ten: Generate a dynamic search mask S using the clustering results after stopping iteration hard and the target mask T hard , and obtain the single-channel grayscale image after dimensionality reduction;

[0018] Step Eleven: Copy the single-channel grayscale image into a three-channel grayscale image and then send it into a Siamese network for feature extraction; the Siamese network includes two branches, namely the template branch and the search branch, and both branches use ResNet50 with the same parameters for feature extraction, denotes the feature extracted from the template branch, denotes the feature extracted from the search branch;

[0019] Step Twelve: Obtain the concatenated features and

[0020] Step Thirteen: Use a 1×1 convolution to adjust the deep features extracted from the conv5-3 layer of ResNet50 in the template branch to and Adjust the deep features extracted from the conv5-3 layer of ResNet50 in the search branch to and where and respectively represent the features for classification and regression in the search branch, and respectively represent the features for classification and regression in the template branch;

[0021] Step Fourteen: and Generate the classification feature P using deep-level cross-correlation cls , and Generate the regression feature P using deep-level cross-correlation reg ;

[0022] Step Fifteen: Pass P cls and P reg through the classification version SPM of the dynamic spectral perception module cls and the regression version SPM of the dynamic spectral perception module reg respectively for feature enhancement to become the enhanced classification feature P cls-en and the enhanced regression feature P reg-en ;

[0023] Step Sixteen: P cls-en passes through the classification branch to output the classification score map Mcls , P reg-en Output the regression score map M through the regression branch reg and the target quality estimation score map M tar ;

[0024] Step Seventeen: M tar and M reg Are weighted and fused into the final score map, and the location of the target is determined according to the final score map.

[0025] In the above solution, Step Two is specifically implemented through the following steps:

[0026] Perform matrix subtraction on the adjacent band images of S t to determine the spectral gradient vector of the j-th pixel in the t-th frame hyperspectral gradient map G t as is

[0027]

[0028] where j represents the index of the pixel in the spatial dimension, and the value range of j is from 1 to H s ×W s , represents the spectral gradient value on the i-th band, and are respectively the gray values of the j-th pixel in S t on the (i + 1)-th band and the i-th band. B represents the number of bands of the hyperspectral image, and i represents the band index.

[0029] In the above solution, Step Five is specifically implemented through the following steps:

[0030] Calculate and the spectral angle distance d t between G j,k , and obtain d j as

[0031]

[0032] where G t (k) represents the spectral gradient vector corresponding to the k-th clustering center, k represents the index of the clustering center, the value range of k is from 1 to K, K represents the number of clustering centers, and d j represents the set of spectral angle distances between the j-th pixel on S t and the K clustering centers. ||·|| represents the vector modulus operation, and (·) T represents the vector transpose.

[0033] In the above solution, Step Six is specifically implemented through the following steps:

[0034] According to d j the minimum spectral angular distance min(d j,k ) to determine the label b(j) of the cluster center to which the j-th pixel belongs as

[0035]

[0036] where b(j) is an integer, the value range of b(j) is from 1 to K, min(·) represents the minimum operation, and the value range of j is from 1 to H s ×W s .

[0037] In the above solution, the seventh step is specifically implemented through the following steps:

[0038] (701) Traverse each pixel on S t and determine whether b(j) is equal to k; if they are equal, the j-th pixel corresponding to b(j) is assigned to the k-th cluster C k and set N(j) k to 1; otherwise, the pixel does not belong to C k , N(j) k is set to 0:

[0039]

[0040] where N(j) k represents the quantity feature of each pixel on S t ;

[0041] (702) Accumulate N(j) k in C k to obtain the number of pixels N k contained in C k , and the value range of k is from 1 to 5;

[0042] (703) Average the spectral gradient vectors of all pixels in C k to obtain the central spectral gradient vector of C k

[0043]

[0044] where ∑ represents the accumulation operation, and the value range of k is from 1 to K.

[0045] In the above solution, the eighth step is specifically implemented through the following steps:

[0046] Define the loss function ε as shown in the following formula

[0047] ​

[0048] Among them, the loss function ε represents the sum of squares of min(d j,k ), where the value range of j is from 1 to H s ×W s .

[0049] In the above solution, step ten is specifically implemented through the following steps:

[0050] (1001) Based on the clustering result after stopping iteration, define the pixels classified into the target cluster as target pixels, and the remaining pixels as background pixels. Set the gray value of the target pixels to 1 and the gray value of the background pixels to 0, and generate a dynamic search mask S according to the above definition hard ; among them, S hard has a size of H s ×W s ×1;

[0051] (1002) Take the S hard of the initial frame, centered on the centroid, and crop the four outer edges into a target mask T with a size of H T ×W T ×1 hard ;

[0052] Among them, H T and W T respectively represent the height and width of the ground truth bounding box of the initial frame;

[0053] (1003) Calculate the average of the spectral vectors of all target pixels to obtain the target spectrum C f ;

[0054] (1004) Use the naive correlation operation between C f and the spectral vector of each pixel on S t , and take the result of the naive correlation as the 1-band gray image after dimensionality reduction.

[0055] In the above solution, step twelve is specifically implemented through the following steps:

[0056] (1201) Store the first 1 / 8 channels of the feature map extracted by the conv1 layer of ResNet50 at the (t - 1)-th frame through the feature memory;

[0057] (1202) Store the first 1 / 8 channels of the feature map extracted by the conv5-3 layer of ResNet50 at the (t - 1)-th frame through the feature memory;

[0058] (1203) After the feature extraction of ResNet50 is completed at the t-th frame, the first 1 / 8 channels of the feature maps output by conv1 and conv5-3 are replaced with the cached feature maps, and the concatenated features are and

[0059] In the above solution, the SPM in step fifteen cls Enhance P cls to P cls-en , which is specifically implemented through the following steps:

[0060] (1501) Respectively compress the channels of P cls and through two 1×1 convolutions. The compressed features are θ cls and

[0061] where θ cls has a size of has a size of C represents the number of channels of P cls and . In the present invention, C is 256;

[0062] (1502) Reshape θ cls into Reshape

[0063] where has a size of has a size of

[0064] (1503) Obtain the similarity matrix V according to the following formula

[0065]

[0066] where represents matrix multiplication, and the size of V is H1W1×H2W2;

[0067] (1504) Normalize V through the softmax function to obtain the normalized similarity matrix where has a size of H1W1×H2W2;

[0068] (1505) Downsample T hard to reduce its spatial dimension to H2×W2;

[0069] (1506) Convert T hard with a size of H2×W2 to the target soft mask Tsoft ;

[0070] Among them, the size of T soft is H2×W2;

[0071] (1507)T soft is reshaped into

[0072] Among them, the size of is 1×H2W2;

[0073] (1508) The spectral perception attention vector O of the classification feature is obtained according to the following formula

[0074]

[0075] Among them, the size of O is H1W1×1;

[0076] (1509) After restoring the size of O to H1×W1, the channel dimension is restored through a 1×1 convolution to obtain the spectral perception attention vector O' of size C×H1×W1 for enhancing the classification feature;

[0077] (1510) P cls is added to O' through matrix addition to obtain the enhanced classification feature P cls-en .

[0078] In the above solution, the SPM in step fifteen reg enhances P reg to P reg-en , which is specifically implemented through the following steps:

[0079] (1601) The channel dimension of P reg is compressed through a 1×1 convolution, and the compressed feature is θ reg ;

[0080] Among them, the size of θ reg is C * ×H1×W1;

[0081] (1602) θ reg is reshaped into reshaped into

[0082] Among them, the size of is C * ×H1W1, the size of is C * ×H3W3;

[0083] (1603) The similarity matrix Q is obtained according to the following formula

[0084]

[0085] Among them, the size of Q is H1W1×H3W3;

[0086] (1604) Normalize Q through the softmax function to obtain the normalized similarity matrix

[0087] Among them, has a size of H1W1×H3W3;

[0088] (1605) Upsample S hard to expand its spatial dimension to H3×W3;

[0089] (1606) Convert S hard to the dynamic search soft mask S soft ;

[0090] Among them, the size of S soft is 1×H3×W3;

[0091] (1607) Reshape S soft into

[0092] Among them, has a size of 1×H3W3;

[0093] (1608) Obtain the spectral perception attention vector Y of the regression feature according to the following formula

[0094]

[0095] Among them, the size of Y is H1W1×1;

[0096] (1609) After restoring the size of Y to H1×W1, then complete the reduction of the channel dimension through 1×1 convolution to obtain the spectral perception attention matrix Y′ of size C×H1×W1 for enhancing the regression feature;

[0097] (1610) Add P reg to Y′ through matrix addition to obtain the enhanced regression feature P reg-en .

[0098] Compared with the prior art, the present invention utilizes the difference in gradients between the spectral vectors of the target and the background to complete the dimensionality reduction of the hyperspectral image, and uses the result of the dimensionality reduction as a dynamic information flow to input into the dynamic spectral perception module. The dynamic spectral perception module designed by the present invention can effectively perceive the position of the target and help the network achieve stable target tracking. Description of the Drawings

[0099] Figure 1 In the present invention, it is the flowchart of the dimensionality reduction method based on spectral gradient vector clustering;

[0100] Figure 2 It is the overall tracking framework of the hyperspectral video target tracking method based on the dynamic spectral sensing module;

[0101] Figure 3 In the present invention, it is the classification version SPM of the dynamic spectral sensing module cls ;

[0102] Figure 4 In the present invention, it is the regression version SPM of the dynamic spectral sensing module reg ;

[0103] Figure 5 In the embodiment of the present invention, it is the ground truth bounding box of the first frame hyperspectral image in the toy sequence;

[0104] Figure 6 In the embodiment of the present invention, it is the template region bounding box of the first frame hyperspectral image in the toy sequence;

[0105] Figure 7 In the embodiment of the present invention, it is the search region bounding box of the 27th frame hyperspectral image in the toy sequence;

[0106] Figure 8 In the embodiment of the present invention, it is the target box of the 27th frame hyperspectral image in the toy sequence;

[0107] Figure 9 In the embodiment of the present invention, it is the spectral vector of the pixel at the 4th row and 21st column (i.e., the pixel corresponding to j equal to 84) in the search region of the 27th frame hyperspectral image in the toy sequence;

[0108] Figure 10 In the embodiment of the present invention, it is the spectral gradient vector of the pixel at the 4th row and 21st column (i.e., the pixel corresponding to j equal to 84) in the search region of the 27th frame hyperspectral image in the toy sequence

[0109] Figure 11 In the embodiment of the present invention, it is the spectral gradient vector G corresponding to the 5 initial clustering centers in the search region of the 27th frame hyperspectral image in the toy sequence t (k);

[0110] Figure 12 In the embodiment of the present invention, it is the central spectral gradient vector corresponding to the 5 clusters in the search region of the 27th frame hyperspectral image in the toy sequence

[0111] Figure 13In the embodiments of the present invention, the target spectrum C of the search region of 27 hyperspectral images in the toy sequence f . Specific implementation

[0112] In order to make the objectives, technical solutions and advantages of the present invention more clear and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific implementations described herein are only used to solve the present invention and are not used to limit the present invention.

[0113] The embodiments of the present invention provide a hyperspectral video target tracking method based on a dynamic spectrum perception module, which is implemented through the following steps:

[0114] Step 1: Read the t-th hyperspectral image in the hyperspectral image sequence, and determine the search region S of the t-th hyperspectral image t .

[0115] Among them, S t is a three-dimensional matrix with a size of B×H s ×W s , t represents the serial number of the hyperspectral image frame in the hyperspectral image sequence, t is an integer greater than or equal to 1, B represents the number of bands of S t , H s represents the number of rows of pixels, and W s represents the number of columns of pixels;

[0116] Specifically, read the 27th hyperspectral image in the toy sequence, and determine the search region S of the 27th hyperspectral image 27 , the loaded toy sequence has 376 frames, the hyperspectral image resolution is 135×271 pixels, B is 16 in the present invention, H s is 83, and W s is 113. Figure 7 In the embodiments of the present invention, it is the bounding box of the search region of the 27th hyperspectral image in the toy sequence, and the image block within the bounding box is the search region S 27 . During the whole sequence, the human hand controls the toy to rotate in the air, and there are phenomena such as light changes and occlusions during this period.

[0117] Step 2: Perform matrix subtraction on the adjacent band images of S t , and determine that the spectral gradient vector of the j-th pixel in the t-th hyperspectral gradient map G t is For

[0118]

[0119] Among them, is the spectral gradient vector of the j-th pixel in G t , is the spectral gradient value at the i-th band, and are respectively the gray values of the j-th pixel in S t at the (i + 1)-th band and the i-th band, B represents the number of bands of the hyperspectral image, i represents the band index, and the value range of i is from 1 to B - 1;

[0120] Specifically, perform matrix subtraction on adjacent band images of S 27 . It is known that the gray values of the spectral vector of the pixel at the 4th row and 21st column (i.e., the pixel corresponding to j equal to 84) in the search area of 27 frames of hyperspectral images in the toy sequence at the 1st band to the 16th band are {38, 32, 27, 30, 35, 36, 42, 49, 51, 62, 59, 65, 68, 72, 80, 102} in sequence. Then, the spectral gradient vector of the 84th pixel in the 27th frame of hyperspectral gradient map G 27 at the 1st band to the 15th band has gradient values {-6, -5, 3, 5, 1, 6, 7, 2, 11, -3, 6, 3, 4, 8, 22} in sequence. is the spectral vector of the pixel at the 4th row and 21st column (i.e., the pixel corresponding to j equal to 84) in the search area of 27 frames of hyperspectral images in the toy sequence in the embodiment of the present invention. Figure 9 is the spectral gradient vector of the pixel at the 4th row and 21st column (i.e., the pixel corresponding to j equal to 84) in the search area of 27 frames of hyperspectral images in the toy sequence in the embodiment of the present invention Figure 10

[0121] Step 3: Establish a coordinate system in the spatial dimension for S t , and determine that the initial spatial coordinates of the 1st clustering center on S t are (0, 0), the initial spatial coordinates of the 2nd clustering center on S t are (0, Hs - 1), the initial spatial coordinates of the 3rd clustering center on S t are (Ws - 1, 0), the initial spatial coordinates of the 4th clustering center on S t are (Ws - 1, Hs - 1), and the initial spatial coordinates of the 5th clustering center on S t are (x0, y0). Where (x0, y0) are the spatial coordinates of the centroid of the target box of the (t - 1)-th frame of hyperspectral image on S t ;

[0122] Specifically, establish a coordinate system in the spatial dimension for S 27 , and determine that the initial spatial coordinates of the 1st clustering center on S 27 are (0, 0), the initial spatial coordinates of the 2nd clustering center on S27 The initial spatial coordinates on it are (0, 82), and the 3rd clustering center is at S 27 The initial spatial coordinates on it are (112, 0), and the 4th clustering center is at S 27 The initial spatial coordinates on it are (112, 82) and the 5th clustering center is at S 27 The initial spatial coordinates on it are (57, 42). Among them, (57, 42) is the centroid of the target bounding box of the 26th frame of the hyperspectral image in the toy sequence at S 27 The spatial coordinates on it.

[0123] Step Four: Determine the spectral gradient vector G of the kth clustering center t (k)

[0124] Among them, the value range of k is from 1 to K, where K represents the number of clustering centers, and in the present invention, K is 5.

[0125] Specifically, G 27 (1) represents the spectral gradient vector corresponding to the 1st clustering center, G 27 (1) The gradient values in the 1st to 15th bands are successively {-5, -7, 2, 6, 3, 8, 5, 4, 13, -5, 4, 8, 2, 14, 16}. G 27 (2) represents the spectral gradient vector corresponding to the 2nd clustering center, G 27 (2) The gradient values in the 1st to 15th bands are successively {-3, -8, 5, 1, 3, 7, 2, -2, 9, -8, 5, 6, 7, 17, 13}. G 27 (3) represents the spectral gradient vector corresponding to the 3rd clustering center, G 27 (3) The gradient values in the 1st to 15th bands are successively {-7, -5, 8, 4, 5, 3, 9, 1, 14, -8, 4, 6, 5, 21, 18}. G 27 (4) represents the spectral gradient vector corresponding to the 4th clustering center, G 27 (4) The gradient values in the 1st to 15th bands are successively {-7, -3, 1, 6, -4, 7, 11, 6, 7, -6, 3, 12, 4, -7, 12}. G 27 (5) represents the spectral gradient vector corresponding to the 5th clustering center, G 27 (5) The gradient values in the 1st to 15th bands are successively {-6, -4, 2, 5, 2, 7, 6, 2, 10, -3, 5, 2, 5, 7, 20}. Figure 11 In the embodiment of the present invention, the spectral gradient vectors G corresponding to the 5 initial clustering centers in the search area of the 27-frame hyperspectral image in the toy sequence t (k).

[0126] Step Five: Calculate the spectral angular distance d t between and G j,k and obtain d j as

[0127]

[0128] where d j represents the set of spectral angular distances between the j-th pixel on S t and the K clustering centers, ||·|| represents the vector modulus operation, and (·) T represents the vector transpose;

[0129] Specifically, is 30.40, ||G 27 (1)|| is 30.95, ||G 27 (2)|| is 29.63, ||G 27 (3)|| is 37.04, ||G 27 (4)|| is 27.64, ||G 27 (5)|| is 28.03. The spectral angular distance d 27 between and G 84，1 (1) is 0.38, The spectral angular distance d 27 between and G 84，2 (2) is 0.58, The spectral angular distance d 27 between and G 84，3 (3) is 0.47, The spectral angular distance d 27 between and G 84，4 (4) is 0.79, The spectral angular distance d 27 between and G 84，5 (5) is 0.10, and d 84 is {0.38, 0.58, 0.47, 0.79, 0.10}.

[0130] Step Six: According to the minimum spectral angular distance min(d j ) in, determine the label b(j) of the clustering center to which the j-th pixel belongs as j,k

[0131]

[0132] where b(j) is an integer, the value range of b(j) is from 1 to K, min(·) represents the minimum value operation, and the value range of j is from 1 to H s ×W​s ;

[0133] Specifically, d 84 The minimum spectral angular distance min(d 84,k ) is d 84,5 , the value is 0.10, so the label b(84) of the cluster center to which the 84th pixel belongs is 5.

[0134] Step 7: Determine the kth cluster center C through b(j) k The number of pixels contained N k , according to N k Get C k The central spectral gradient vector

[0135] (701) Traverse S t For each pixel on the cluster, determine whether b(j) and k are equal. If they are equal, the jth pixel corresponding to b(j) is classified into the kth cluster C. k And N(j) k Set to 1.

[0136] Otherwise the pixel does not belong to C k ,N(j) k Set to 0.

[0137]

[0138] Among them, N(j) k Indicates S t The number of features for each pixel.

[0139] Specifically, it is known that b(84) is 5, b(84) is not equal to 1, so N(84)1 is 0. b(84) is not equal to 2, so N(84)2 is 0. b(84) is not equal to 3, so N(84)3 is 0. b(84) is not equal to 4, so N(84)4 is 0. b(84) is equal to 5, so N(84)5 is 1, and the pixel in the 4th row and 21st column of the search area of the 27 frames of hyperspectral images in the toy sequence (i.e., the pixel corresponding to when j is equal to 84) is classified into the 5th cluster C5.

[0140] (702) Accumulate C k N(j) in k , according to the following formula, we can get C k The number of pixels N contained in k

[0141]

[0142] The value of k ranges from 1 to K.

[0143] Specifically, S27 The spatial resolution of 27 is 83×113, and there are a total of 9,379 pixels. The number of pixels N1 included in C1 is 1,641, the number of pixels N2 included in C2 is 1,315, the number of pixels N3 included in C3 is 1,875, the number of pixels N4 included in C4 is 2,043, and the number of pixels N5 included in C5 is 2,505.

[0144] (703) For C k The average of the spectral gradient vectors of all pixels in k is obtained to get the central spectral gradient vector of

[0145]

[0146] Among them, ∑ represents the accumulation operation, and the value range of k is from 1 to K.

[0147] Specifically, the central spectral gradient vector of C1 The gradient values in the 1st to 15th bands are {-4, -6, 2, 8, -5, 9, 8, 4, 11, -3, 4, 10, 7, -5, 15} in sequence. The central spectral gradient vector of C2 The gradient values in the 1st to 15th bands are {-11, 2, -1, 4, -8, 7, 14, 9, 2, -5, 6, 17, 1, -5, 13} in sequence. The central spectral gradient vector of C3 The gradient values in the 1st to 15th bands are {-9, 2, 7, 5, -2, 3, 12, 5, 9, -4, 6, 14, 7, -5, 14} in sequence. The central spectral gradient vector of C4 The gradient values in the 1st to 15th bands are {-12, -5, 4, 3, -4, 5, 14, 5, 2, -5, 4, 17, 5, -3, 11} in sequence. The central spectral gradient vector of C5 The gradient values in the 1st to 15th bands are {-5, -7, 3, 9, -5, 9, 16, 3, 4, -9, 5, 15, 7, -3, 16} in sequence. Figure 12 In the embodiments of the present invention, the central spectral gradient vectors corresponding to 5 clusters in the search area of 27 hyperspectral images in the toy sequence

[0148] Step Eight: Calculate the loss function ε according to the following formula

[0149]

[0150] Among them, the loss function ε represents the sum of the squares of min(d j,k ), and the value range of j is from 1 to H s ×W s .

[0151] Specifically, the value of the loss function ε of the 27th frame hyperspectral image in the toy sequence after the first iteration of the present invention is 5.74.

[0152] Step Nine: Set the number of iterations q, and repeat Steps One to Seven until the set number of iterations q is reached or ε monotonically decreases within q times to obtain a minimum value, and then stop the iteration to obtain the clustering result after stopping the iteration;

[0153] Specifically, q in the present invention is set to 20. The value of the loss function ε of the 27th frame hyperspectral image in the toy sequence after the second iteration is 5.83, the value of the loss function ε of the 27th frame hyperspectral image in the toy sequence after the third iteration is 5.32, the value of the loss function ε of the 27th frame hyperspectral image in the toy sequence after the fourth iteration is 4.75, the value of the loss function ε of the 27th frame hyperspectral image in the toy sequence after the fifth iteration is 4.68, the value of the loss function ε of the 27th frame hyperspectral image in the toy sequence after the sixth iteration is 4.16, the value of the loss function ε of the 27th frame hyperspectral image in the toy sequence after the seventh iteration is 3.74, and the value of the loss function ε of the 27th frame hyperspectral image in the toy sequence after the eighth iteration is 3.97. It is known that the minimum value of ε appears in the seventh iteration. Therefore, the algorithm stops the iteration after the eighth iteration.

[0154] Step Ten: Generate a dynamic search mask S hard and the target mask T hard , and obtain a single-channel grayscale image after dimensionality reduction;

[0155] (1001) Based on the clustering result after stopping the iteration, define the pixels classified into C5 as target pixels, and the remaining pixels as background pixels. Set the grayscale value of the target pixels to 1 and the grayscale value of the background pixels to 0, and generate S according to the above definition hard .

[0156] Among them, the size of S hard is H s ×W s ×1.

[0157] Specifically, the size of S hard of the 27th frame hyperspectral image in the toy sequence is 83×113×1.

[0158] (1002) Take the centroid of the initial frame of S hard as the center, and crop the four outer edges to be cropped into T T with a size of H T ×W hard ×1. Among them, H T and WT respectively represent the height and width of the initial frame ground truth bounding box.

[0159] Specifically, in the present invention, H T is 32, W T is 33, and T hard has a size of 32×33×1.

[0160] (1003) Calculate the average of the spectral vectors of all target pixels to obtain the target spectrum C f .

[0161] Specifically, for the C of the search region of 27 hyperspectral images in the toy sequence f the gray values in the 1st band to the 16th band are successively {40, 36, 25, 31, 33, 37, 42, 47, 52, 64, 57, 63, 70, 71, 83, 97}. Figure 13 In the embodiment of the present invention, for the target spectrum C of the search region of 27 hyperspectral images in the toy sequence f .

[0162] (1004) Use C f and the spectral vector of each pixel on S t to perform a naive correlation operation, and use the result of the naive correlation as the 1-band gray image after dimensionality reduction.

[0163] Specifically, in the present invention, the specific calculation method of the naive correlation is to multiply and add the value of C f in each band respectively with the value of the spectral curve of each pixel in S 27 in the corresponding band, and use the result as the pixel value after dimensionality reduction of the pixel.

[0164] Step Eleven: Copy the single-channel gray image into a three-channel gray image and then send it into the siamese network for feature extraction. The siamese network includes two branches, namely the template branch and the search branch, and both branches use ResNet50 with the same parameters for feature extraction. represents the feature extracted from the template branch. represents the feature extracted from the search branch;

[0165] Specifically, the ResNet50 adopted in the present invention is divided into five parts, namely conv1, conv2_x, conv3_x, conv4_x, and conv5_x. The conv1 layer has 64 convolutional kernels with a size of 7×7, and the convolutional stride is 2. The conv2_x has a maximum pooling layer with a pooling kernel size of 3×3 and a stride of 2, as well as 3 residual blocks. The conv3_x has 4 residual blocks, the conv4_x has 6 residual blocks, and the conv5_x has 3 residual blocks. In addition, the downsampling operations in conv4_x and conv5_x of the ResNet50 adopted in the present invention are removed, and the ordinary convolutions in conv4_x and conv5_x are replaced with dilated convolutions. The stride of the dilated convolution is 1, the dilation rate in conv4_x is 2, and the dilation rate in conv5_x is 4.

[0166] Step Twelve: Obtain the concatenated features through temporal channel shifting operations and

[0167] (1201) Store the first 1 / 8 channels of the feature map extracted by the conv1 layer of ResNet50 in the t-1th frame through the feature memory;

[0168] (1202) Store the first 1 / 8 channels of the feature map extracted by the conv5-3 layer of ResNet50 in the t-1th frame through the feature memory;

[0169] (1203) After the feature extraction of ResNet50 in the tth frame is completed, replace the first 1 / 8 channels of the feature maps output by conv1 and conv5-3 with the cached feature maps, and the concatenated features are and

[0170] Specifically, the size of the feature map output by the conv1 layer of ResNet50 is 125×125×64, and the size of the feature map output by the conv5-3 layer of ResNet50 is 31×31×2048. In the present invention, the feature memory stores the first 8 channels of the feature map extracted by the conv1 layer of ResNet50 in the 26th frame and the first 256 channels of the feature map extracted by the conv5-3 layer of ResNet50 in the 26th frame. Replace the first 8 channels of the feature map extracted by the conv1 layer of ResNet50 in the 27th frame with the first 8 channels of the feature map extracted by the conv1 layer of ResNet50 in the 26th frame, and the concatenated features are Replace the first 256 channels of the feature map extracted by the conv5-3 layer of ResNet50 in the 27th frame with the first 256 channels of the feature map extracted by the conv5-3 layer of ResNet50 in the 26th frame, and the concatenated features are The size is 125×125×64, The size of this timing channel shift operation can capture the relative position information of the target, which is beneficial to improving the robustness of tracking.

[0171] Step thirteen: Use a 1×1 convolution to Adjust to And Make Adjust to And Where Represents the depth feature extracted from the template branch by the conv5-3 layer of ResNet50, Represents the depth feature extracted from the search branch by the conv5-3 layer of ResNet50, And Respectively represent the features for classification and regression of the search branch, And Respectively represent the features for classification and regression of the template branch;

[0172] Specifically, in the present invention, the convolution kernel size of the 1×1 convolution is 1×1×2048, and the number of convolution kernels is 256. And The size of is 31×31×256, And The size of is 7×7×256.

[0173] Step fourteen: And Use depth-level cross-correlation to generate classification feature P cls , And Use depth-level cross-correlation to generate regression feature P reg ;

[0174] Specifically, the size of P cls and P reg is 25×25×256.

[0175] Step fifteen: Through the classification version SPM of the dynamic spectrum sensing module cls Enhance P cls to the enhanced classification feature P cls-en , which is specifically implemented through the following steps:

[0176] (1501) Respectively perform channel dimension compression on P cls and through two 1×1 convolutions, and the compressed features are θ cls and

[0177] Among them, θ cls has a size of has a size of C represents P cls and the number of channels. In the present invention, C is 256;

[0178] Specifically, in the present invention, the convolution kernel size of the 1×1 convolution is 1×1×256, and the number of convolution kernels is 128. θ cls has a size of 128×25×25, has a size of 128×7×7.

[0179] (1502) θ cls is reshaped into is reshaped into

[0180] Among them, has a size of has a size of

[0181] Specifically, in the present invention has a size of 128×625, has a size of 128×49.

[0182] (1503) The similarity matrix V is obtained according to the following formula

[0183]

[0184] Among them, represents matrix multiplication, and the size of V is H1W1×H2W2;

[0185] Specifically, in the present invention, the size of V is 625×49.

[0186] (1504) V is normalized by the softmax function to obtain the normalized similarity matrix Among them, has a size of H1W1×H2W2;

[0187] Specifically, in the present invention has a size of 625×49.

[0188] (1505) The spatial dimension of T hard is reduced to H2×W2 by downsampling;

[0189] Specifically, in the present invention, the spatial dimension of T hard is reduced from 32×33 to 7×7.

[0190] (1506) Convert T with a size of H2×W2 into the target soft mask T through dilated convolution. hard where the size of T is H2×W2; soft .

[0191] Specifically, in the present invention, the size of T is 1×7×7. soft

[0192] Specifically, in the present invention, the size of T is 1×7×7. soft

[0193] (1507) Reshape T into soft where the size of is 1×H2W2;

[0194] where the size of is 1×H2W2;

[0195] Specifically, in the present invention, the size of is 1×49.

[0196] (1508) Obtain the spectral perception attention vector O of the classification feature according to the following formula

[0197]

[0198] where the size of O is H1W1×1;

[0199] Specifically, in the present invention, the size of O is 625×1.

[0200] (1509) After restoring the size of O to H1×W1, perform channel dimension restoration through 1×1 convolution to obtain the spectral perception attention vector O' with a size of C×H1×W1 for enhancing the classification feature.

[0201] Specifically, in the present invention, the size of O' is 256×25×25.

[0202] (1510) Add P cls to O' through matrix addition to obtain the enhanced classification feature P cls-en ;

[0203] Specifically, in the present invention, the size of P cls-en is 256×25×25.

[0204] Step Sixteen: Enhance P reg to the enhanced regression feature P reg through the regression version SPM of the dynamic spectral perception module, which is specifically implemented through the following steps: reg-en

[0205] (1601) Compress the channel dimension of P reg through 1×1 convolution, and the compressed feature is θreg .

[0206] Among them, θ reg has a size of C * ×H1×W1;

[0207] Specifically, in the present invention, the convolution kernel size of the 1×1 convolution is 1×1×256, the number of convolution kernels is 64, and the size of θ cls is 64×25×25.

[0208] (1602) θ reg is reshaped into reshaped into

[0209] Among them, has a size of C * ×H1W1, has a size of C * ×H3W3;

[0210] Specifically, has a size of 64×625, H3 = W3 = 125, has a size of 64×15625.

[0211] (1603) The similarity matrix Q is obtained according to the following formula

[0212]

[0213] Among them, the size of Q is H1W1×H3W3;

[0214] Specifically, the size of Q is 625×15625.

[0215] (1604) Q is normalized by the softmax function to obtain the normalized similarity matrix

[0216] Among them, has a size of H1W1×H3W3;

[0217] Specifically, has a size of 625×15625.

[0218] (1605) The spatial dimension of S hard is expanded to H3×W3 through upsampling;

[0219] Specifically, in the present invention, the spatial dimension of S hard is expanded from 83×113 to 125×125.

[0220] (1606) S is subjected to dilated convolutionhard Convert to the dynamic search soft mask S soft .

[0221] Among them, S soft has a size of 1×H3×W3;

[0222] Specifically, in the present invention, S soft has a size of 1×125×125. During the tracking process, since the target is changing, the search area is changing, and S soft generated based on the search area is also changing. Then, taking S soft as one of the inputs, the SPM reg can dynamically perceive the state of the target

[0223] (1607)S soft is reshaped into

[0224] Among them, has a size of 1×H3W3;

[0225] Specifically, in the present invention has a size of 1×15625.

[0226] (1608) Obtain the spectral perception attention vector Y of the regression feature according to the following formula

[0227]

[0228] Among them, the size of Y is H1W1×1;

[0229] Specifically, in the present invention, the size of Y is 625×1.

[0230] (1609) After restoring the size of Y to H1×W1, then complete the reduction of the channel dimension through a 1×1 convolution to obtain the spectral perception attention matrix Y′ with a size of C×H1×W1 for enhancing the regression feature.

[0231] Specifically, in the present invention, the size of Y′ is 25×25×256.

[0232] (1610) Add P reg and Y′ through matrix addition to obtain the enhanced regression feature P reg-en ;

[0233] Specifically, in the present invention, P reg-en has a size of 25×25×256.

[0234] Step Seventeen, P cls-en passes through the classification branch to output the classification score map M cls , P reg-enOutput the regression score map M through the regression branch reg and the target quality estimation score map M tar ;

[0235] Specifically, the classification branch includes 1 1×1 convolutional layer and 4 3×3 convolutional layers. The regression branch includes 1 1×1 convolutional layer and 4 3×3 convolutional layers. The size of M cls is 25×25×2, and the size of M reg is 25×25×4, and the size of M tar is 25×25×1.

[0236] Step Eighteen, M tar and M cls are weighted and fused into the final score map, and the position of the target box is determined according to the final score map.

[0237] Specifically, in the present invention, the weight of M tar is 0.6, and the weight of M cls is 0.4.

[0238] The present invention proposes a hyperspectral video object tracking method based on a dynamic spectral perception module. First, according to the difference in gradients of the spectral vectors of target pixels and background pixels, the present invention proposes a dimensionality reduction method based on spectral gradient vector clustering, which can compress the data of hyperspectral images and complete the binary classification of target pixels and background pixels on hyperspectral images. A binary mask containing target spectral information and appearance cues is generated based on the binary classification result. In addition, the present invention adds a feature memory to store partial channels of historical frame features, and captures the trajectory cues of the target by means of channel shift operations in time series. Finally, the present invention uses a non-local attention mechanism to complete the integration of the target's trajectory cues, appearance cues, and spectral information. Two spectral perception modules designed based on this idea are used to enhance the target expression of corresponding features.

[0239] The above is only a preferred embodiment of the present invention, and is not used to limit the protection scope of the present invention.

Claims

1. A hyperspectral video target tracking method based on a dynamic spectral sensing module, characterized in that, The method is as follows: Step 1: Read the t-th hyperspectral image in the hyperspectral image sequence and determine the search region S of the t-th hyperspectral image t ; where S t represents a three-dimensional matrix of size B×H s ×W s , t represents the serial number of the hyperspectral image frame in the hyperspectral image sequence, t is an integer greater than or equal to 1, B represents the number of bands of S t , H s represents the number of rows of pixels of each band image on S t , and W s represents the number of columns of pixels of each band image on S t ; Step 2: Determine S through matrix subtraction t of the hyperspectral gradient map G t , and determine the spectral gradient vector of the j-th pixel in G t ​ Step 3. For S t Establish a coordinate system in the spatial dimension; Step 4. Determine the spectral gradient vector G t (k) of the k-th clustering center; t (k); Step Five: Determine the spectral angular distance d t between G j,k (k), and obtain the spectral angular distance set d j ; Step 6. Determine the label b(j) of the cluster center to which the j-th pixel belongs through the smallest spectral angle distance in d j ; Step 7: Determine the cluster center to which the j-th pixel belongs through b(j), and determine the number of pixels N included in the k-th cluster center C k The number of pixels N k , for C k Average the spectral gradient vectors of all pixels within to obtain the central spectral gradient vector of C k The central spectral gradient vector Step 8: Set and calculate the loss function ε; Step 9: Set the number of iterations q, and repeat Step 1 to Step 8 until the set number of iterations q is reached or ε monotonically decreases within q times to obtain a minimum value, then stop the iteration to obtain the clustering result after stopping the iteration; Step Ten: Generate a dynamic search mask S using the clustering results after stopping iteration hard and the target mask T hard , and obtain a single-channel grayscale image after dimensionality reduction; Step Eleven: After copying the single-channel grayscale image into a three-channel grayscale image, send it into the Siamese network for feature extraction. The Siamese network includes two branches, namely the template branch and the search branch. Both branches use ResNet50 with the same parameters for feature extraction. represents the features extracted from the template branch. represents the features extracted from the search branch. Step Twelve: Obtain the stitched features through the time-sequence channel shift operation and Step 13: Use 1×1 convolution to adjust the depth features extracted from the conv5-3 layer of ResNet50 in the template branch to and adjust the depth features extracted from the conv5-3 layer of ResNet50 in the search branch to and where and represent the features for classification and regression in the search branch respectively, and represent the features for classification and regression in the template branch respectively; Step Fourteen: and generate classification feature P using deep-level cross-correlation cls , and generate regression feature P using deep-level cross-correlation reg ; Step 15: Through the classification version SPM of the dynamic spectrum perception module cls and the regression version SPM of the dynamic spectrum perception module reg Enhance the features of P cls and P reg respectively to become the enhanced classification feature P cls-en and the enhanced regression feature P reg-en ; Step Sixteen, P cls-en Output the classification score map M through the classification branch cls , P reg-en Output the regression score map M through the regression branch reg and the target quality estimation score map M tar ; Step Seventeen: M tar and M reg are weighted and fused into the final score map, and the location of the target is determined according to the final score map.

2. The hyperspectral video target tracking method based on a dynamic spectral sensing module according to claim 1, characterized in that The specific implementation of Step 2 is as follows: For S t Perform matrix subtraction on adjacent band images to determine the hyperspectral gradient map G of the t-th frame t The spectral gradient vector of the j-th pixel in is Among them, j represents the index of pixels in the spatial dimension, and the value range of j is from 1 to H s ×W s , denotes the spectral gradient value at the i-th band, and are respectively the gray values of the j-th pixel in the (i + 1)-th band and the i-th band in S t . B represents the number of bands of the hyperspectral image, and i represents the band index 3. The hyperspectral video object tracking method based on a dynamic spectral sensing module according to claim 1 or 2, characterized in that The specific implementation of Step 5 is as follows: Calculation with G t (k) to obtain the spectral angular distance d j,k and obtain d j as Among them, G t (k) represents the spectral gradient vector corresponding to the k-th cluster center, k represents the index of the cluster center, the value range of k is from 1 to K, and K represents the number of cluster centers, d j represents S t the set of spectral angular distance between the j-th pixel on and the K cluster centers, ||·|| represents the vector modulus operation, (·) T represents the vector transpose.

4. The hyperspectral video target tracking method based on a dynamic spectral sensing module according to claim 3, wherein The specific implementation of Step 6 is as follows: According to d j Among them, the smallest spectral angular distance min(d j,k ) determines that the label b(j) of the cluster center to which the j-th pixel belongs is where b(j) is an integer, the value range of b(j) is from 1 to K, min(·) represents the minimum value operation, and the value range of j is from 1 to H s ×W s 。 5. The hyperspectral video target tracking method based on a dynamic spectral sensing module according to claim 4, wherein The specific implementation of Step 7 is as follows: (701) Traverse each pixel in S t and determine whether b(j) is equal to k; if they are equal, the j-th pixel corresponding to b(j) is assigned to the k-th cluster C k and set N(j) k to 1; otherwise, the pixel does not belong to C k , N(j) k is set to 0: where N(j) k represents the quantity feature of each pixel on S t ​ (702) Accumulate C k N(j) in k to obtain C k the number of pixels N included in k , where the value range of k is from 1 to 5; (703) Average the spectral gradient vectors of all pixels in C to obtain the central spectral gradient vector of C k k ​​ Among them, ∑ represents the accumulation operation, and the value range of k is from 1 to K.

6. The hyperspectral video target tracking method based on a dynamic spectral sensing module according to claim 5, wherein The specific implementation of Step 8 is as follows: Define the loss function ε as shown in the following formula where the loss function ε represents the sum of squares of min(d j,k ), and the value range of j is from 1 to H s ×W s .

7. The hyperspectral video target tracking method based on a dynamic spectral sensing module according to claim 6, wherein The specific implementation of Step 10 is as follows: (1001) Based on the clustering result after stopping iteration, define the pixels classified into the target cluster as target pixels, and the remaining pixels as background pixels. Set the gray value of the target pixels to 1 and the gray value of the background pixels to 0, and generate a dynamic search mask S according to the above definition. hard ; Among them, S hard has a size of H s ×W s ×1; (1002) Crop the S of the initial frame hard Centered on the centroid, crop the four outer edges into a target mask T with dimensions H T ×W T ×1 hard ; Among them, H T and W T represent the height and width of the ground truth bounding box of the initial frame, respectively; (1003) Average the spectral vectors of all target pixels to obtain the target spectrum C f ; (1004) Using C f and S t perform a naive correlation operation on the spectral vectors of each pixel on it, and use the result of the naive correlation as the 1-band grayscale image after dimensionality reduction.

8. The hyperspectral video target tracking method based on a dynamic spectral sensing module according to claim 7, characterized in that, The specific implementation of Step 12 is as follows: (1201) Store the first 1 / 8 channels of the feature map extracted by the conv1 layer of ResNet50 in the t-1th frame through the feature memory; (1202) Store the first 1 / 8 channels of the feature map extracted by the conv5-3 layer of ResNet50 in the t-1th frame through the feature memory; After the feature extraction of ResNet50 is completed at the t-th frame, the first 1 / 8 channels of the feature maps output by conv1 and conv5-3 are replaced with the cached feature maps, and the concatenated features are and 9. The hyperspectral video target tracking method based on a dynamic spectral sensing module according to claim 8, wherein The SPM in Step 15 cls Enhance P cls to P cls-en , which is specifically implemented through the following steps: (1501) Compress P through two 1×1 convolutions respectively cls and along the channel dimension. The compressed features are θ cls and Among them, θ cls has a size of has a size of C represents P cls and the number of channels. In the present invention, C is 256; (1502)θ cls reshaped into reshaped into Among them, has a size of has a size of (1503) Obtain the similarity matrix V according to the following formula Among them, represents matrix multiplication, and the size of V is H1W1×H2W2; (1504) Normalize V through the softmax function to obtain the normalized similarity matrix where has the size of H1W1×H2W2; (1505) Downsample T to reduce its spatial dimension to H2×W2; hard ​ (1506) Convert T with size H2×W2 into the target soft mask T through dilated convolution hard soft ;​ Among them, T soft has dimensions of H2 × W2; (1507)T soft reshaped into Among them, has a size of 1×H2W2; (1508) Obtain the spectral perception attention vector O of the classification feature according to the following formula Among them, the size of O is H1W1×1; (1509) After restoring the size of O to H1×W1, then complete the reduction of the channel dimension through a 1×1 convolution to obtain the spectral perception attention vector O′ of size C×H1×W1 for enhancing the classification feature; (1510) Add P through matrix addition cls to O' to obtain the enhanced classification feature P cls-en .

10. The hyperspectral video target tracking method based on a dynamic spectral sensing module according to claim 9, further characterized in that The SPM in Step 15 reg Enhance P reg to P reg-en , which is specifically implemented through the following steps: (1601) Compress P through 1×1 convolution, and the compressed feature is θ reg reg ;​ where θ reg has dimensions C * ×H1×W1, (1602)θ reg reshaped into reshaped into Among them, has a size of C * ×H1W1, has a size of C * ×H3W3; (1603) Obtain the similarity matrix Q according to the following formula Among them, the size of Q is H1W1×H3W3; (1604) Normalize Q through the softmax function to obtain the normalized similarity matrix Among them, has dimensions of H1W1×H3W3; (1605) Expand the spatial dimension of S to H3×W3 through upsampling hard ; (1606) Convert S through dilated convolution hard to the dynamic search soft mask S soft ; Among them, S soft has a size of 1×H3×W3; (1607)S soft reshaped into Among them, has a size of 1×H3W3; (1608) Obtain the spectral perception attention vector Y of the regression feature according to the following formula Among them, the size of Y is H1W1×1; (1609) After restoring the size of Y to H1×W1, then complete the reduction of the channel dimension through a 1×1 convolution to obtain the spectral perception attention matrix Y′ of size C×H1×W1 for enhancing the regression feature; (1610) Add P by matrix addition reg to Y' to obtain the enhanced regression feature P reg-en .

Images