Hyperspectral video tracking method based on spectral mapping enhancement

Abstract

The invention provides a hyperspectral video tracking method based on spectral mapping enhancement, and the method comprises the steps: obtaining a spectral curve of a local region of a tth-frame hyperspectral image, carrying out the spectral angular distance operation of the spectral curve and a spectral curve of each pixel position of a search region, segmenting the image into a target region and a background region, and carrying out the spectral angular distance operation; the method comprises the steps of obtaining flatness coefficients and a flatness coefficient average value of pixels in a target area, judging whether pixel points are located at the junction of a target and a background, replacing the flatness coefficients not located at the junction of the target and the background with the flatness coefficient average value in the target pixels, and calculating enhancement coefficients corresponding to the pixels to complete final dimension reduction. And extracting depth features of the target area, the search area and the template image to carry out cross-correlation operation, respectively obtaining multi-dimensional response diagrams, carrying out weighted fusion to obtain a final response diagram, carrying out convolution on the final response diagram to obtain classification features and regression features, and determining a target position.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a hyperspectral video tracking method based on spectral mapping enhancement. Background Art

[0002] Visual object tracking is a basic task in the field of computer vision. Given the position of the target in the first frame, the state of the tracked target in the subsequent frames is predicted and the position bounding box is given. It has a wide range of practical applications in real life, such as autonomous driving, robotics, human-computer interaction and other fields. Although great progress has been made in recent years, trackers are prone to tracking drift due to scale changes, fast movement, lighting changes, etc. How to design a stable and accurate tracker remains a difficult problem.

[0003] The typical filter-based target tracking algorithm is the context-aware correlation filter tracking algorithm. In the field of target tracking, the context-aware correlation filter tracking algorithm shows a certain flexibility. It successfully solves the challenges of many traditional tracking algorithms in complex scenes. The core idea of ​​this algorithm is to integrate the spatial context information of the target area to make the tracking system more robust and accurate. However, this algorithm has the problem of over-reliance on the spatial characteristics of the target and lack of effective use of spectral information, resulting in inaccurate tracking.

[0004] In the field of target tracking, the target tracking algorithm based on the dual-branch fully convolutional twin network represents the latest progress in deep learning in solving the target tracking problem. The design of this algorithm is inspired by the twin network structure, which contains two similar networks, one for processing the target area image and the other for processing the search area image. The advantage of this design is that it can capture the spatial and contextual information of the target and improve the accuracy and robustness of the target in complex scenes. The algorithm adopts an end-to-end learning method and makes full use of the advantages of deep learning. The structure is usually lightweight. Coupled with the development of deep learning hardware, the algorithm performs well in real-time target tracking. However, the algorithm has the problem that the extracted deep features are not rich enough and the position of the target centroid in the corresponding response map is not accurate enough, which leads to the decline of tracker performance. Summary of the invention

[0005] In view of this, the main object of the present invention is to provide a hyperspectral video tracking method based on spectral mapping enhancement.

[0006] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0007] The embodiment of the present invention provides a hyperspectral video tracking method based on spectral mapping enhancement, the method comprising:

[0008] Step 1: Load the target position, target frame and target image block of the first frame of the hyperspectral image in the grayscale normalized hyperspectral image sequence, then load the tth frame of the hyperspectral image in the grayscale normalized hyperspectral image sequence, and determine the search area S of the tth frame of the hyperspectral image in the grayscale normalized hyperspectral image sequence. t , where the grayscale range of the pixel of the t-th frame of the hyperspectral image in the hyperspectral image sequence after grayscale normalization is a left-closed and right-closed interval from 0 to 1, t represents the number of frames of the hyperspectral image, and t is an integer greater than or equal to 2 and less than or equal to 1000;

[0009] Step 2: Taking the target position of the t-1th frame of the hyperspectral image in the grayscale normalized hyperspectral image sequence as the center, t A 3×3 pixel image block is intercepted as the local area L of the t-th frame hyperspectral image after grayscale normalization. t , determine L t The i-th band image The spectral average C li , according to C li Determine L t The local spectral curve C l ;

[0010] Step 3: Use C l For S t The spectral curve C corresponding to all pixel positions m,n Perform the spectral angle distance operation to obtain the initial dimension reduction result N of the t-th frame hyperspectral image in the hyperspectral image sequence after grayscale normalization. m,n , where m represents S t The horizontal coordinate of the pixel in S t The vertical coordinate of the pixel in

[0011] Step 4: According to the N m,n The comparison relationship with the error threshold Th obtains the spectral mask M m,n ;

[0012] Step 5: According to the M m,n and N m,n Determine the pixel position flatness coefficient σ m,n ;

[0013] Step 6: According to the σ m,n The comparison relationship with the threshold β determines whether the pixel belongs to the boundary position between the target and the background, and obtains the target and background boundary matrix Q m,n ;

[0014] Step 7: According to the Q mn and σ mn Determine the average value of the target internal flatness coefficient σ mean ;

[0015] Step 8: σ at the boundary between the target and the background m,n Replace with σ mean Get the updated pixel position flatness coefficient

[0016] Step nine, according to the The spectral mapping enhancement coefficient a is determined by the constant coefficient k;

[0017] Step 10: According to a, N m,n and M m,n Determine the final dimension reduction result E m,n , the dimensionality reduction result of the target image block of the first frame of the hyperspectral image in the hyperspectral image sequence after grayscale normalization is recorded as the target image X, and the search area S of the tth frame of the hyperspectral image in the hyperspectral image sequence after grayscale normalization is recorded as t The dimension reduction result is recorded as the search image Z, and the dimension reduction result of the target image block of the hyperspectral image of the t-1th frame in the hyperspectral image sequence after grayscale normalization is recorded as the template image P;

[0018] Step 11: According to the following formula, use the Resnet50 network to extract features of X, Z and P respectively, and concatenate the features of the third, fourth and fifth layers to obtain the concatenated target image features: Search image features after stitching Template image features after splicing Specifically include: Among them, Cat(·) represents the concatenation operation, F3(X) represents the features extracted from X using the third layer of Resnet50, F4(X) represents the features extracted from X using the fourth layer of Resnet50, F5(X) represents the features extracted from X using the fifth layer of Resnet50, F3(Z) represents the features extracted from Z using the third layer of Resnet50, F4(Z) represents the features extracted from Z using the fourth layer of Resnet50, F5(Z) represents the features extracted from Z using the fifth layer of Resnet50, F3(P) represents the features extracted from P using the third layer of Resnet50, F4(P) represents the features extracted from P using the fourth layer of Resnet50, and F5(P) represents the features extracted from P using the fifth layer of Resnet50;

[0019] Step 12: Respectively and Perform cross-correlation operation and obtain and Multidimensional response map R1 after cross-correlation operation, and The multidimensional response map R2 after the cross-correlation operation specifically includes: Among them, ⊙ represents the cross-correlation operation;

[0020] Step 13: According to the following formula, R1 and R2 are average-weighted to obtain the final multi-dimensional response graph R, specifically including: R=0.5*R1+0.5*R2;

[0021] Step 14: According to the following formula, use 1×1 classification convolution kernel K cls and a 1×1 regression convolution kernel K reg Perform convolution operation on R to obtain classification feature R cls With regression feature R reg , R cls The maximum value is taken as the centroid position of the target, R reg The distance from the center of the target to the four boundaries of the target frame is calculated using R cls and R reg You can draw the target frame to determine the location of the target, including: in, Represents the convolution operation;

[0022] Step 15: Load each frame of the hyperspectral image in the hyperspectral image sequence in turn, and repeat steps 1 to 14 to complete the target tracking of the hyperspectral image sequence.

[0023] In the above scheme, according to C li Determine L t The local spectral curve C l , including: Determine C l ; Where A(·) represents the averaging operation, i represents the serial number of the band, i is an integer greater than or equal to 1 and less than or equal to B, and B represents the total number of bands.

[0024] In the above scheme, the use of C l For S t The spectral curve C corresponding to all pixel positions m,n Perform the spectral angle distance operation to obtain the initial dimension reduction result N of the t-th frame hyperspectral image in the hyperspectral image sequence after grayscale normalization. m,n , including: Determine N m,n ; Where acrcos(·) represents the arccosine operation, and T represents the transpose operation.

[0025] In the above scheme, according to the N m,n The comparison relationship with the error threshold Th obtains the spectral mask M m,n , including: Determine the spectral mask M m,n , where N m,n When it is greater than Th, it is the background area, and the corresponding position value of the spectrum mask is 0. m,n When it is less than or equal to Th, it is the target area, the corresponding position value of the spectral mask is 1, and Th is set to 0.22.

[0026] In the above scheme, the M m,n and N m,n Determine the pixel position flatness coefficient σ m,n , including: Determine σ m,n , where N k,l It means traversing the eight neighborhood values ​​of the target pixel. The value range of k is [m-1, m+1] and it is an integer. The value range of l is [n-1, n+1] and it is an integer. ∑(·) means the sum operation, |·| means the absolute value operation, and * means the multiplication of corresponding pixels at the same position.

[0027] In the above scheme, according to the σ m,n The comparison relationship with the threshold β determines whether the pixel belongs to the boundary position between the target and the background, and obtains the target and background boundary matrix Q m,n , including: Determine whether the pixel belongs to the boundary between the target and the background, and obtain the boundary matrix Q between the target and the background m,n , where if σ m,n If σ is less than or equal to β, the pixel does not belong to the boundary between the target and the background. m,n If it is greater than β, the pixel belongs to the boundary between the target and the background.

[0028] In the above scheme, according to the Q m,n and σ m,n Determine the average value of the target internal flatness coefficient σ mean , including: Determine the average value of the target internal flatness coefficient σ mean , p represents the search area S t width, q represents the search area S t of high.

[0029] In the above scheme, the σ at the boundary between the target and the background is m,n Replace with σ mean Get the updated pixel position flatness coefficient Specifically include: Replace the flatness coefficient at the boundary between the target and the background with σ mean , get the updated pixel position flatness coefficient

[0030] In the above scheme, according to the The spectral mapping enhancement coefficient a is determined by the constant coefficient k, specifically including: The spectral mapping enhancement coefficient a is determined, where the constant coefficient k is set to 0.7.

[0031] In the above scheme, according to a, N m,n and M m,n Determine the final dimension reduction result E m,n , the dimensionality reduction result of the target image block of the first frame of the hyperspectral image in the hyperspectral image sequence after grayscale normalization is recorded as the target image X, and the search area S of the tth frame of the hyperspectral image in the hyperspectral image sequence after grayscale normalization is recorded as t The dimension reduction result is recorded as the search image Z, and the dimension reduction result of the target image block of the hyperspectral image of the t-1th frame in the hyperspectral image sequence after grayscale normalization is recorded as the template image P. Specifically, it includes: according to E m,n =N m,n -N m,n *a*M m,n Determine the final dimension reduction result E m,n .

[0032] Compared with the prior art, the hyperspectral video tracking method based on spectral mapping enhancement of the present invention has significant advantages. First, by introducing hyperspectral information, the present invention effectively overcomes the problem that the filter-based target tracking algorithm is overly dependent on the target's spatial characteristics and lacks effective use of spectral information. Through spectral angular distance operations and spectral mapping enhancement technology, this method can retain the spectral characteristics of the target during the dimensionality reduction process, enhance the distinction between the target and the background, and thus improve the tracking accuracy. In addition, compared with the existing two-branch fully convolutional twin network, a new deep feature extraction branch of the template image is introduced, and a multidimensional response map generated by the template image features and the search image features is obtained, which further optimizes the accuracy of the target position. Compared with the existing method, the present invention effectively overcomes the problem that the extracted deep features of the two-branch fully convolutional twin network are not rich enough and the target centroid position in the corresponding response map is not accurate enough. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings described herein are used to disclose a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0034] Figure 1 is a flow chart of the present invention;

[0035] Figure 2 This is the 38th frame of hyperspectral image after grayscale normalization in the embodiment of the present invention;

[0036] Figure 3 is the search area S of the 38th frame of the hyperspectral image after grayscale normalization in the embodiment of the present invention. 38 ;

[0037] Figure 4 is the target local area L of the 38th frame of the hyperspectral image after grayscale normalization in the embodiment of the present invention. 38 The local spectral curve C l ;

[0038] Figure 5 is the search area S of the 38th frame of the hyperspectral image after grayscale normalization in the embodiment of the present invention. 38 A band grayscale image after dimensionality reduction;

[0039] Figure 6 This is a graph of target tracking results of the 38th frame of hyperspectral image after grayscale normalization in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0041] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, article or device. In the absence of more restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, article or device including the element.

[0042] The embodiment of the present invention provides a hyperspectral video tracking method based on spectral mapping enhancement, such as Figure 1 As shown, the method is:

[0043] Step 1: Load the target position, target frame and target image block of the first frame of the hyperspectral image in the grayscale normalized hyperspectral image sequence, then load the tth frame of the hyperspectral image in the grayscale normalized hyperspectral image sequence, and determine the search area S of the tth frame of the hyperspectral image in the grayscale normalized hyperspectral image sequence. t , where the grayscale range of the pixel of the t-th frame of the hyperspectral image in the hyperspectral image sequence after grayscale normalization is a left-closed and right-closed interval from 0 to 1, t represents the number of frames of the hyperspectral image after grayscale normalization, and t is an integer greater than or equal to 2 and less than or equal to 1000;

[0044] For example, the hyperspectral dataset IMEC16 is downloaded from the website (www.hsitracking.com), the hyperspectral image sequence of a coke can is selected, the 38th hyperspectral image in the hyperspectral image sequence of the coke can is loaded, and the grayscale of the 38th hyperspectral image is normalized to obtain the grayscale normalized 38th hyperspectral image, and the search area S of the grayscale normalized 38th hyperspectral image is determined. 38 , S 38 Using the 37th frame target image block, S 38 The size of S is 1.5 times the size of the target image block in the 37th frame. 38 There are 16 bands, namely B is 16, S 38 The image resolution is 118 × 184 pixels, that is, S 38 The width p is 118, the height q is 184, and there are 331 frames in the coke can hyperspectral image sequence. The resolution of each hyperspectral image in the coke can hyperspectral image sequence is 256×512 pixels. Figure 2 It is the 38th frame of hyperspectral image after grayscale normalization in the embodiment of the present invention. Figure 3 is the search area S of the 38th frame of the hyperspectral image after grayscale normalization in the embodiment of the present invention. 38 .

[0045] Step 2: Taking the target position of the t-1th frame of the hyperspectral image in the grayscale normalized hyperspectral image sequence as the center, t A 3×3 pixel image block is intercepted as the local area L of the t-th frame hyperspectral image after grayscale normalization. t , determine L t The i-th band image The spectral average C li , according to C li Determine L t The local spectral curve C l , including: Determine C l ; Where A(·) represents the averaging operation, i represents the serial number of the band, i is an integer greater than or equal to 1 and less than or equal to B, and B represents the total number of bands;

[0046] For example, the target position of the 37th hyperspectral image in the grayscale normalized hyperspectral image sequence of the cola can is taken as the center, and the center coordinates of the target position of the 37th hyperspectral image in the grayscale normalized hyperspectral image sequence are (211, 97). 37 A 3×3 pixel target image block is intercepted at the target center as the local area L of the 37th frame hyperspectral image after grayscale normalization. 37 , calculate L 37 The 4th band image The average value of the spectrum is 0.3472. 37 The spectral average of the 16 band images is used to determine the target local area L of the 37th frame of the hyperspectral image after grayscale normalization. 37 The local spectral curve C l , and obtain C l The spectral average values ​​in the 1st to 16th bands are {0.2121, 0.3462, 0.3732, 0.3472, 0.4351, 0.4418, 0.4765, 0.5834, 0.5242, 0.7527, 0.6437, 0.6823, 0.5834, 0.5947, 0.6825, 0.7834} respectively. Figure 4 is the target local area L of the 37th frame hyperspectral image after grayscale normalization 37 The local spectral curve C l , which is a line graph composed of 16 spectral averages depicted in band order.

[0047] Step 3: Use C l For S t The spectral curve C corresponding to all pixel positions m,n Perform the spectral angle distance operation to obtain the initial dimension reduction result N of the t-th frame hyperspectral image in the hyperspectral image sequence after grayscale normalization. m,n , where m represents S t The horizontal coordinate of the pixel in S t The vertical coordinate of the pixel in , specifically including: Determine N m,n , where acrcos(·) represents the arccosine operation and T represents the transpose operation;

[0048] For example, for the 38th hyperspectral image in the hyperspectral image sequence of a cola can after grayscale normalization, the value range of m is [0, 117], the value range of n is [0, 183], and S 38 The third row and fourth column are the results of the initial dimensionality reduction N 3,4 It is 0.5079.

[0049] Step 4: According to the N m,n The comparison relationship with the error threshold Th obtains the spectral mask M m,n , including: Determine the spectral mask M m,n , where N m,n When it is greater than Th, it is the background area, and the corresponding position value of the spectrum mask is 0. m,n When it is less than or equal to Th, it is the target area, the corresponding position value of its spectral mask is 1, and Th is set to 0.22;

[0050] For example, the search area S of the 38th hyperspectral image in the hyperspectral image sequence of the cola can after grayscale normalization is 38 The result after the initial dimensionality reduction is N m,n The third row and fourth column N 3，4 The value of is 0.5079, which is greater than 0.22, so the corresponding position spectral mask M 3，4 The value is 0.

[0051] Step 5: According to the M m,n and N m,n Determine the pixel position flatness coefficient σ m,n , including: Determine σ m,n , where N k,l It means traversing the eight neighborhood values ​​of the target pixel. The value range of k is [m-1, m+1] and it is an integer. The value range of l is [n-1, n+1] and it is an integer. ∑(·) means the sum operation, |·| means the absolute value operation, and * means the multiplication of corresponding pixels at the same position.

[0052] For example, in the hyperspectral image sequence of the cola can after grayscale normalization, in the 38th frame of the hyperspectral image, N k,l Indicates traversing the eight neighborhood values ​​of the target pixel. The value range of k is [210, 212], the value range of l is [96, 98], and S 38 The flatness coefficient σ corresponding to the pixel position at row 68 and column 74 68,74 It is 0.1732, and each pixel position is traversed to obtain its corresponding flatness coefficient.

[0053] Step 6: According to the σ m,n The comparison relationship with the threshold β determines whether the pixel belongs to the boundary position between the target and the background, and obtains the target and background boundary matrix Q m,n , including: Determine whether the pixel belongs to the boundary between the target and the background, and obtain the boundary matrix Q between the target and the background m,n , where if σ m,n If σ is less than or equal to β, the pixel does not belong to the boundary between the target and the background. m,n If it is greater than β, the pixel belongs to the boundary between the target and the background;

[0054] For example, the β value is 0.1, S 38 The flatness coefficient σ corresponding to the pixel position in row 68 and column 74 68,74 is 0.1732, greater than 0.1, Q 68,74 =0, the pixel position belongs to the boundary between the target and the background.

[0055] Step 7: According to the Qm,n and σ m,n Determine the average value of the target internal flatness coefficient σ mean , including: Determine the average value of the target internal flatness coefficient σ mean , p represents the search area S t width, q represents the search area S t of high;

[0056] For example, the grayscale normalized hyperspectral image of the 38th frame in the hyperspectral image sequence of the cola can is σ mean is 0.07.

[0057] Step 8: σ at the boundary between the target and the background m,n Replace with σ mean Get the updated pixel position flatness coefficient Specifically include: Replace the flatness coefficient at the boundary between the target and the background with σ mean , get the updated pixel position flatness coefficient

[0058] For example, after grayscale normalization, the 38th frame of the hyperspectral image in the hyperspectral image sequence of the cola can is S 38 The Q corresponding to the pixel position in row 68 and column 74 68,74 =0, so σ 68,74 Replace with σ mean , updated pixel position flatness coefficient

[0059] Step nine, according to the The spectral mapping enhancement coefficient a is determined by the constant coefficient k, specifically including: Determine the spectral mapping enhancement coefficient a, where the constant coefficient k is set to 0.7;

[0060] For example, k is set to 0.7, S 38 The pixel position at row 68 and column 74 corresponds to the updated pixel position flatness coefficient The corresponding spectral mapping enhancement coefficient a is 0.651.

[0061] Step 10: According to a, N m,n and M m,n Determine the final dimension reduction result E m,n , the dimensionality reduction result of the target image block of the first frame of the hyperspectral image in the hyperspectral image sequence after grayscale normalization is recorded as the target image X, and the search area S of the tth frame of the hyperspectral image in the hyperspectral image sequence after grayscale normalization is recorded as tThe dimension reduction result is recorded as the search image Z, and the dimension reduction result of the target image block of the hyperspectral image of the t-1th frame in the hyperspectral image sequence after grayscale normalization is recorded as the template image P. Specifically, the following steps are performed according to E m,n =N m,n -N m,n *a*M m,n Determine the final dimension reduction result E m,n ;

[0062] For example, S 38 The pixel position a at row 68 and column 74 is 0.651, N 68,74 The value of M is 0.5079. 68,74 The value of is 1, and the corresponding pixel value changes from 0.2352 to 0.15237. Figure 5 In the example of the present invention, the search area S of the 38th frame of the hyperspectral image after grayscale normalization is 38 A one-band grayscale image after dimensionality reduction.

[0063] Step 11: According to the following formula, use the Resnet50 network to extract features of X, Z and P respectively, and concatenate the features of the third, fourth and fifth layers to obtain the concatenated target image features respectively: Search image features after stitching Template image features after splicing Specifically include: Among them, Cat(·) represents the concatenation operation, F3(X) represents the features extracted from X using the third layer of Resnet50, F4(X) represents the features extracted from X using the fourth layer of Resnet50, F5(X) represents the features extracted from X using the fifth layer of Resnet50, F3(Z) represents the features extracted from Z using the third layer of Resnet50, F4(Z) represents the features extracted from Z using the fourth layer of Resnet50, F5(Z) represents the features extracted from Z using the fifth layer of Resnet50, F3(P) represents the features extracted from P using the third layer of Resnet50, F4(P) represents the features extracted from P using the fourth layer of Resnet50, and F5(P) represents the features extracted from P using the fifth layer of Resnet50;

[0064] For example, the third layer of Resnet50 is used to 38 The extracted features have 256 channels and a size of 107×107. The fourth layer of Resnet50 is used to extract S 38 The extracted features have 512 channels and a size of 53×53. The fifth layer of Resnet50 is used to extract S 38 The extracted features have 1024 channels and a size of 26×26.

[0065] Step 12, Respectively and Perform cross-correlation operation and obtain

[0066] and Multidimensional response map R1 after cross-correlation operation, and The multidimensional response map R2 after the cross-correlation operation specifically includes: Among them, ⊙ represents the cross-correlation operation;

[0067] For example, the target image feature after splicing the first frame of the hyperspectral image sequence of the coke can after the grayscale normalization is And the spliced ​​S 38 Searching for image features The multidimensional response map R1 obtained after the cross-correlation operation has 2048 channels and a size of 10×10. The template image features after the splicing of the 37th frame of the hyperspectral image sequence of the coke can after the grayscale normalization are And the spliced ​​S 38 Searching for image features The multidimensional response map R2 obtained after the cross-correlation operation has 2048 channels and a size of 10×10.

[0068] Step 13: According to the following formula, R1 and R2 are average-weighted to obtain the final multi-dimensional response graph R, specifically including: R=0.5*R1+0.5*R2;

[0069] For example, the target image features of the first frame of the hyperspectral image sequence of the coke can after grayscale normalization are: and S 38 Searching for image features The multidimensional response map R1 is obtained by the grayscale normalized template image features of the 37th frame of the Coke can hyperspectral image sequence and S 38 Searching for image features The obtained multidimensional response map R2 is weighted averagely processed by R1 and R2 to obtain the final multidimensional response map R, where the number of channels of R is 2048 and the size is 10×10.

[0070] Step 14: According to the following formula, use 1×1 classification convolution kernel K cls and a 1×1 regression convolution kernel K reg Perform convolution operation on R to obtain classification feature R cls With regression feature R reg , R cls The maximum value is taken as the centroid position of the target, R regThe distance from the center of the target to the four boundaries of the target frame is calculated using R cls and R reg You can draw the target frame to determine the location of the target, including: in, Represents the convolution operation;

[0071] For example, by S 38 The obtained classification feature R cls The number of channels is 1, the size is 10×10, and the regression feature R reg The number of channels is 4, the size is 10×10, and R cls The maximum value is at (5, 4), and R cls The maximum position (5, 4) of is mapped to the search area S 38 The corresponding target centroid position is (107, 86), R reg The distances from the center of the target to the four boundaries of the target box are (8, 12, 8, 12). 38 The corresponding target centroid position and the distance from the centroid position to the four boundaries of the target frame can be used to draw the target frame and determine S 38 The position of the target

[0072] Step 15, sequentially load each frame of the hyperspectral image in the hyperspectral image sequence, repeat steps 1 to 14, and complete the target tracking of the hyperspectral image sequence.

[0073] Exemplarily, the grayscale normalized hyperspectral image sequence of a coke can is loaded in sequence, with a total of 331 frames. The resolution of each hyperspectral image frame in the coke can hyperspectral image sequence is 256×512 pixels. Steps one to fourteen are repeated to complete target tracking of the coke can hyperspectral image sequence.

[0074] The present invention proposes a hyperspectral video tracking method based on spectral mapping enhancement. First, by introducing hyperspectral information, the present invention effectively overcomes the dependence of existing tracking algorithms based on filters and fully convolutional twin networks on spatial information and the lack of effective use of spectral information. Through spectral angular distance operations and spectral mapping enhancement techniques, this method can retain the spectral characteristics of the target during dimensionality reduction and enhance the distinction between the target and the background, thereby improving the robustness and accuracy of tracking. In addition, compared with the existing dual-branch fully convolutional twin network, a new deep feature extraction branch of the template image is introduced to obtain a multidimensional response map generated by the template image features and the search image features, which further optimizes the accuracy of determining the target position. Compared with the existing methods, the present invention has higher adaptability under lighting and occlusion conditions. It has stronger robustness and robustness.

[0075] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention.

Claims

1. A hyperspectral video tracking method based on spectral mapping enhancement, characterized in that: The method is: Step 1: Load the target position, target frame and target image block of the first frame of the hyperspectral image in the grayscale normalized hyperspectral image sequence, then load the tth frame of the hyperspectral image in the grayscale normalized hyperspectral image sequence, and determine the search area S of the tth frame of the hyperspectral image in the grayscale normalized hyperspectral image sequence. t , where the grayscale range of the pixel of the t-th frame of the hyperspectral image in the hyperspectral image sequence after grayscale normalization is a left-closed and right-closed interval from 0 to 1, t represents the number of frames of the hyperspectral image, and t is an integer greater than or equal to 2 and less than or equal to 1000; Step 2: Taking the target position of the t-1th frame of the hyperspectral image in the grayscale normalized hyperspectral image sequence as the center, t A 3×3 pixel image block is intercepted as the local area L of the t-th frame hyperspectral image after grayscale normalization. t , determine L t The i-th band image The spectral average C li , according to C li Determine L t The local spectral curve C l ; Step 3: Use C l For S t The spectral curve C corresponding to all pixel positions m,n Perform the spectral angle distance operation to obtain the initial dimension reduction result N of the t-th frame hyperspectral image in the hyperspectral image sequence after grayscale normalization. m,n , where m represents S t The horizontal coordinate of the pixel in S t The vertical coordinate of the pixel in Step 4: According to the N m,n The comparison relationship with the error threshold Th obtains the spectral mask M m,n ; Step 5: According to the M m,n and N m,n Determine the pixel position flatness coefficient σ m,n ; Step 6: According to the σ m,n The comparison relationship with the threshold β determines whether the pixel belongs to the boundary position between the target and the background, and obtains the target and background boundary matrix Q m,n ; Step 7: According to the Q m,n and σ m,n Determine the average value of the target internal flatness coefficient σ mean ; Step 8: σ at the boundary between the target and the background m,n Replace with σ mean Get the updated pixel position flatness coefficient Step nine, according to the The spectral mapping enhancement coefficient a is determined by the constant coefficient k; Step 10: According to a, N m,n and M m,n Determine the final dimension reduction result E m,n , the dimension reduction result of the target image block of the first frame of the hyperspectral image in the hyperspectral image sequence after grayscale normalization is recorded as the target image X, and the search area S of the tth frame of the hyperspectral image in the hyperspectral image sequence after grayscale normalization is recorded as t The dimension reduction result is recorded as the search image Z, and the dimension reduction result of the target image block of the hyperspectral image of the t-1th frame in the hyperspectral image sequence after grayscale normalization is recorded as the template image P; Step 11: According to the following formula, use the Resnet50 network to extract features of X, Z and P respectively, and concatenate the features of the third, fourth and fifth layers to obtain the concatenated target image features: Search image features after stitching Template image features after splicing Specifically include: Among them, Cat(·) represents the concatenation operation, F3(X) represents the features extracted from X using the third layer of Resnet50, F4(X) represents the features extracted from X using the fourth layer of Resnet50, F5(X) represents the features extracted from X using the fifth layer of Resnet50, F3(Z) represents the features extracted from Z using the third layer of Resnet50, F4(Z) represents the features extracted from Z using the fourth layer of Resnet50, F5(Z) represents the features extracted from Z using the fifth layer of Resnet50, F3(P) represents the features extracted from P using the third layer of Resnet50, F4(P) represents the features extracted from P using the fourth layer of Resnet50, and F5(P) represents the features extracted from P using the fifth layer of Resnet50; Step 12: Respectively and Perform cross-correlation operation and obtain the following formulas: and Multidimensional response map R1 after cross-correlation operation, and The multidimensional response graph R2 after the cross-correlation operation specifically includes: Among them, ⊙ represents the cross-correlation operation; Step 13: According to the following formula, R1 and R2 are average-weighted to obtain the final multi-dimensional response graph R, specifically including: R=0.5*R1+0.5*R2; Step 14: According to the following formula, use 1×1 classification convolution kernel K cls and a 1×1 regression convolution kernel K reg Perform convolution operation on R to obtain classification feature R cls With regression feature R reg , R cls The maximum value is taken as the centroid position of the target, R reg The distance from the center of the target to the four boundaries of the target frame is calculated using R cls and R reg You can draw the target frame to determine the location of the target, including: in, Represents the convolution operation; Step 15: Load each frame of the hyperspectral image in the hyperspectral image sequence in turn, and repeat steps 1 to 14 to complete the target tracking of the hyperspectral image sequence.

2. The hyperspectral video tracking method based on spectral mapping enhancement according to claim 1 is characterized in that: According to C li Determine L t The local spectral curve C l , including: Determine C l ; Where A(·) represents the averaging operation, i represents the serial number of the band, i is an integer greater than or equal to 1 and less than or equal to B, and B represents the total number of bands.

3. The hyperspectral video tracking method based on spectral mapping enhancement according to claim 1 or 2, characterized in that: The utilization of C l For S t The spectral curve C corresponding to all pixel positions m,n Perform the spectral angle distance operation to obtain the initial dimension reduction result N of the t-th frame hyperspectral image in the hyperspectral image sequence after grayscale normalization. m,n , including: Determine N m,n ; Where acrcos(·) represents the arccosine operation, and T represents the transpose operation.

4. The hyperspectral video tracking method based on spectral mapping enhancement according to claim 3 is characterized in that: According to the N m,n The comparison relationship with the error threshold Th obtains the spectral mask M m,n , including: Determine the spectral mask M m,n , where N m,n When it is greater than Th, it is the background area, and the corresponding position value of the spectrum mask is 0. m,n When it is less than or equal to Th, it is the target area, the corresponding position value of the spectral mask is 1, and Th is set to 0.

22.

5. The hyperspectral video tracking method based on spectral mapping enhancement according to claim 4 is characterized in that: According to the M m,n and N m,n Determine the pixel position flatness coefficient σ m,n , including: Determine σ m,n , where N k,l It means traversing the eight neighborhood values ​​of the target pixel. The value range of k is [m-1, m+1] and it is an integer. The value range of l is [n-1, n+1] and it is an integer. ∑(·) means the sum operation, |·| means the absolute value operation, and * means the multiplication of corresponding pixels at the same position.

6. The hyperspectral video tracking method based on spectral mapping enhancement according to claim 5 is characterized in that: According to the σ m,n The comparison relationship with the threshold β determines whether the pixel belongs to the boundary position between the target and the background, and obtains the target and background boundary matrix Q m,n , including: Determine whether the pixel belongs to the boundary between the target and the background, and obtain the boundary matrix Q between the target and the background m,n , where if σ m,n If σ is less than or equal to β, the pixel does not belong to the boundary between the target and the background. m,n If it is greater than β, the pixel belongs to the boundary between the target and the background.

7. The hyperspectral video tracking method based on spectral mapping enhancement according to claim 6 is characterized in that: According to the Q m,n and σ m,n Determine the average value of the target internal flatness coefficient σ mean , including: Determine the average value of the target internal flatness coefficient σ mean , p represents the search area S t width, q represents the search area S t of high.

8. The hyperspectral video tracking method based on spectral mapping enhancement according to claim 7 is characterized in that: The σ at the boundary between the target and the background m,n Replace with σ mean Get the updated pixel position flatness coefficient Specifically include: Replace the flatness coefficient at the boundary between the target and the background with σ mean , get the updated pixel position flatness coefficient 9. The hyperspectral video tracking method based on spectral mapping enhancement according to claim 8 is characterized in that: According to the The spectral mapping enhancement coefficient a is determined by the constant coefficient k, specifically including: The spectral mapping enhancement coefficient a is determined, where the constant coefficient k is set to 0.

7.

10. The hyperspectral video tracking method based on spectral mapping enhancement according to claim 9 is characterized in that: According to a, N m,n and M m,n Determine the final dimension reduction result E m,n , the dimension reduction result of the target image block of the first frame of the hyperspectral image in the hyperspectral image sequence after grayscale normalization is recorded as the target image X, and the search area S of the tth frame of the hyperspectral image in the hyperspectral image sequence after grayscale normalization is recorded as t The dimension reduction result is recorded as the search image Z, and the dimension reduction result of the target image block of the hyperspectral image of the t-1th frame in the hyperspectral image sequence after grayscale normalization is recorded as the template image P. Specifically, it includes: according to E m,n =N m,n -N m,n *a*M m,n Determine the final dimension reduction result E m,n .

Images