Hyperspectral video target tracking method based on depth spectrum target perception characteristics

Abstract

The invention discloses a hyperspectral video target tracking method based on depth spectrum target perception features, and the method comprises the steps: determining a target region Tt of a tth frame of hyperspectral image after normalization; determining a selected local area # imgabs0 # in the first frame of hyperspectral image Tt, determining a maximum spectrum curve # imgabs1 # and a minimum spectrum curve # imgabs1 # through the spectrum curve of each pixel, and comparing the pixel spectrum curve of the search area with # imgabs3 # of # imgabs2 #; segmenting the image into a target area and a background area, obtaining a target spectrum curve Co and a background spectrum curve Ck, obtaining a spectrum curve of each pixel point in a search area, carrying out dimension reduction processing on the spectrum curve to obtain a band graph after dimension reduction, extracting depth features and edge features of the image, covering the depth features and the edge features with a spectrum mask in dimension reduction, and obtaining a band graph after dimension reduction; obtaining a target sensing feature P, obtaining a weight of each spectrum channel according to an information entropy curve of the P, multiplying the weight with the P of the corresponding channel to obtain a spectrum target sensing feature # imgabs4 #, performing pixel-by-pixel convolution on the spectrum target sensing feature # imgabs4 # and the depth feature E to obtain a depth spectrum target sensing feature Uz, sending the Uz of a first frame image into a CACF filter to train a template, and obtaining a target sensing feature P; uz of the (t + 1) th frame of hyperspectral image is sent to a filter template to obtain a response graph, confidence determination and scale estimation are carried out on the response graph, a tracking target of the current frame of hyperspectral image is obtained, and whether the (t + 1) th frame of hyperspectral image is updated or not is judged according to confidence determination.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a hyperspectral video target tracking method based on deep spectral target perception characteristics. Background Art

[0002] As a research hotspot in the field of computer vision, object tracking is widely used in hot areas such as autonomous driving, public safety, and human-computer interaction.

[0003] With the increasing attention paid to aerospace by countries around the world, and thanks to the development and maturity of hyperspectral imaging technology, the field of hyperspectral target tracking has also seen rapid growth. Its technology is widely used in a variety of fields, including the environment, agriculture, military, and geography. Hyperspectral imaging, similar to other spectral imaging methods, collects and processes information from the electromagnetic spectrum. Hyperspectral imaging aims to obtain the spectrum of each pixel in an image of a specific scene, with the goal of finding objects, identifying materials, or detecting targets. Spectral imaging, on the other hand, divides the spectrum into multiple spectral segments. This technique of segmenting images into spectral segments can be extended beyond visible light. Hyperspectral target tracking has attracted attention due to its accuracy and sensitivity in target tracking. A hyperspectral image is a three-dimensional data cube composed of two spatial dimensions and one spectral dimension. Hyperspectral images are an emerging area in the field of target tracking. The rich spectral band information in the spectral dimension offers irreplaceable advantages for target tracking. Unlike visible light image data (RGB) or single-channel images, hyperspectral target tracking can distinguish different objects even when visible light or infrared light is difficult to discern.

[0004] The significant breakthroughs achieved by deep convolutional neural networks have fueled research on a range of fundamental tasks in computer vision. In the field of object tracking, a wide variety of tracking algorithms can benefit from the powerful feature extraction capabilities of convolutional neural networks. Deep learning-based object tracking algorithms can be broadly categorized into two types, based on their core model design concepts: discriminative correlation filtering and siamese network tracking. The discriminative correlation filtering algorithm frames the tracking task as a foreground and background classification problem, employing circulant matrix theory and kernel function techniques to solve a ridge regression problem. While maintaining real-time tracking, its tracking accuracy surpasses that of other contemporary algorithms. Unlike the discriminative correlation filtering algorithm, the siamese network tracking algorithm benefits from extensive offline data, enabling fully end-to-end training without relying on any online update strategies to improve the algorithm's discriminability. Consequently, the siamese network tracking algorithm strikes a balance between tracking performance and inference speed, making it one of the most popular tracking algorithms.

[0005] Traditional visual object tracking faces various challenges, such as difficulty updating the tracker quickly when the target resembles the background, when the target moves rapidly, or when the target's scale changes. Accurately acquiring target information and updating the tracker in a timely manner are crucial for fast and accurate tracking. This paper focuses on the research within the framework of correlation filtering, combining the method of tracking spatial targets using Fourier transforms to the frequency domain with spectral information. Summary of the Invention

[0006] In view of this, the main object of the present invention is to provide a hyperspectral target tracking method based on deep spectral target perception characteristics.

[0007] A hyperspectral video target tracking method based on deep spectral target perception features, the method is:

[0008] According to the first frame of hyperspectral image in the hyperspectral image sequence, the target area T of the normalized t-th frame of hyperspectral image is obtained. t and search area S t ;

[0009] Acquire a spectral curve of the search area according to the first hyperspectral image in the hyperspectral image sequence Maximum spectral curve C max and the minimum spectral curve C min ;

[0010] According to the and Determine S t Spectral deviation diagram D ib ;

[0011] The spectral deviation map D ib Divide into target area and background area to obtain the target spectrum curve C o and background spectrum curve C k ;

[0012] The pair S t Perform dimensionality reduction processing to obtain a band image after dimensionality reduction, extract the depth features and edge features of the image, cover it with the spectral mask in the dimensionality reduction, and obtain the target perception feature P;

[0013] The weight of each spectral channel is obtained according to the information entropy curve of P, and multiplied by the P of the corresponding channel to obtain the spectral target perception feature

[0014] The spectral target perception feature Perform pixel-by-pixel convolution with the depth feature E to obtain the deep spectral target perception feature U z ;

[0015] The U of the first frame of hyperspectral imagez The template is trained by CACF filter and the U of the t+1 frame hyperspectral image is converted to z The filter template is fed into the response map to obtain the confidence level and scale estimation, and the tracking target of the current frame hyperspectral image is obtained. Then, the confidence level is used to determine whether to update the t+1 frame hyperspectral image.

[0016] Preferably, the and Determine S t Spectral deviation diagram D ib , specifically including:

[0017] (301) Determine S t The deviation between the pixel and the maximum spectral curve is

[0018] (302) Determine S t The deviation between the pixel and the minimum spectral curve is

[0019] (303) Determine S t Spectral difference map D ib for

[0020]

[0021] Preferably, the S t X on o (i,b) and N o (i,b) Determine the target spectrum curve C o , according to the S t X on k (i,b) and N k (i,b) Determine the background spectrum curve C k , specifically including:

[0022] (601) Confirm The spectral intensity feature X corresponding to the background k The cumulative sum X of (i,b) k (b)

[0023]

[0024] Among them, ∑(·) is the cumulative sum operation, represents the search area of the bth band of the tth frame hyperspectral image;

[0025] (602) Confirm The number of features N corresponding to the background k The cumulative sum N of (i,b) k (b)

[0026]

[0027] (603) Determine the background spectrum curve C of the search area according to the following formula k for

[0028]

[0029] Among them, C kb Indicates the target spectrum curve C k Spectral value in the bth band;

[0030] (604) OK The spectral intensity feature X corresponding to the target k The cumulative sum X of (i,b) o (b)

[0031]

[0032] (605) OK The number of features N corresponding to the target k The cumulative sum N of (i,b) o (b)

[0033]

[0034] (606) Determine the target spectrum curve C of the search area according to the following formula o for

[0035]

[0036] Among them, C ob Indicates the target spectrum curve C o Spectral value in the bth band.

[0037] Preferably, the C o and C k Determine S t The spectral weight map μ i , specifically including:

[0038] (701) Determine S t The pixel curve and background spectrum curve C k The spectral angular distance is

[0039] λ k =C Si ΘC k

[0040] Among them, Θ is the spectral angular distance operation, C Si Indicates that in St The spectral curve of the i-th pixel in;

[0041] (702) Determine S t Pixel curve and target spectrum curve C o The spectral angular distance is

[0042] λ o =C Si ΘC o ;

[0043] (703) Determine S t The spectral weight map μ i for

[0044] Preferably, the obtaining of S t The b-th band image Target perception features on Sure Image information entropy according to Determine the spectral information entropy value C of the target perception feature of the tth frame in the bth band eb , through C eb Obtain the spectral information entropy curve C of the search area e , according to C e Determine the weight ω of the bth spectral channel b , specifically including:

[0045] (1301) is determined as follows: for

[0046]

[0047] Where n is the gray level of the image, and p(n) is the probability of the nth gray level appearing;

[0048] (1302) Determine the spectral information entropy curve C of the search area according to the following formula e for

[0049]

[0050] Among them, C eb Represents the target perception feature of the tth frame The spectral information entropy value in the bth band;

[0051] (1303) According to the following formula Perform normalization processing to obtain the normalized value ω of the image signal-to-noise ratio of the b-th band b for

[0052]

[0053] Preferably, the method of P and ω b Determine S t+1 Spectral target perception characteristics Specifically include:

[0054] (1401) is obtained according to the following formula Spectral target perception characteristics in the bth band for

[0055]

[0056] Among them, P b represents the target perception feature of P in the bth band;

[0057] (1402) After b traverses 1 to B, all P b The collection of

[0058]

[0059] Preferably, the R t+1 Determine the interference factor β of the t+1th frame t+1 , specifically including:

[0060] (1701) Determine R t+1 Maximum value max(R t+1 );

[0061] (1702) Determine R t+1 The jth maximum value of the surrounding eight neighborhoods

[0062] (1703) According to the following formula, using R t+1 and Determine the current frame interference factor β t+1 for

[0063]

[0064] Preferably, the t+1 and Determine the current frame reference value Specifically include:

[0065] (1801) Determine R t+1 Interference factor β t+1 ;

[0066] (1802) According to the following formula, using β t+1 Determine the current frame reference value for

[0067]

[0068] in, Indicates the reference value of the previous frame,

[0069] Preferably, the interference factor β according to the t+1 frame t+1 and the current frame reference value Update the current frame confidence F c (t+1), specifically including:

[0070] (1901) According to the following formula, using β t+1 and Update the current frame confidence F c (t+1) is

[0071]

[0072] Preferably, the current frame confidence F c (t+1) Update R t+1 , specifically including:

[0073] (2001)Set Record historical frame information

[0074]

[0075] Among them, m represents the number of historical frames of the hyperspectral image, m∈[1,t]. When m traverses from 1 to t, the mth historical frame information is obtained.

[0076] (2002) set the update threshold μ;

[0077] (2003) According to the following formula, using F c (t+1) and μ to update R t+1

[0078]

[0079] Compared with the existing technology, the present invention has the following advantages: Compared with the traditional hyperspectral target tracking method, the present invention enhances the spectral difference between the target and the background during the dimensionality reduction process by dynamically generating a spectral mask. The proposed method combines the features with the depth features and dynamically adjusts the weights of each band through spectral information entropy, highlighting the salient areas of the target while simultaneously capturing both spatial-spectral and semantic information. Furthermore, the proposed method improves the model update strategy by introducing an interference coefficient β to assess the reliability of the current frame and dynamically select the optimal historical frame, thereby enhancing the robustness of tracking.

[0080] In summary, the present invention achieves significant improvements in accuracy, robustness, and efficiency in hyperspectral video tracking through spectral-spatial joint modeling, dynamic feature fusion, and adaptive model updating. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 is a flow chart of the present invention;

[0082] Figure 2 This is the 30th frame of the normalized hyperspectral image of a car in an embodiment of the present invention;

[0083] Figure 3 In the embodiment of the present invention, the search area S of the 30th frame of the normalized hyperspectral image of the car is 30 ;

[0084] Figure 4 In the embodiment of the present invention, the target area T of the first frame of the normalized hyperspectral image of the car is 30 ;

[0085] Figure 5 Select a local area for the first frame of hyperspectral image in the present invention The maximum spectral curve and the minimum spectral curve;

[0086] Figure 6 In the embodiment of the present invention, the search area S of the 31st frame of the normalized hyperspectral image of the car is 31 A band grayscale image after dimensionality reduction;

[0087] Figure 7 In the embodiment of the present invention, the search area S of the 31st frame of the normalized hyperspectral image of the car is 31 The depth features of a band of grayscale images after dimensionality reduction;

[0088] Figure 8 In the embodiment of the present invention, the search area S of the 31st frame of the normalized hyperspectral image of the car is 31 Spectrum of MSAK;

[0089] Figure 9 In the embodiment of the present invention, the search area S of the 31st frame of the normalized hyperspectral image of the car is 31 edge feature map;

[0090] Figure 10 In the embodiment of the present invention, the normalized search area S of the 31st frame of the hyperspectral image of the car is 31 Spectral target perception feature map;

[0091] Figure 11 In the embodiment of the present invention, the normalized search area S of the 31st frame of the hyperspectral image of the car is 31Deep spectral target perception U z Feature map;

[0092] Figure 12 In the embodiment of the present invention, the normalized search area S of the 31st frame of the hyperspectral image of the car is 31 Enter the response graph obtained by CACF filter;

[0093] Figure 13 This is a normalized target tracking result diagram of the 31st frame of the hyperspectral image of a car in an embodiment of the present invention. Specific implementation plan

[0094] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific implementations described herein are only used to solve the present invention and are not intended to limit the present invention.

[0095] The embodiment of the present invention provides a hyperspectral video target tracking method based on deep spectral target perception characteristics, such as Figure 1 As shown, the method is:

[0096] Step 1: Obtain the target area T of the normalized t-th frame hyperspectral image according to the first frame hyperspectral image in the hyperspectral image sequence. t and search area S t ;

[0097] Specifically, the target position, target frame and target image block of the first frame of the hyperspectral image in the hyperspectral image sequence are loaded, and then the t-th frame of the hyperspectral image in the hyperspectral image sequence is loaded, and the grayscale of the t-th frame of the hyperspectral image is normalized to obtain the normalized t-th frame of the hyperspectral image. Determine the target area T of the normalized hyperspectral image frame t t and search area S t ;

[0098] Among them, the pixel of the hyperspectral image of the tth frame The grayscale range is a left-closed and right-closed interval from 0 to 1, t represents the number of frames of the hyperspectral image, and t is an integer greater than or equal to 2. ib represents the i-th pixel in the b-th band, i represents the pixel number, b represents the band number, b∈{1,...,B}, B represents the number of bands in the hyperspectral image, B is 16, H represents The number of rows, W represents The number of columns, H T Indicates T t The number of rows, W T Indicates T t The number of columns, H S Indicates S tThe number of rows, W S Indicates S t The number of columns;

[0099] Specifically, the 30th frame of hyperspectral image in the hyperspectral image sequence is loaded, and the grayscale of the 30th frame of hyperspectral image is normalized to obtain the normalized 30th frame of hyperspectral image Determine the 30th frame of hyperspectral image after normalization Search area S 30 , S t The target image block T of the previous frame t Get S t The size of the target image block T in the previous frame t 1.5 times the size, S 30 There are B bands, the image resolution is p×q, p is 197, q is 105, B is 16, and the loaded hyperspectral image sequence has a total of 674 frames, with an image resolution of 526×347 pixels. Figure 2 In the embodiment of the present invention, the 30th frame of the normalized hyperspectral image of the car is shown. Figure 3 In the embodiment of the present invention, the search area S of the 30th frame of the normalized hyperspectral image of the car is 30 ,Because the hyperspectral image is obtained by a hyperspectral camera with 16 wavelength bands from 470nm to 620nm, the hyperspectral camera is snapshot VIS produced by IMEC, so B is 16.

[0100] Step 2: Obtain the maximum spectral curve C according to the first frame of the hyperspectral image in the hyperspectral image sequence max and the minimum spectral curve C min ;

[0101] Specifically, the target position of the first frame of the hyperspectral image in the hyperspectral image sequence is taken as the center, and the 1 A 3×3 pixel image block is intercepted as the local area of the first frame of the hyperspectral image after normalization. Through statistics The spectral curve of each pixel in the image is obtained, and the maximum and minimum values of each band are screened out to determine the maximum spectral curve C max and the minimum spectral curve C min , The grayscale value range of the i-th pixel in the b-th band is a left-closed and right-closed interval from 0 to 1. max The maximum spectral value in the bth band is C min The minimum spectral value in the bth band is

[0102] In the 30th frame of the hyperspectral image, Take the maximum gray value of the 9 pixels in the first band, then traverse bands 1 to 16, select the 16 maximum gray values to obtain the maximum spectral curve C max , When b changes from 1 to 16, we get C max ; C max The maximum spectral values in the 1st to 16th bands are {0.2987, 0.3934, 0.3007, 0.3143, 0.2871, 0.2803, 0.2621, 0.2714, 0.2573, 0.2471, 0.2201, 0.2603, 0.2931, 0.3093, 0.2904, 0.3190}, respectively. Take the minimum gray value of the 9 pixels in the first band, then traverse bands 1 to 16, select the 16 minimum gray values to obtain the minimum spectral curve When b changes from 1 to 16, we get C min ; C min The minimum spectral values in the 1st to 16th bands are {0.1811, 0.1446, 0.1026, 0.1093, 0.1293, 0.1252, 0.1532, 0.1491, 0.1342, 0.1421, 0.1193, 0.1203, 0.1671, 0.1147, 0.1076, 0.1307} respectively.

[0103] Figure 4 In the embodiment of the present invention, the target area T of the first frame of the normalized hyperspectral image of the car is 1 ; Figure 5 Select a local area for the first frame of hyperspectral image in the present invention The maximum and minimum spectral curves.

[0104] Step 3: Pass and Determine S t Spectral deviation diagram D ib ,in,

[0105] Specifically, (301) determines S t The deviation between the pixel and the maximum spectral curve is

[0106]

[0107] (302) Determine S t The deviation between the pixel and the minimum spectral curve is

[0108]

[0109] (303) Determine S t Spectral difference map D ib for

[0110]

[0111] Specifically, in the present invention, the search area S of the 30th frame of hyperspectral image is 30 The maximum spectral curve deviation m of the 63rd pixel in the 4th band is -0.4233, the minimum spectral curve deviation n is -0.1370, and the spectral difference value is The maximum spectral curve deviation m of the 122nd pixel in the 9th band is -0.0234, the minimum spectral curve deviation n is 0.5478, and the spectral difference value is It is -0.0149.

[0112] Step 4: Determine the spectral intensity characteristic X of the target o (i,b) and quantity feature N o (i,b);

[0113] Specifically, in the present invention, the search area S of the 30th frame of hyperspectral image is 30 The spectral difference of the 72nd pixel in the 5th band is equal to The pixel is classified as a target pixel, and the spectral value of the current target pixel is assigned to the spectral intensity feature X o (72,5), quantity feature N o (72,5) is assigned the value 1.

[0114] Step 5: Determine the background spectral intensity characteristic X by complement operation k (i,b) and quantity feature N k (i,b);

[0115] Specifically, in the present invention, the search area S of the 30th frame of hyperspectral image is 30 The spectral difference of the 95th pixel in the 7th band is not equal to The pixel is classified as a background pixel, and the spectral value of the current background pixel is assigned to the spectral intensity feature X k (95,7), quantity feature N k (72,5) is assigned the value 0.

[0116] Step 6: According to the S t X on o (i,b) and N o (i,b) Determine the target spectrum curve C o , according to the S t X on k (i,b) and N k(i,b) Determine the background spectrum curve C k ;

[0117] (601) Confirm The spectral intensity feature X corresponding to the background k The cumulative sum X of (i,b) k (b)

[0118]

[0119] Among them, ∑(·) is the cumulative sum operation, represents the search area of the bth band of the tth frame hyperspectral image;

[0120] Specifically, the search area of the 30th frame of hyperspectral image after normalization The sum of the spectral intensity characteristics of the fifth band belonging to the background pixels Its value is 2478.

[0121] (602) Confirm The number of features N corresponding to the background k The cumulative sum N of (i,b) k (b)

[0122]

[0123] Specifically, the search area of the 30th frame of hyperspectral image after normalization The cumulative sum of the number of background pixels in the fifth band Its value is 4876.

[0124] (603) Determine the background spectrum curve C of the search area according to the following formula k for

[0125]

[0126] Among them, C kb Indicates the target spectrum curve C k Spectral value in the bth band;

[0127] Specifically, the search area of the 30th frame of the normalized hyperspectral image is The spectral intensity feature X of the background pixel k (i,5) and the number of features N k (i,5) are accumulated respectively, and then determined The spectral average value C of the background pixels k5 is 0.4217, through S 30 The spectral average of the 16 band images is used to determine the search area S of the 30th frame of the normalized hyperspectral image.30 Background spectrum curve C k , and obtain C k The spectral average values in the 1st to 16th bands are {0.2871, 0.3196, 0.3542, 0.3619, 0.4217, 0.4329, 0.4783, 0.5896, 0.5132, 0.7403, 0.6325, 0.6832, 0.5849, 0.5912, 0.6839, 0.7825} respectively.

[0128] (604) OK The spectral intensity feature X corresponding to the target k The cumulative sum X of (i,b) o (b)

[0129]

[0130] Specifically, the search area of the 30th frame of hyperspectral image after normalization The cumulative sum of the spectral intensity features of the fifth band belonging to the target pixel Its value is 17589.

[0131] (605) OK The number of features N corresponding to the target k The cumulative sum N of (i,b) o (b)

[0132]

[0133] Specifically, the search area of the 30th frame of hyperspectral image after normalization The cumulative sum of the number of features of the 5th band belonging to the target pixel Its value is 15809.

[0134] (606) Determine the target spectrum curve C of the search area according to the following formula o for

[0135]

[0136] Among them, C ob Indicates the target spectrum curve C o Spectral value in the bth band.

[0137] Specifically, the search area of the 30th frame of the normalized hyperspectral image is The spectral intensity feature X belonging to the target pixel o (i,5) and the number of features N o (i,5) are accumulated respectively, and then determined The spectral average value C of the target pixel o5 is 0.4217, through S 30 The spectral average of the 16 band images is used to determine the search area S of the 30th frame of the normalized hyperspectral image. 30 The target spectrum curve C o , and obtain C o The spectral average values in the 1st to 16th bands are {0.8217, 0.9316, 0.5324, 0.9163, 0.7241, 0.9324, 0.8473, 0.8569, 0.5364, 0.7403, 0.6325, 0.6832, 0.8549, 0.9152, 0.8369, 0.7825} respectively.

[0138] Step 7: Pass C o and C k Determine S t The spectral weight map μ i ,in,

[0139] (701) Determine S t The pixel curve and background spectrum curve C k The spectral angular distance is

[0140] λ k =C Si ΘC k (10)

[0141] Among them, Θ is the spectral angular distance operation, C Si Indicates that in S t The spectral curve of the i-th pixel in;

[0142] (702) Determine S t Pixel curve and target spectrum curve C o The spectral angular distance is

[0143] λ o =C Si ΘC o (11)

[0144] (703) Determine S t The spectral weight map μ i for

[0145] Specifically, in the present invention, the 30th frame of hyperspectral image S 30 The C of the 79th pixel spectrum curve S79The spectral values in the 1st to 16th bands are {0.3771, 0.4232, 0.4572, 0.5123, 0.5783, 0.5863, 0.6114, 0.6235, 0.5874, 0.7840, 0.6455, 0.6845, 0.5874, 0.6324, 0.7145, 0.6874}, which are consistent with the background spectrum curve C. k The spectral angle distance is 1.781; the 30th frame of hyperspectral image S t The C of the 79th pixel spectrum curve S79 The spectral values in the 1st to 16th bands are {0.3571, 0.4512, 0.4572, 0.5233, 0.5493, 0.5943, 0.6354, 0.6455, 0.5754, 0.7640, 0.6554, 0.6875, 0.5774, 0.6414, 0.7235, 0.6754}, which are consistent with the target spectrum curve C. o The spectral angle distance is 0.75; determine the 30th frame of hyperspectral image S 30 The C of the 79th pixel spectrum curve S79 The spectral weight μ 79 for

[0146] Step 8: Confirm and C o The naive correlation of i Retarget the naive correlation results and obtain the dimension reduction result D i ;

[0147] Specifically, the dimensionality reduction result D is determined according to the following formula: i

[0148]

[0149] in, Represents the naive correlation operation of two vectors. In the present invention, the 30th frame of hyperspectral image S 30 The 79th pixel spectrum curve C S79 The spectral weight μ 79 is 2.375, the target spectrum curve C o with C S79 The naive correlation results in a spectrum value of 0.891, so its dimensionality reduction result is D i The value of the 79th pixel is D 79 It is 2.116.

[0150] Step 9: Extract the deep features E of the one-band grayscale image sequence after dimensionality reduction through the DenseNet network;

[0151] Specifically, the DenseNet network is an existing technology. The Dense Block (3) layer of the DenseNet used in the present invention is used to extract deep features. The size of E in the present invention is a deep feature of 14×14×512, where 14×14 represents the spatial size and 196 is the number of channels. Figure 7 In the embodiment of the present invention, the search area S of the 31st frame of the normalized hyperspectral image of the small car is 31 Deep features of a one-band grayscale image after dimensionality reduction.

[0152] Step 10: Pass N o (i, b) Generate multiple spectral position prediction mask M;

[0153] Specifically, generate a o (i,b) are matrices M of the same size, and N o The value of (i,b) is assigned to the matrix M. Figure 8 In the embodiment of the present invention, the search area S of the 31st frame of the normalized hyperspectral image of the car is 31 Spectrum of MSAK;

[0154] Step 11: Extract S through Canny operator t+1 The edge feature F;

[0155] The Canny edge feature extraction operator is used in the present invention, and the size of F is 197×105×16, where 197×105 is the spatial resolution of F and 16 is the number of channels of F. Figure 9 In the embodiment of the present invention, the search area S of the 31st frame of the normalized hyperspectral image of the car is 31 The edge feature map of .

[0156] Step 12: Determine S through F and M t+1 The target perception feature P;

[0157] Perform dot multiplication of the multi-spectral position prediction mask M and the edge feature F, and obtain the target perception feature P according to the following formula:

[0158] P=F⊙M (13)

[0159] Among them, ⊙ represents the dot product operation.

[0160] Specifically, in the present invention, the size of P is 197×105×16, 197×105 is the spatial resolution of P, and 16 is the number of channels of P.

[0161] Step 13: Get S t The b-th band image Target perception features on Sure Image information entropy according to Determine the spectral information entropy value C of the target perception feature of the tth frame in the bth band eb , through C eb Obtain the spectral information entropy curve C of the search area e , according to C e Determine the weight ω of the bth spectral channel b ;

[0162] (1301) is determined as follows: for

[0163]

[0164] Where n is the gray level of the image, and p(n) is the probability of the nth gray level appearing;

[0165] (1302) Determine the spectral information entropy curve C of the search area according to the following formula e for

[0166]

[0167] Among them, C eb Represents the target perception feature of the tth frame The spectral information entropy value in the bth band;

[0168] Specifically, the normalized search area S of the 30th hyperspectral image is obtained: 30 16 band images, normalized 30th frame hyperspectral image search area S 30 In the 1st to 16th bands, They are {8.62, 5.32, 6.12, 8.51, 6.31, 5.63, 6.45, 7.82, 6.33, 5.65, 7.65, 5.45, 6.65, 8.43, 7.45, 6.32. The order is 4.82, 4.73, 3.48, 2.87, 3.54, 4.21, 4.33, 5.01, 4.21, 3.12, 3.46, 3.21, 4.83, 4.16, 4.32, 3.78}. Step (1303) Normalize to [0,1];

[0169] (1303) According to the following formula Perform normalization processing to obtain the normalized value ω of the image signal-to-noise ratio of the b-th band b for

[0170]

[0171] Specifically, the search area S of the 30th frame hyperspectral image after normalization 30 The weights of the ENT information entropy channel ω1 are 0.0752, ω2 is 0.0738, ω3 is 0.0543, ω4 is 0.0447, ω5 is 0.0552, ω6 is 0.0285, ω7 is 0.0657, ω8 is 0.0676, ω9 is 0.0782, ω 10 is 0.0657, ω 11 is 0.0487, ω 12 is 0.0540, ω 13 is 0.0501, ω 14 is 0.0754, ω 15 is 0.0674, ω 16 It is 0.0590. For edge features, the higher the ENT is, the more obvious the spectral features of the target are;

[0172] Step 14: Through P and ω b Determine S t+1 Spectral target perception characteristics

[0173] (1401) is obtained according to the following formula Spectral target perception characteristics in the bth band for

[0174]

[0175] Among them, P b represents the target perception feature of P in the bth band;

[0176] (1402) After b traverses 1 to B, all P b The collection of

[0177]

[0178] Specifically, in the present invention The size is 197×105×16, 197×105 is The spatial resolution is 16 The number of channels, Figure 10 In the embodiment of the present invention, the normalized search area S of the 31st frame of the hyperspectral image of the car is 31 Spectral target perception feature map.

[0179] Step 15: Pass And E are convolved pixel by pixel to determine S t+1 Deep spectral target perception feature U z ;

[0180] Specifically, in the present invention, U z The size of the U is 197×105×196, 197×105 is z Resolution, 196 for U z The number of channels, in this embodiment of the present invention, the spectral target perception feature of the 31st frame hyperspectral image The size is 197×105×16, and the size of the deep feature E is 14×14×512. The deep feature E is divided into 32 groups, each with 16 channels. In addition, each group of deep features is divided into 196 convolution kernels, each with a size of 1×1×16. Finally, each convolution kernel is combined with Perform convolution to obtain a U with a size of 197×105×196 z feature, Figure 11 In the embodiment of the present invention, the normalized search area S of the 31st frame of the hyperspectral image of the car is 31 Deep spectral target perception U z Feature map.

[0181] Step 16: Obtain the U extracted from the search area of the first frame of the hyperspectral image z Features, the U extracted from the search area of the first frame z The features are sent to the CACF filter for training to obtain the filter template, and then the S t+1 Extracted U z The features are fed into the filter template to obtain S t+1 Response graph R t+1 ;

[0182] Figure 12 In the embodiment of the present invention, the normalized search area S of the 31st frame of the hyperspectral image of the car is 31 Response plot obtained by entering the CACF filter.

[0183] Step 17: According to the R t+1 Determine the interference factor β of the t+1th frame t+1 ;

[0184] (1701) Determine R t+1 Maximum value max(R t+1 );

[0185] (1702) Determine R t+1 The jth maximum value of the surrounding eight neighborhoods

[0186] Specifically, read the depth spectrum target perception U z , take the target area as the positive sample area R t+1 , with the positive sample area R t+1 As the center, divide the positive sample area Rt+1 Eight surrounding neighborhoods j∈[1,8], which are eight neighborhoods, namely, top, bottom, left, right, upper left, lower left, upper right and lower right. The input filter obtains the response map and the positive sample area R t+1 The corresponding maximum value in the response graph is recorded as max(R t+1 ), eight neighborhood R t+1 The corresponding maximum value in the response graph is recorded as Using interference factor β t+1 To judge the interference degree of the t+1th hyperspectral image, the interference factor β is determined according to the following formula t+1 for

[0187] (1703) According to the following formula, using R t+1 and Determine the current frame interference factor β t+1 for

[0188]

[0189] Specifically, the maximum response peak value max(R 31 ) is 9.7463, The peak value corresponding to the neighborhood is 6.7267, The peak value corresponding to the neighborhood is 4.9714, The peak value corresponding to the neighborhood is 0.0020, The peak value corresponding to the neighborhood is 5.9023, The peak value corresponding to the neighborhood is 0.0011, The peak value corresponding to the neighborhood is 0.0243, The peak value corresponding to the neighborhood is 8.6021, Neighborhood corresponding The peak value is 6.5346; the current frame interference factor β can be determined 31 It is 2.380.

[0190] Step 18: Pass β t+1 and Determine the current frame reference value

[0191] (1801) Determine R t+1 Interference factor β t+1 ;

[0192] (1802) According to the following formula, using βt+1 Determine the current frame reference value for

[0193]

[0194] in, Indicates the reference value of the previous frame,

[0195] Specifically, in the embodiment of the present invention, when the 31st frame of the car hyperspectral image β 31 The value is greater than the reference value of the 30th frame When The value of the 31st frame reference value is assigned to Reference value of frame 30 The value of is 0.88. When the 31st frame of the basketball court hyperspectral image β 31 The value is less than or equal to the reference value of the 30th frame When β 31 The value of the 31st frame reference value is assigned to Reference value of frame 30 The value is 0.9, the reference value of the 31st frame The value of is 0.899.

[0196] Step 19: Update the current frame confidence F c (t+1);

[0197] (1901) According to the following formula, using β t+1 and Update the current frame confidence F c (t+1) is

[0198]

[0199] Specifically, in the embodiment of the present invention, the 31st frame reference value The value is 0.899, and the interference factor β of the 31st frame 31 The value of F is 2.380, and the confidence level of the 31st frame is F c The value of (31) is updated to 2.647.

[0200] Step 20: Update R t+1 ;

[0201] (2001)Set Record historical frame information

[0202]

[0203] Among them, m represents the number of historical frames of the hyperspectral image, m∈[1,t]. When m traverses from 1 to t, the mth historical frame information is obtained.

[0204] (2002) set the update threshold μ;

[0205] (2003) According to the following formula, using F c (t+1) and μ to update R t+1

[0206]

[0207] Among them, the larger the value of μ, the t+1 The more frequent the update, the larger the value of μ, and R t+1 The slower the update, μ is a number greater than 0 and less than 1. Since too frequent updates will cause target tracking drift, and too slow updates will reduce the effect of the proposed model, μ is set to 0.15 in the present invention.

[0208] Specifically, the confidence F of the 31st frame of the car hyperspectral image is c The value of (31) is 1.0216. When μ is set to 0.7, the confidence F of the hyperspectral image of the car in the 31st frame is c (70) is greater than μ, which means that the target of the 31st frame hyperspectral image after normalization has not drifted, and there is no need to change the 31st frame hyperspectral image after normalization. When μ is set to 1.2, the confidence F of the 31st frame hyperspectral image of the car is c (31) is less than μ, which means that the target of the 31st normalized hyperspectral image may have drifted. It is necessary to update the 31st normalized hyperspectral image back to the hyperspectral image with the least interference in the previous 31 frames to overcome the target drift phenomenon.

[0209] Step 21: Through R t+1 The scale estimation module realizes hyperspectral image target tracking and outputs the centroid coordinates and target frame size of the hyperspectral image target in the current frame;

[0210] Specifically, the response map R of the 31st frame of the car hyperspectral image is 31 Input into the scale estimation module to obtain the 31st frame of the car hyperspectral image target. The centroid coordinates of the target are (62, 55). The size of the target box is 15×15, where 15 represents the number of rows and 15 represents the number of columns.

[0211] Step 22: read each frame of the hyperspectral image in the hyperspectral image sequence in sequence, repeat steps 1 to 22, and complete target tracking of the hyperspectral image sequence.

[0212] Specifically, Figure 13This is the normalized target tracking result of the 31st frame of the hyperspectral image of a car in an embodiment of the present invention. Repeat steps 1 to 22 to obtain target blocks of 674 frames of hyperspectral images, completing the target tracking of the final hyperspectral image sequence.

[0213] This paper proposes a new feature, called spectral target perception feature, which is formed by combining edge features extracted by the Canny operator with the information entropy (ENT) spectral curve. As a new feature that can be applied to hyperspectral video target tracking, this feature contains rich spatial and spectral information.

[0214] The dimensionality reduction algorithm of the present invention is based on the difference between the target spectrum and the background spectrum. It not only reduces data redundancy, but also improves the separation between the target and the background. In addition, based on the characteristic of edge features to suppress background clutter, the depth feature is combined with the target perception feature containing spectral information. The generated deep target perception feature is used to overcome the challenge of background clutter and use the prior spectral information of the previous frame to improve the accuracy of target tracking in subsequent frames. In subsequent tracking, the present invention also proposes a model update strategy based on interference factors. The proposed confidence judgment model corrects the tracking results that are severely interfered with. The model combines the best positioning information in the historical frame.

[0215] The above descriptions are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention.

Claims

1. A hyperspectral video target tracking method based on deep spectral target perception features, characterized in that: The method is: According to the first frame of hyperspectral image in the hyperspectral image sequence, the target area T of the normalized t-th frame of hyperspectral image is obtained. t and search area S t ; Acquire a spectral curve of the search area according to the first hyperspectral image in the hyperspectral image sequence Maximum spectral curve C max and the minimum spectral curve C min ; According to the and Determine S t Spectral deviation diagram D ib ; The spectral deviation map D ib Divide into target area and background area to obtain the target spectrum curve C o and background spectrum curve C k ; The pair S t Perform dimensionality reduction processing to obtain a band image after dimensionality reduction, extract the depth features and edge features of the image, cover it with the spectral mask in the dimensionality reduction, and obtain the target perception feature P; The weight of each spectral channel is obtained according to the information entropy curve of P, and multiplied by the P of the corresponding channel to obtain the spectral target perception feature The spectral target perception feature Perform pixel-by-pixel convolution with the depth feature E to obtain the deep spectral target perception feature U z ; The U of the first frame of hyperspectral image z The template is trained by CACF filter and the U of the t+1 frame hyperspectral image is converted to z The filter template is fed into the response map to obtain the confidence level and scale estimation, and the tracking target of the current frame hyperspectral image is obtained. Then, the confidence level is used to determine whether to update the t+1 frame hyperspectral image.

2. The hyperspectral video target tracking method based on deep spectral target perception characteristics according to claim 1 is characterized in that: According to the and Determine S t Spectral deviation diagram D ib , specifically including: (301) Determine S t The deviation between the pixel and the maximum spectral curve is (302) Determine S t The deviation between the pixel and the minimum spectral curve is (303) Determine S t Spectral difference map D ib for 3. The hyperspectral video target tracking method based on deep spectral target perception characteristics according to claim 2 is characterized in that: According to the S t X on o (i,b) and N o (i,b) Determine the target spectrum curve C o , according to the S t X on k (i,b) and N k (i,b) Determine the background spectrum curve C k , specifically including: (601) Confirm The spectral intensity feature X corresponding to the background k The cumulative sum X of (i,b) k (b) Among them, ∑(·) is the cumulative sum operation, represents the search area of the bth band of the tth frame hyperspectral image; (602) Confirm The number of features N corresponding to the background k The cumulative sum N of (i,b) k (b) (603) Determine the background spectrum curve C of the search area according to the following formula k for Among them, C kb Indicates the target spectrum curve C k Spectral value in the bth band; (604) OK The spectral intensity feature X corresponding to the target k The cumulative sum X of (i,b) o (b) (605) OK The number of features N corresponding to the target k The cumulative sum N of (i,b) o (b) (606) Determine the target spectrum curve C of the search area according to the following formula o for Among them, C ob Indicates the target spectrum curve C o Spectral value in the bth band.

4. The hyperspectral video target tracking method based on deep spectral target perception characteristics according to claim 3 is characterized in that: Said through C o and C k Determine S t The spectral weight map μ i , specifically including: (701) Determine S t The pixel curve and background spectrum curve C k The spectral angular distance is λ k =C Si ΘC k Among them, Θ is the spectral angular distance operation, C Si Indicates that in S t The spectral curve of the i-th pixel in; (702) Determine S t Pixel curve and target spectrum curve C o The spectral angular distance is λ o =C Si ΘC o ; (703) Determine S t The spectral weight map μ i for 5. The hyperspectral video target tracking method based on deep spectral target perception characteristics according to claim 4 is characterized in that: The obtained S t The b-th band image Target perception features on Sure Image information entropy according to Determine the spectral information entropy value C of the target perception feature of the tth frame in the bth band eb , through C eb Obtain the spectral information entropy curve C of the search area e , according to C e Determine the weight ω of the bth spectral channel b , specifically including: (1301) is determined as follows: for Where n is the gray level of the image, and p(n) is the probability of the nth gray level appearing; (1302) Determine the spectral information entropy curve C of the search area according to the following formula e for Among them, C eb Represents the target perception feature of the tth frame The spectral information entropy value in the bth band; (1303) According to the following formula Perform normalization processing to obtain the normalized value ω of the image signal-to-noise ratio of the b-th band b for 6. The hyperspectral video target tracking method based on deep spectral target perception characteristics according to claim 5 is characterized in that: The P and ω b Determine S t+1 Spectral target perception characteristics Specifically include: (1401) is obtained according to the following formula Spectral target perception characteristics in the bth band for Among them, P b represents the target perception feature of P in the bth band; (1402) After b traverses 1 to B, all P b The collection of 7. The hyperspectral video target tracking method based on deep spectral target perception characteristics according to claim 6 is characterized in that: According to the R t+1 Determine the interference factor β of the t+1th frame t+1 , specifically including: (1701) Determine R t+1 Maximum value max(R t+1 ); (1702) Determine R t+1 The jth maximum value of the surrounding eight neighborhoods (1703) According to the following formula, using R t+1 and Determine the current frame interference factor β t+1 for 8. The hyperspectral video target tracking method based on deep spectral target perception characteristics according to claim 7 is characterized in that: The β t+1 and Determine the current frame reference value Specifically include: (1801) Determine R t+1 Interference factor β t+1 ; (1802) According to the following formula, using β t+1 Determine the current frame reference value for in, Indicates the reference value of the previous frame, 9. The hyperspectral video target tracking method based on deep spectral target perception characteristics according to claim 8 is characterized in that: The interference factor β according to the t+1 frame t+1 and the current frame reference value Update the current frame confidence F c (t+1), specifically including: (1901) According to the following formula, using β t+1 and Update the current frame confidence F c (t+1) is 10. The hyperspectral video target tracking method based on deep spectral target perception characteristics according to claim 9 is characterized in that: The confidence F of the current frame c (t+1) Update R t+1 , specifically including: (2001)Set Record historical frame information Among them, m represents the number of historical frames of the hyperspectral image, m∈[1,t]. When m traverses from 1 to t, the mth historical frame information is obtained. (2002) set the update threshold μ; (2003) According to the following formula, using F c (t+1) and μ to update R t+1

Images