Feature-enhanced infrared hyperspectral video target tracking methods, systems, and media

Abstract

This invention relates to a feature-enhanced infrared hyperspectral video target tracking method, system, and medium. The invention pertains to the field of digital image processing technology. The method includes: extracting the three bands with the highest information entropy from the original high-dimensional infrared hyperspectral image, thereby reducing the dimensionality of the original image. Then, infrared features are extracted and fused into the original image to achieve feature enhancement. Using the hyperspectral image with fused infrared features, HOG features and depth features are extracted, and a tracker is constructed based on a DCF framework. Feature-level fusion and decision-level fusion are implemented in the tracking process, ultimately achieving hyperspectral target tracking. This method enables fast and effective tracking of targets in hyperspectral image sequences against complex backgrounds.

Description

Technical Field

[0001] This invention relates to the field of digital image processing technology, and in particular to a feature-enhanced infrared hyperspectral video target tracking method, system, medium, and device. Background Technology

[0002] Visual tracking is a fundamental task in computer vision with wide applications. It is challenging because only the initial state of the target is available. While significant progress has been made over the past few decades, many challenges remain. In recent years, tracking methods based on Discriminative Correlation Filters (DCF) have attracted considerable attention due to their impressive performance and high speed. In DCF-based trackers, the filter is trained by minimizing the least-squares loss of all cyclic shifts of the training samples. Since the correlation operation can be performed in the Fourier domain, DCF has the advantage of high computational efficiency. The combination of features from DCF and deep convolutional neural networks (CNNs) has demonstrated state-of-the-art results.

[0003] Currently, the Hierarchical Convolutional Features for Visual Tracking (HCF) algorithm has been proposed, which constructs multiple DCFs on low, medium, and high-level features to capture spatial details and semantics. It uses multi-level DCF response maps to predict target location in a coarse-to-fine manner. Although feature-level fusion methods have been widely used or extended to improve performance, there is still room for improvement. This approach sometimes still fails to fully explore the relationships between multiple features. Furthermore, it is quite difficult to handle various challenging variations, and using a single model and relying on certain feature-level fusion strategies somewhat limits model diversity. The performance of a single tracker can sometimes be unstable, but decision-level fusion of the outputs of multiple trackers can effectively improve robustness. Summary of the Invention

[0004] Therefore, the technical problem to be solved by this invention is to overcome the limitations of existing technologies, such as the inability to fully explore the relationships between multiple features, the use of a single model, reliance on certain feature-level fusion strategies which to some extent restricts the diversity of models, and the occasional instability in the performance of a single tracker. To solve these technical problems, this invention provides a feature-enhanced infrared hyperspectral video target tracking method, system, medium, and device to achieve accurate and effective tracking of targets in hyperspectral image sequences against complex backgrounds.

[0005] To achieve the above objectives, the main technical solutions adopted by the present invention include:

[0006] In a first aspect, the present invention provides a feature-enhanced infrared hyperspectral video target tracking method, comprising: acquiring an original infrared hyperspectral image; preprocessing the original infrared hyperspectral image to obtain a preprocessed image; loading a first frame image from the preprocessed image as the current frame image; extracting a region of interest based on the current frame image; performing infrared feature enhancement on the region of interest to obtain an enhanced image; extracting the HOG feature of the enhanced image as a first feature; training the preprocessed image according to a preset file to obtain a trained image; extracting the Conv4-4 convolutional layer feature of the trained image as a second feature; extracting the Conv5-4 convolutional layer feature of the trained image as a third feature; obtaining a discriminant correlation filter tracker using a Gaussian response template based on the first, second, and third features; obtaining a first feature response map, a second feature response map, and a third feature response map based on the first, second, and third features and the discriminant correlation filter tracker; and performing infrared feature enhancement on the first feature response map, the second feature response map, and the third feature response map. Linear combination of the feature response maps yields the fourth, fifth, sixth, and seventh feature response maps; each feature response map serves as a tracking discriminator. The position of the maximum value in the first, second, third, fourth, fifth, sixth, and seventh feature response maps is used as the predicted location of the tracking target center. Robustness scores for each tracking discriminator in the current frame are obtained based on paired evaluation and self-evaluation scores. The tracking discriminator with the highest robustness score is selected to track the current frame. The discriminative correlation filter tracker is updated, and the tracking discriminators are adaptively updated. It is determined whether the current frame is the last frame of the preprocessed image: if yes, the target tracking task is completed, and the prediction result is obtained; if not, the next frame from the preprocessed image is loaded as the current frame, and the region of interest is extracted based on the current frame. Infrared feature enhancement is then performed on the region of interest to obtain the enhanced image.

[0007] In one embodiment of the present invention, the specific steps of the preprocessing include: performing band selection on the infrared hyperspectral image to obtain a preprocessed image; the specific steps of the band selection include: selecting three bands with the maximum information entropy from the original infrared hyperspectral image to form the preprocessed image.

[0008] In one embodiment of the present invention, the specific steps of obtaining a first feature response map, a second feature response map, and a third feature response map based on a first feature, a second feature, a third feature, and a discriminant correlation filter tracker include: obtaining a first feature response map based on the first feature and the discriminant correlation filter tracker; obtaining a second feature response map based on the second feature and the discriminant correlation filter tracker; and obtaining a third feature response map based on the third feature and the discriminant correlation filter tracker.

[0009] In one embodiment of the present invention, the specific steps of obtaining a fourth, fifth, sixth, and seventh characteristic response map by linearly combining the first, second, and third characteristic response maps include: obtaining a fourth characteristic response map by linearly combining the first and second characteristic response maps; obtaining a fifth characteristic response map by linearly combining the first and third characteristic response maps; obtaining a sixth characteristic response map by linearly combining the second and third characteristic response maps; and obtaining a seventh characteristic response map by linearly combining the first, second, and third characteristic response maps.

[0010] In one embodiment of the present invention, the specific steps for extracting the region of interest (ROI) from the current frame image and performing infrared feature enhancement on the ROI to obtain the enhanced image include: obtaining the coordinates of the upper left corner of the target box and the size of the target box in the current frame image; selecting the ROI based on the target box; extracting the first band of the current frame image; decomposing the current frame image into blocks using a quadtree to obtain multiple block images; performing an erosion operation on each block image and reconstructing the background of the current frame image using Bezier interpolation to obtain a reconstructed background image; subtracting the reconstructed background image from the current frame image to obtain a one-dimensional image of the infrared target enhancement; and fusing the one-dimensional image with the current frame image to obtain the enhanced image.

[0011] In one embodiment of the present invention, the Bessel interpolation method is expressed as:

[0012] Q(u,v)=UMPM T V T ,

[0013] Where (u,v) are interpolation points, U and V are variable interpolation coefficients; M is a constant interpolation coefficient matrix, and P is a 4*4 control point matrix consisting of 16 control points.

[0014] In one embodiment of the present invention, the first feature response map, the second feature response map, and the third feature response map are obtained according to the following formula:

[0015]

[0016] Among them, F-1 For the inverse Fourier transform, R is the Gaussian response plot of the HOG feature and two deep features. The filter is the result of the Fourier transform of the d-th band. It is the conjugate of the feature map image after the Fourier transform of the d-th band.

[0017] In one embodiment of the present invention, the discriminant correlation filter tracker is represented as:

[0018]

[0019]

[0020]

[0021]

[0022]

[0023] Where ω is the Gaussian filter, λ is the regularization parameter, and X is the data matrix linked by all cyclic shifts. Let be the filter on the d-th channel, d∈{1,2,…,D}, and ⊙ be the element-wise product; For the Discrete Fourier Transform (DFT) of a vector, for The complex conjugate of , where λ is the regularization parameter. For filter molecules, For filter The denominator is η, where η is the learning rate and t is the index of the current frame image.

[0024] In a second aspect, the present invention provides a feature-enhanced infrared hyperspectral video target tracking system, comprising:

[0025] The preprocessing module is used to acquire the original infrared hyperspectral image, preprocess the original infrared hyperspectral image, and obtain the preprocessed image.

[0026] The infrared feature enhancement module is used to load the first frame image from the preprocessed image as the current frame image, extract the region of interest based on the current frame image, perform infrared feature enhancement operation on the region of interest, and obtain the enhanced image.

[0027] The feature extraction module is used to extract the HOG features of the enhanced image as the first feature; train the preprocessed image according to the preset file to obtain the trained image, extract the Conv4-4 convolutional layer features of the trained image as the second feature, and extract the Conv5-4 convolutional layer features of the trained image as the third feature.

[0028] The discriminant correlation filter tracker generation module is used to obtain a discriminant correlation filter tracker based on the first feature, the second feature, and the third feature using a Gaussian response template.

[0029] The feature response map generation module is used to obtain a first feature response map, a second feature response map, and a third feature response map based on the first feature, the second feature, the third feature, and the discriminant correlation filter tracker; and to obtain a fourth feature response map, a fifth feature response map, a sixth feature response map, and a seventh feature response map by linearly combining the first feature response map, the second feature response map, and the third feature response map; wherein each feature response map serves as a tracking discriminant expert;

[0030] The prediction module uses the positions of the maximum values ​​in the first, second, third, fourth, fifth, sixth, and seventh feature response maps as the predicted location of the center of the tracking target.

[0031] The tracking module is used to obtain the robustness score of each tracking expert for tracking the current frame image based on the paired evaluation and self-evaluation scores of the tracking experts, and select the tracking expert with the highest robustness score to track the current frame image;

[0032] The update module is used to update the discrimination correlation filter tracker and adaptively update the tracking discrimination expert;

[0033] The judgment module is used to determine whether the current frame image is the last frame image of the preprocessed image:

[0034] If so, the target tracking task is completed and the prediction result is obtained;

[0035] If not, the next frame of the preprocessed image is loaded as the current frame image, and the region of interest is extracted based on the current frame image. Infrared feature enhancement is then performed on the region of interest to obtain the enhanced image.

[0036] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed, implements the feature-enhanced infrared hyperspectral video target tracking method described in any of the first aspects above.

[0037] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0038] The feature-enhanced infrared hyperspectral video target tracking method, system, medium, and device described in this invention achieves dimensionality reduction of the infrared hyperspectral image by extracting the three bands with the highest information entropy. Then, infrared features are extracted and fused into the infrared hyperspectral image to enhance its features. Using the fused infrared features, HOG and depth features are extracted from the hyperspectral image, and a tracker is constructed based on the DCF framework. Feature-level and decision-level fusion are implemented in the tracking process, ultimately achieving hyperspectral target tracking. This enables fast and effective tracking of targets in hyperspectral image sequences against complex backgrounds. Attached Figure Description

[0039] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein...

[0040] Figure 1 A flowchart illustrating a feature-enhanced infrared hyperspectral video target tracking method according to a preferred embodiment of the present invention;

[0041] Figure 2 for Figure 1 The t-th frame of the feature-enhanced infrared hyperspectral video target tracking method described above;

[0042] Figure 3 for Figure 1 The region of interest in the feature-enhanced infrared hyperspectral video target tracking method described above;

[0043] Figure 4 for Figure 1 The enhanced image obtained by the feature-enhanced infrared hyperspectral video target tracking method described above;

[0044] Figure 5 for Figure 1 Feature response map of the feature-enhanced infrared hyperspectral video target tracking method;

[0045] Figure 6 for Figure 1 The prediction results of the feature-enhanced infrared hyperspectral video target tracking method described above;

[0046] Figure 7 A block diagram of a feature-enhanced infrared hyperspectral video target tracking system provided in a preferred embodiment of the present invention;

[0047] Explanation of reference numerals in the accompanying drawings: 100, Feature-enhanced infrared hyperspectral video target tracking system; 101, Preprocessing module; 102, Preprocessing module; 103, Feature extraction module; 104, Discriminant correlation filter tracker generation module; 105, Feature response map generation module; 106, Prediction module; 107, Prediction module; 108, Update module; 109, Judgment module. Detailed Implementation

[0048] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0049] Firstly, referring to Figure 1 As shown, a preferred embodiment of the present invention provides a feature-enhanced infrared hyperspectral video target tracking method, comprising:

[0050] S1, acquire the original infrared hyperspectral image, preprocess the original infrared hyperspectral image to obtain the preprocessed image;

[0051] Specifically, this application first performs a band selection preprocessing operation, selecting the three bands with the highest information entropy in the original infrared hyperspectral image to form a new image, removing information redundancy; wherein, the original infrared hyperspectral image includes 15 bands;

[0052] In one specific implementation, the band selection steps include:

[0053] S1-1, Read in the first frame of the original infrared hyperspectral image sequence;

[0054] S1-2, calculate the information entropy of each band pixel value of the read-in frame image;

[0055] S1-3, Determine if the current image is the last frame:

[0056] If so, the band selection process ends, and the 15 bands are arranged in descending order of information entropy. The three bands with the highest information entropy are selected to form a new image, which is then used as the preprocessed image for subsequent tracking. In this embodiment, the t-th frame image used for subsequent tracking is as follows: Figure 2 As shown;

[0057] If not, then read in the next frame image and continue executing S1-2.

[0058] S2, load the first frame image from the preprocessed image as the current frame image, extract the region of interest based on the current frame image, perform infrared feature enhancement on the region of interest, and obtain the enhanced image; the specific steps include:

[0059] S2-1, Load the first frame image from the preprocessed image as the current frame image.

[0060] S2-2, obtain the coordinates of the top left corner of the target box in the current frame image and the size of the target box, and select the region of interest based on the target box;

[0061] Specifically, S2-2 includes:

[0062] S2-2-1, use a rectangle to outline the target image region to be tracked in the first frame of the preprocessed image sequence as the target bounding box, and use the centroid position of the target image region to be tracked as the initial position of the target to be tracked, so as to obtain the center point of the target and the height and width of the target bounding box;

[0063] S2-2-2 determines the area of ​​the target bounding box relative to the entire image size. If the target area is larger than 0.05 of the entire image, it is a large target; otherwise, it is a small target. For large targets, less contextual information is usually needed, so only a small amount of padding needs to be added around the large target. The padding parameter is set to 1. For small targets, more padding needs to be added around the target to better consider the target's contextual information. The padding parameter is set to 1.8.

[0064] S2-2-3, The background region is used for feature extraction. The method for calculating the size of the background region is as follows, for both large and small targets:

[0065] w bg =w+(w+h)*P / 2 (1),

[0066] h bg =h+(w+h)*P / 2 (2),

[0067] Among them, w bg h is the width of the background area bg w is the height of the background area, h is the width of the first frame rectangle, and P is the padding parameter for the background area.

[0068] S2-2-4, the foreground region is a smaller region relative to the target region, used to avoid mislabeling. `inner_padding` is defined as the inner padding ratio, with an initial value of 0.2. The foreground region size is calculated as follows:

[0069] w fg =w-(w+h)*P i / twenty three),

[0070] h fg=h-(w+h)*P i / twenty four),

[0071] Among them, w fg h is the width of the foreground region fg Let w be the height of the foreground region, w be the width of the first frame rectangle, h be the height of the first frame rectangle, and P be the height of the foreground region. i Fill the foreground region with the parameter inner_padding.

[0072] S2-2-5 performs saturation processing on the foreground and background areas to ensure they do not exceed the image range.

[0073] S2-2-6, Set other initialization parameters.

[0074] S2-3: Extract the first band of the current frame image, and use a quadtree to decompose the current frame image into blocks to obtain multiple block images;

[0075] Specifically, the preprocessed image is a three-band hyperspectral image. The information entropy of this three-band hyperspectral image is ranked from highest to lowest and placed into bands one, two, and three. The image of the first band (with the highest information entropy) is selected for infrared feature extraction. First, the single-band infrared image is initialized as a quadtree structure block. If the difference between the maximum and minimum gray values ​​of the current block is greater than a set threshold, it is further divided into four smaller image blocks. This process continues until no image blocks are further divided or the image block size reaches the minimum allowable size. In quadtree decomposition, the gray value subtraction threshold and the image block size are two important parameters. The threshold is used to determine whether an image block in the quadtree needs further decomposition. Generally, a smaller threshold is chosen to suppress the influence of noise on an image block. We set T... threashold =20 is used to eliminate noise. The initial minimum size should be set as large as possible to include most infrared targets, so we set this parameter to the maximum size of the infrared targets.

[0076] S2-4, perform erosion operation on each block image, and use Bezier interpolation to reconstruct the background of the current frame image to obtain the reconstructed background image;

[0077] Specifically, the positions of 16 points are uniformly sampled from each patch in the quadtree structure as control points for the Bézier surface. The pixel value of the control point is set to the local minimum at that location, because the quadtree structure can determine that most vertices are located on the infrared background, but a few points are located inside the infrared target, and the local minimum pixel value usually points to the background. Later, Bézier can reconstruct the Bézier surface for each patch by interpolating based on the corresponding 16 control points.

[0078] Q(u,v)=UMPM T V T (5),

[0079] Where (u,v) represents an interpolation point, the position of which is indicated by the interpolation ratio between 0 and 1; U and V represent variable interpolation coefficients, which are related to (u,v); M represents a constant interpolation coefficient matrix; and P represents a 4*4 control point matrix consisting of 16 control points.

[0080] Because each Bezier surface uses different control points, a block effect occurs when the surfaces are directly combined and stitched together. Therefore, a Gaussian smoothing filter is used to smooth the infrared background.

[0081] I FBIR =I BIR *g(s,ρ) (6),

[0082] Among them, I FBIR I represents the smoothed infrared background. BIR This represents the reconstructed infrared background; s represents the size of the Gaussian filter, and ρ represents the sigma of the Gaussian filter.

[0083] Thus, this application obtains a reconstructed infrared background image I that is natural and smooth. FBIR .

[0084] S2-5, Subtract the reconstructed background image from the current frame image to obtain a single-dimensional image of the infrared target enhancement;

[0085] Specifically, by subtracting the reconstructed background image from the current frame image, the infrared brightness features are extracted using the following formula:

[0086] BF = max(I IR -I FBIR ,0) (7),

[0087] Where BF represents the extracted brightness feature, and I IR For the original single-band infrared image, I FBIR The reconstructed and smoothed infrared background image, To find the maximum value between a and b.

[0088] However, subtracting the reconstructed infrared background image from the infrared image of the first band may result in the loss of too many infrared features. Therefore, this application multiplies the estimated background by an appropriate suppression ratio α and subtracts the brightness features to reduce a large amount of redundant information while retaining useful infrared brightness features.

[0089] BF new =BF-α*max(I three –IIR ,0) (8),

[0090] Among them, BF new The improved infrared features are shown in Figure 1, while BF represents the original infrared features. α represents the background suppression ratio, and I represents the infrared features extracted before improvement. three For three-band infrared hyperspectral images, I IR This is the original infrared image in the single band.

[0091] However, this method sometimes produces overexposed fused images. To address this issue, this application uses the following formula to further compress bright infrared features:

[0092] BF Final =β*BF new (9),

[0093] Among them, BF Final Represented as the compressed infrared bright feature, β represents the feature suppression ratio, calculated using the following formula:

[0094]

[0095] Among them, Aver 0.5 This represents the average value of the first 0.5% of pixels in the image after feature fusion. This means taking the minimum value between a and b. This average value will be used to adjust the pixel value range of the image after feature fusion to ensure that the grayscale range of the fusion result is within a reasonable range and to avoid overexposure.

[0096] S2-6, fuse the one-dimensional image with the current frame image to obtain the enhanced image.

[0097] Specifically, this application adds the final infrared brightness features to the original three-band infrared hyperspectral image to enhance the infrared features of the original image, resulting in an enhanced image:

[0098] I Fusion =BF Final +I three (11),

[0099] Among them, I Fusion This is an enhanced infrared image obtained by fusing extracted infrared features.

[0100] S3, extract the HOG features of the enhanced image as the first feature; train the preprocessed image according to the preset file to obtain the trained image, extract the Conv4-4 convolutional layer features of the trained image as the second feature, and extract the Conv5-4 convolutional layer features of the trained image as the third feature.

[0101] Specifically, in S2, the region of interest (ROI) extracted from the current frame image is called the ROI. In this embodiment, the ROI is as follows: Figure 3 As shown, the infrared features of the ROI are extracted as described in S2 above, and the resulting fused image with enhanced infrared features is used as the enhanced image. In this embodiment, the region of interest after extracting infrared features is as follows: Figure 4 As shown in S3, this application uses a three-band infrared hyperspectral image after band selection for feature extraction. The HOG feature of the current frame image is extracted as the first feature, and two depth features of the image are extracted as the second and third features. The specific steps include:

[0102] The HOG (Histogram of Oriented Gradient) feature is extracted from the enhanced image and used as the first feature.

[0103] We used a VGG-19 pre-trained convolutional neural network to extract depth features from the ROI of the current frame image, and used the fourth layer (conv4-4) of the fourth convolutional group and the fourth layer (conv5-4) of the fifth convolutional group in the VGG-19 network as the second and third features.

[0104] S4. Based on the first feature, the second feature, and the third feature, a discriminant correlation filter tracker is obtained using a Gaussian response template.

[0105] Specifically, this application uses a DCF-based tracker framework for tracking, and the specific steps include:

[0106] The DCF-based tracker is trained using an image patch of size M*N, centered on the target. All cyclic displacement patches x(m,n)∈{0,1,…,M-1}*{0,1,…,N-1} are generated as training samples, where the Gaussian label y(m,n) is the cyclic displacement. The filter ω is trained by minimizing the regression error, as shown in the following formula:

[0107]

[0108] Where ω is the Gaussian filter, λ is the regularization parameter, and X is the data matrix that links all cyclic shifts;

[0109] The filter solution for the d-th (d∈{1,2,…,D}} channel is defined as:

[0110]

[0111] Where ⊙ represents element-wise product; The Discrete Fourier Transform (DFT) of a vector; express The complex conjugate of , where λ is the regularization parameter;

[0112] To avoid boundary effects, the Hann window is applied to the feature matrix;

[0113] As the number of frames increases, the filter... molecules and denominator The update is as follows:

[0114]

[0115]

[0116]

[0117] Where η is the learning rate and t is the index of the current frame.

[0118] S5, based on the first feature, second feature, third feature, and discriminant correlation filter tracker, a first feature response map, a second feature response map, and a third feature response map are obtained; by linearly combining the first feature response map, the second feature response map, and the third feature response map, a fourth feature response map, a fifth feature response map, a sixth feature response map, and a seventh feature response map are obtained; wherein, each feature response map serves as a tracking discriminant expert; in this embodiment, the feature response maps are as follows: Figure 5 As shown;

[0119] Specifically, the Gaussian response map of the feature map image z is calculated using the following formula:

[0120]

[0121] Among them, F -1 R represents the inverse Fourier transform, and R represents the Gaussian response plots of the HOG feature and the two deep features. The filter is the result of the Fourier transform of the d-th band. It is the conjugate of the feature map image after the Fourier transform of the d-th band.

[0122] This application obtains a first feature response map based on a first feature and a discriminant correlation filter tracker, a second feature response map based on a second feature and a discriminant correlation filter tracker, and a third feature response map based on a third feature and a discriminant correlation filter tracker; this application obtains a fourth feature response map by linearly combining the first and second feature response maps, a fifth feature response map by linearly combining the first and third feature response maps, a sixth feature response map by linearly combining the second and third feature response maps, and a seventh feature response map by linearly combining the first, second, and third feature response maps;

[0123] In one specific implementation, the Gaussian response map corresponding to each feature is used as a tracking discriminant expert, and multiple experts track the target in parallel. HOG features can capture low-level details of the target, while deep features represent the semantic information of the target; different features have their own advantages. Furthermore, considering the role of feature-level fusion, this application linearly combines the tracking discriminant experts corresponding to three features into a new expert. The tracking discriminant expert corresponding to the HOG feature is called the low-level expert (denoted by `low`), the tracking discriminant expert corresponding to the conv4-4 feature is called the middle-level expert (denoted by `middle`), and the tracking discriminant expert corresponding to the conv5-4 feature is called the high-level expert (denoted by `high`). Each expert is a Gaussian response map of size 50*50. The seven linearly combined tracking discriminant experts are as follows:

[0124] expertⅠ: low → junior expert

[0125] expertⅡ: middle-→ intermediate expert

[0126] expertⅢ: high-→Senior Expert

[0127] expertⅣ: middle + 0.5 * low → a linear combination of intermediate and low-level experts

[0128] expert V: high + 0.5 * low → a linear combination of high-level and low-level experts

[0129] expertⅥ: high + 0.5 * middle → a linear combination of advanced and intermediate experts

[0130] expertⅦ: high + 0.5 * middle + 0.02 * low → a linear combination of high, middle, and low-level experts

[0131] Multiple experts track the target in parallel, each generating different cues (bounding boxes). The evaluation between different bounding boxes reflects the degree of consistency among the experts, called pairwise evaluation. Most experts in the ensemble can stably track the target, and a good expert should be consistent with other experts. The overlap rate of bounding boxes from different experts is calculated, and at frame t, the overlap ratio between expert i and expert j is O. t (E i E j The calculation is as follows:

[0132]

[0133] Among them, O t (E i E jThe area (a∩b) represents the overlap ratio between expert i and expert j at frame t, and the area (a∪b) represents the intersection area of ​​the bounding box rectangles of a and b. It is a bounding box that contains only the target state.

[0134] To reduce the gap between low and high overlap rates, the following nonlinear Gaussian function is used. The formula is as follows:

[0135]

[0136] Average overlap rate It reveals the consistency of the trajectories of expert i with those of other experts, where k represents the number of experts.

[0137] S6, the positions of the maximum values ​​in the first, second, third, fourth, fifth, sixth, and seventh feature response maps are used as the predicted location of the tracking target center;

[0138] S7. Based on the paired evaluation and self-evaluation scores of the tracking discrimination experts, obtain the robustness score of each tracking discrimination expert for the current frame image tracking, and select the tracking discrimination expert with the highest robustness score to track the current frame image;

[0139] Specifically, generally speaking, the pairwise comparison scores between two experts should be time-stable. Therefore, the degree of fluctuation in the overlap rate over a short period of time (e.g., 5 frames) reveals the stability of the overlap assessment between Ei and other experts, which is determined by the equation:

[0140]

[0141] The decision is made, where k represents the number of experts. τ∈[t-Δt+1,t].

[0142] To avoid performance fluctuations and further consider time stability, this application introduces an increasing sequence, W = {ρ 0 ,ρ 1 ,……,ρ Δt-1 Variance is calculated as follows:

[0143]

[0144]

[0145] Among them, W τ Let N represent the τ-t+Δt-th element in sequence W, where N is N=∑ τ W τThe normalization factor, the pairwise robustness of expert i in frame t is defined as:

[0146]

[0147] Here, ε is a small constant, avoiding the case of zero denominator, and larger... This means that the expert is more consistent with other experts.

[0148] The smoothness of the expert trajectory reflects the reliability of the tracking results to some extent, by measuring the previous bounding boxes of each expert. With the current bounding box The Euclidean distance between the displacements is given by the following formula:

[0149]

[0150] in, Represents bounding box The center.

[0151] To avoid performance fluctuations and further consider time stability, an increasing sequence is introduced, W = {ρ 0 ,ρ 1 ,……,ρ Δt-1 In frame t, the trajectory fluctuation of expert i is given by the following formula:

[0152]

[0153] in, The average length of the bounding box's width and height is provided by expert i: The smoothness of the expert trajectory reflects the reliability of the tracking results to some extent. The formula for the self-assessment robustness score is:

[0154]

[0155] In frame t, the tracking box defined by the expert with the highest final robustness score is selected as the tracking expert for the current frame, and the point with the highest response value on the expert response map is taken as the center position of the target. The final robustness score is calculated as follows:

[0156]

[0157] It is worth noting that because this application uses an ROI-sharing strategy, certain methods must be employed to prevent expert corruption, specifically including:

[0158] PSR is widely used in DCF to quantify the reliability of tracking results. PSR is defined as:

[0159] P = (R)max -m) / δ (28),

[0160] Where Rmax is the maximum confidence level, m is the mean of the response, and δ is the standard deviation.

[0161] The average PSR for different features is calculated as follows:

[0162]

[0163] The above formula is used to evaluate the tracking result at the t-th time, where, These represent the PSR values ​​of the high, medium, and low level response maps for frame t, respectively.

[0164] When the target is occluded or severely deformed, the expert's average robustness score The reliability score will decrease significantly, which can be seen as a disagreement among multiple experts when faced with unreliable samples. By comprehensively considering the average PSR score and the average expert robustness score, a new combined reliability score is proposed:

[0165]

[0166] It can effectively identify unreliable samples and better evaluate the quality of current tracking results.

[0167] Since DCF achieves target tracking by learning target and background information, simply discarding unreliable samples is unreasonable, given the current reliability score S. t Significantly lower than the average reliability score in the past At that time, the learning rate η is changed to:

[0168]

[0169] Where c is the standard learning rate of DCF, α is the reliability threshold, and β penalizes samples with low confidence to protect experts from corruption.

[0170] S8, update the discrimination correlation filter tracker and adaptively update the tracking discrimination expert;

[0171] S9, determine whether the current frame image is the last frame image of the preprocessed image:

[0172] If so, the target tracking task is completed, and the prediction result is obtained; in this embodiment, the prediction result is as follows: Figure 6 As shown;

[0173] If not, the next frame of the preprocessed image is loaded as the current frame image, and the region of interest is extracted based on the current frame image. Infrared feature enhancement is then performed on the region of interest to obtain the enhanced image.

[0174] This application preprocesses a near-infrared dataset of 15 bands to avoid information redundancy. The preprocessing method involves selecting the three bands with the highest information entropy to form a new image, a process known as band selection. The first frame image after band selection is loaded, and the coordinates of the top-left corner and the size of the target bounding box are obtained. The size of this target bounding box is appropriately enlarged to select the region of interest (ROI). Infrared feature enhancement is then performed on the ROI. Specifically, the first band of the preprocessed image is extracted, the image is decomposed into blocks using a quadtree, erosion is performed on each block, the background is reconstructed using Bezier interpolation, and the original image is subtracted from the reconstructed background image to obtain a single-dimensional image of the infrared target enhancement. This single-dimensional image is then fused with the original three-dimensional image to achieve the infrared feature enhancement effect. After infrared feature enhancement, the HOG features of the current frame image are extracted as the first feature. The preprocessed infrared hyperspectral image is trained using a CNN-19 pre-trained weight file. Conv4-4 and Conv5-4 convolutional layer features are extracted as depth features, and a discriminative correlation filter is created using a Gaussian response template combined with the extracted features. The response map is obtained by multiplying the ROI by the discriminant filter. Different response maps are formed by linearly combining different feature response maps, which are called discriminant experts. Each expert tracks the target individually. There are paired evaluations and self-evaluations between experts. The linear combination of the robustness of the two results in the final robustness of the current expert in tracking the target. In each frame, the expert with the highest final robustness is selected for tracking. The filters for each feature are updated in real time as the number of frames increases, and the experts also have certain adaptive strategies to prevent expert corruption. This invention combines infrared feature enhancement, tracker feature-level fusion methods, and decision-level fusion methods to achieve effective target tracking in infrared hyperspectral image sequences with complex backgrounds.

[0175] Secondly, such as Figure 7 As shown, this embodiment provides a feature-enhanced infrared hyperspectral video target tracking system 100, including:

[0176] Preprocessing module 101 is used to acquire the original infrared hyperspectral image, preprocess the original infrared hyperspectral image, and obtain the preprocessed image;

[0177] The infrared feature enhancement module 102 is used to load the first frame image in the preprocessed image as the current frame image, extract the region of interest based on the current frame image, perform infrared feature enhancement operation on the region of interest, and obtain the enhanced image.

[0178] The feature extraction module 103 is used to extract the HOG features of the enhanced image as the first feature; train the preprocessed image according to the preset file to obtain the trained image, extract the Conv4-4 convolutional layer features of the trained image as the second feature, and extract the Conv5-4 convolutional layer features of the trained image as the third feature.

[0179] Discriminant correlation filter tracker generation module 104 is used to obtain a discriminant correlation filter tracker based on the first feature, the second feature and the third feature using a Gaussian response template;

[0180] The feature response map generation module 105 is used to obtain a first feature response map, a second feature response map, and a third feature response map based on a first feature, a second feature, a third feature, and a discriminant correlation filter tracker; and to obtain a fourth feature response map, a fifth feature response map, a sixth feature response map, and a seventh feature response map by linearly combining the first feature response map, the second feature response map, and the third feature response map; wherein each feature response map serves as a tracking discriminant expert;

[0181] The prediction module 106 is used to determine the location of the center of the tracked target as the position of the maximum value in the first feature response map, the second feature response map, the third feature response map, the fourth feature response map, the fifth feature response map, the sixth feature response map, and the seventh feature response map.

[0182] The tracking module 107 is used to obtain the robustness score of each tracking expert for tracking the current frame image based on the paired evaluation and self-evaluation scores of the tracking experts, and select the tracking expert with the highest robustness score to track the current frame image.

[0183] The update module 108 is used to update the discrimination correlation filter tracker and adaptively update the tracking discrimination expert;

[0184] The judgment module 109 is used to determine whether the current frame image is the last frame image of the preprocessed image:

[0185] If so, the target tracking task is completed and the prediction result is obtained;

[0186] If not, the next frame of the preprocessed image is loaded as the current frame image, and the region of interest is extracted based on the current frame image. Infrared feature enhancement is then performed on the region of interest to obtain the enhanced image.

[0187] The feature-enhanced infrared hyperspectral video target tracking system provided in this embodiment, since it is used to implement the steps of the feature-enhanced infrared hyperspectral video target tracking method provided in the first aspect embodiment of the present invention, thus possesses all the technical effects of the feature-enhanced infrared hyperspectral video target tracking method, which will not be elaborated here.

[0188] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed, implements the feature-enhanced infrared hyperspectral video target tracking method described in any of the first aspects above.

[0189] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0190] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0191] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0192] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0193] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A feature-enhanced infrared hyperspectral video target tracking method, characterized in that: an original infrared hyperspectral image is acquired, and the original infrared hyperspectral image is preprocessed to obtain a preprocessed image; a first frame image in the preprocessed image is loaded as a current frame image, a region of interest is extracted according to the current frame image, and an infrared feature enhancement operation is performed on the region of interest to obtain an enhanced image; wherein the specific steps of obtaining the enhanced image comprise: S2-2, obtaining the upper left corner coordinates of a target frame of the current frame image and the size of the target frame, and selecting the region of interest according to the target frame; wherein the steps of S2-2 are: S2-2-1, outlining a to-be-tracked target image region in the first frame image as a target frame, taking the centroid position of the to-be-tracked target image region as the initial position of the to-be-tracked target, and obtaining the center point of the tracking target and the height and width of the target frame; S2-2-2, judging the area of the region where the target frame is located in the overall first frame image; if the area of the region where the target frame is located is greater than 0.05 of the overall first frame image, it is determined that the target frame is a large target; otherwise, it is determined that the target frame is a small target; S2-2-3, calculating a background region according to the large target or the small target; S2-2-4, calculating a foreground region according to the large target or the small target; S2-2-5, performing saturation processing on the background region and the foreground region; S2-3, the preprocessed image is a three-band hyperspectral image, the highest information entropy of the three-band hyperspectral image is placed in the first band, the first band of the current frame image is extracted, the current frame image is decomposed into blocks using a quadtree, and a plurality of block images are obtained; S2-4, performing an erosion operation on each block image, and reconstructing the background of the current frame image using a Bezier interpolation method to obtain a reconstructed background image; S2-5, subtracting the reconstructed background image from the current frame image to obtain an infrared target-enhanced single-dimensional image; S2-6, fusing the single-dimensional image with the current frame image to obtain the enhanced image; extracting HOG features of the enhanced image as a first feature; training the preprocessed image according to a preset file to obtain a trained image, extracting Conv4-4 convolution layer features of the trained image as a second feature, and extracting Conv5-4 convolution layer features of the trained image as a third feature; obtaining a discriminant correlation filter using a Gaussian response template according to the first feature, the second feature, and the third feature; obtaining a first feature response map, a second feature response map, and a third feature response map according to the first feature, the second feature, the third feature, and the discriminant correlation filter; obtaining a fourth feature response map, a fifth feature response map, a sixth feature response map, and a seventh feature response map by linearly combining the first feature response map, the second feature response map, and the third feature response map; wherein each feature response map serves as a tracking discriminant expert. The maximum value position in the first feature response map, the second feature response map, the third feature response map, the fourth feature response map, the fifth feature response map, the sixth feature response map and the seventh feature response map is used as a predicted tracking target center position; A robustness score of each tracking discrimination expert in tracking the current frame image is obtained according to the pairing evaluation and self-evaluation scores of the tracking discrimination experts, and a tracking discrimination expert with the highest robustness score is selected to track the current frame image; The discrimination correlation filter tracker is updated, and the tracking discrimination expert is adaptively updated; wherein the discrimination correlation filter tracker is represented as: ， ， ， ， ， in, It is a Gaussian filter. For regularization parameters, To link the data matrix of all cyclic shifts, For the first Filters on the channel, , ⊙ represents element-wise product; For the Discrete Fourier Transform (DFT) of a vector. for The complex conjugate, For regularization parameters, For filter molecules, For filter The denominator is η, which is the learning rate; It is judged whether the current frame image is the last frame image in the preprocessed images or not: If yes, the target tracking task is completed, and a prediction result is obtained; If no, the next frame image in the preprocessed images is loaded as the current frame image, and a region of interest is extracted according to the current frame image, and an infrared feature enhancement operation is performed on the region of interest to obtain an enhanced image.

2. The feature-enhanced infrared hyperspectral video target tracking method according to claim 1, wherein: The specific steps of the preprocessing include: The original infrared hyperspectral image is subjected to band selection to obtain the preprocessed image; The specific steps of the band selection include: selecting three bands with the largest information entropy in the original infrared hyperspectral image to form the preprocessed image.

3. The feature-enhanced infrared hyper-spectral video target tracking method of claim 1, wherein, The specific steps of obtaining the first feature response map, the second feature response map and the third feature response map according to the first feature, the second feature, the third feature and the discrimination correlation filter tracker include: The first feature response map is obtained according to the first feature and the discrimination correlation filter tracker; The second feature response map is obtained according to the second feature and the discrimination correlation filter tracker; The third feature response map is obtained according to the third feature and the discrimination correlation filter tracker.

4. The feature-enhanced infrared hyper-spectral video target tracking method of claim 3, wherein, The specific steps of obtaining the fourth feature response map, the fifth feature response map, the sixth feature response map and the seventh feature response map by linearly combining the first feature response map, the second feature response map and the third feature response map include: The fourth feature response map is obtained by linearly combining the first feature response map and the second feature response map; The fifth feature response map is obtained by linearly combining the first feature response map and the third feature response map; The sixth feature response map is obtained by linearly combining the second feature response map and the third feature response map; The seventh feature response map is obtained by linearly combining the first feature response map, the second feature response map and the third feature response map.

5. The feature-enhanced infrared hyperspectral video target tracking method according to claim 1, wherein: The Bezier interpolation method is represented as: ， wherein is an interpolation point, , is a variable interpolation coefficient; is a constant interpolation coefficient matrix, is a 4*4 control point matrix consisting of 16 control points.

6. The feature-enhanced infrared hyperspectral video target tracking method according to claim 5, wherein: The first feature response map, the second feature response map and the third feature response map are obtained according to the following formula: ， wherein, is the inverse Fourier transform, is the Gaussian response map of the HOG feature and two depth features, is the d-th is the filter after the waveband Fourier transform, is the conjugate of the d-th feature map image after the waveband Fourier transform.

7. A feature enhanced infrared hyperspectral video target tracking system for implementing the feature enhanced infrared hyperspectral video target tracking method of any one of claims 1 to 6, characterized in that, The preprocessing module is configured to acquire an original infrared hyperspectral image, preprocess the original infrared hyperspectral image, and obtain a preprocessed image; ​ The infrared feature enhancement module is configured to load a first frame image in the preprocessed image as a current frame image, extract a region of interest from the current frame image, and perform an infrared feature enhancement operation on the region of interest to obtain an enhanced image. The process of obtaining the enhanced image includes: S2-2-1, frame the preprocessed first frame image to be tracked target image region as a target frame, and take the centroid position of the to-be-tracked target image region as the initial position of the to-be-tracked target to obtain the center point of the tracking target and the height and width of the target frame; S2-2-2, judge the area of the target frame in the entire first frame image; if the area of the target frame is greater than 0.05 of the entire first frame image, it is determined to be a large target; otherwise, it is determined to be a small target; S2-2-3, calculate the background region according to the large target or the small target; S2-2-4, calculate the foreground region according to the large target or the small target; S2-2-5, perform saturation processing on the background region and the foreground region; The feature extraction module is configured to extract HOG features of the enhanced image as a first feature, train the preprocessed image according to a preset file to obtain a trained image, extract Conv4-4 convolution layer features of the trained image as a second feature, and extract Conv5-4 convolution layer features of the trained image as a third feature; The discriminative correlation filter tracker generation module is configured to use a Gaussian response template to obtain a discriminative correlation filter tracker according to the first feature, the second feature, and the third feature; The feature response map generation module is configured to obtain a first feature response map, a second feature response map, and a third feature response map according to the first feature, the second feature, the third feature, and the discriminative correlation filter tracker; obtain a fourth feature response map, a fifth feature response map, a sixth feature response map, and a seventh feature response map by linearly combining the first feature response map, the second feature response map, and the third feature response map; wherein each feature response map serves as a tracking discrimination expert; The prediction module is configured to take the maximum value position in the first feature response map, the second feature response map, the third feature response map, the fourth feature response map, the fifth feature response map, the sixth feature response map, and the seventh feature response map as the predicted tracking target center position; The tracking module is configured to obtain a robustness score of each tracking discrimination expert in tracking the current frame image according to the pairing evaluation and self-evaluation scores of the tracking discrimination experts, and select the tracking discrimination expert with the highest robustness score to track the current frame image; The update module is configured to update the discriminative correlation filter tracker and adaptively update the tracking discrimination experts; The judgment module is configured to judge whether the current frame image is the last frame image of the preprocessed image: If yes, the target tracking task is completed, and a prediction result is obtained; If not, the next frame image in the preprocessed image is loaded as the current frame image, and the region of interest is extracted from the current frame image, and an infrared feature enhancement operation is performed on the region of interest to obtain an enhanced image.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the enhanced infrared hyperspectral video target tracking method according to any one of claims 1 to 6.

Images