A multi-layer deep feature fusion and adaptive re-detection target tracking method

Abstract

The present application relates to the technical field of image processing, and more particularly to a multi-level deep feature fusion and adaptive re-detection target tracking method. The method comprises the following steps: multi-level feature extraction is performed to extract template features and search features; the template features and the search features are subjected to feature mean concatenation fusion to construct feature information of different levels; the fused feature information is merged into a region candidate network to predict a next frame target frame; an adaptive multi-window re-detection method is used to judge target occlusion, and then a nearby window is constructed for re-detection to realize anti-occlusion target tracking of a SiamRPN tracker. Through the setting of feature extraction-feature fusion-target frame prediction-occlusion judgment-re-detection, the present application can effectively ensure high timeliness, accurately extract feature information, accurately track a target, and improve the robustness of tracking.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a target tracking method with multi-level deep feature fusion and adaptive re-detection. BACKGROUND

[0002] Video target tracking technology is to predict and locate the target in the subsequent video by giving the target coordinate information of the current frame. The engineering application prospect is very extensive. Traditional target tracking methods usually rely on artificial features, color models and classifiers to establish a robust feature representation, such as optical flow method, particle filter, etc. The performance of these traditional algorithms cannot meet the requirements of practical applications. The target tracking method based on twin network gradually becomes the mainstream method in the field of target tracking due to its simple network structure, high accuracy and fast real-time performance.

[0003] The idea of twin network target tracking is to use two identical or similar networks to extract the features of the template branch and the search branch, and measure the similarity of the features to determine the position of the tracking frame. The SiamRPN tracker adds a region candidate network in the field of target detection based on the traditional twin network target tracking, and pushes the twin network target tracking technology to a peak. After the twin network extracts the template features and search features, the features are sent to the region candidate network to obtain the recommended frame. SiamRPN combines target tracking and target detection, effectively improving the accuracy of the twin network target tracking technology.

[0004] At present, the target tracking technology based on twin network has high tracking accuracy under the condition of no occlusion, but part of the feature information is lost under the occlusion scene, resulting in target loss. Moreover, most of the monitoring data sets belong to far scene small targets, which further increases the difficulty of feature extraction, resulting in reduced tracking accuracy. At the same time, in order to improve the timeliness, the twin network based target tracking technology does not take the entire frame as the search area, but takes the target frame of the last frame as the center, and clips the current frame into a search area with a size of 255x255x3. When the object is completely occluded by consecutive frames and appears again, it may exceed the current frame clipping area, i.e. it cannot be searched in the current search area, resulting in target loss. Therefore, under the occlusion scene, how to ensure high timeliness while accurately extracting feature information and accurately tracking the target is a problem that technicians in the field need to solve. SUMMARY

[0005] The purpose of the present application is to provide a target tracking method with multi-level deep feature fusion and adaptive re-detection, which solves the technical problem of reduced tracking accuracy in the occlusion scene in the prior art.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0007] The application provides a multi-level deep feature fusion and adaptive re-detection target tracking method, comprising the following steps:

[0008] S1. initializing a video sequence and performing multi-level feature extraction thereon to extract template features and search features;

[0009] S2. performing feature mean value cascade fusion on the template features and the search features to construct feature information at different levels;

[0010] S3. feeding the fused feature information into a region candidate network to predict a target frame;

[0011] S4. judging target occlusion by using an adaptive multi-window re-detection method, then constructing a nearby window to perform re-detection, and realizing anti-occlusion target tracking of a SiamRPN tracker.

[0012] Further, in S1, the initialization of the video sequence and the multi-level feature extraction thereon comprise:

[0013] taking the center coordinates of the target bounding box of the ground_truth as a reference point, a template image with a size of 127*127*3 is cropped on the template frame;

[0014] for the subsequent frames of the template frame, a search frame is taken as a reference point, and a search region image with a size of 255*255*3 is cropped with the center coordinates of the target prediction bounding box of the previous frame as a reference point;

[0015] the step of the ResNet50 network is reduced to 8 pixels, and the original feature extraction network of the SiamRPN tracker is replaced to construct a deep feature extraction network;

[0016] the template features and the search features of the template frame and the search frame in the video are extracted by using the deep feature extraction network.

[0017] Further, in S2, the feature mean value cascade fusion of the template features and the search features comprises:

[0018] the feature information extracted from the three residual blocks of the ResNet50 network is denoted as 、 、 ;

[0019] the three layers of features are fused by using a pixel-by-pixel addition operation, and the fusion mode is:

[0020]

[0021] wherein m is 0.33;

[0022] The fused template feature map is 7x7x256, 7x7 is the feature size, and 256 is the channel number;

[0023] The fused search feature map is 31x31x256, 31x31 is the feature size, and 256 is the channel number.

[0024] Further, in S3, the fused feature information is merged into the region candidate network to predict the target frame, including:

[0025] The template feature and the search feature are respectively merged into the classification branch and the regression branch of the region candidate network, and convolution operation is performed on the target feature and the search feature respectively to calculate the correlation.

[0026] For the classification branch, the correlation calculation formula is:

[0027]

[0028] For the regression branch, the correlation calculation formula is:

[0029]

[0030] wherein, and represent the output results of the classification branch and the regression branch respectively, represents the cross-correlation operation, , represent the output template feature and the search feature respectively.

[0031] The classification branch finally contains 2k channel vectors, and the regression branch finally contains 4k channel vectors.

[0032] Further, after the region candidate network, the feature mapping of the classification branch and the regression branch is represented as a point set:

[0033]

[0034]

[0035] wherein, the foreground and background information output by the classification branch is used to determine whether the target exists, that is, the possible target frame is output; the regression branch further corrects the target frame. is the classification, , , , represent the center coordinates and the length and width of the corrected target frame respectively.

[0036] Further, in S4, the adaptive multi-window re-detection method is used to determine the target occlusion, including:

[0037] The average value of the average peak correlation energy of the response map of the first four frames of images , as a shielding or not judging standard;

[0038] The calculation method is:

[0039]

[0040] Among them, The average peak correlation energy of each frame is calculated as:

[0041]

[0042]

[0043]

[0044] Among them, , Corresponding to the highest, the lowest position on the response, Is the image block obtained according to the last frame, Is the target position, Is the classifier parameter, Indicates that the target width is , the response value at High;

[0045] The current frame When less than the average value of the last four frames , The response map oscillates, and it is considered that the target is shielded; otherwise, it is considered that the target is not shielded.

[0046] Further, in S4, a nearby window is constructed for re-detection, realizing the anti-occlusion target tracking of SiamRPN tracker,

[0047] When it is considered that the target is not shielded, it is considered that the current search area is correct, and the target frame obtained by the target frame obtained by the target frame is the next frame target center frame.

[0048] When it is considered that the target is shielded:

[0049] Eight other nearby windows are constructed around the current detection window, and different weights are assigned to the eight nearby windows. The eight nearby windows are detected according to the weight size; the selected window is marked, and the new target frame is obtained by re-detecting in the selected search window. The average peak correlation energy corresponding to the new target frame is calculated and recorded as ;

[0050] Compare With Size, if less than Then compare The size corresponding to the previous detection window, and update To a larger value, that is: ; ;

[0051] Repeat the above steps until Greater than The tracking frame corresponding to the detection window is taken as the next frame target center frame; if all windows are detected, Greater than all less than The tracking frame corresponding to the largest The updated value is selected as the next frame target center frame.

[0052] Further, the eight adjacent windows are assigned different weights, and the eight adjacent windows are detected according to the weight size, including:

[0053] The center window weight is higher than the weight of the four windows above and below and left and right, and the weight of the four windows above and below and left and right is higher than the weight of the four windows diagonally above and below.

[0054] Preferentially select the window with high weight for detection, and randomly select the window between the windows with the same weight for detection.

[0055] The application also provides a multi-level deep feature fusion and adaptive re-detection twin network anti-occlusion target tracking system, the system comprising:

[0056] An acquisition module is used to acquire and initialize frame information;

[0057] A feature extraction module is used to acquire different layer features of a template frame and a search frame through a backbone network;

[0058] A feature fusion module is used to construct multi-level fusion features;

[0059] A target frame prediction module is used to predict the position of a next frame target frame through a candidate network;

[0060] An occlusion judgment module is used to judge whether there is occlusion;

[0061] A re-detection module is used to construct adjacent window re-detection until the average peak value correlation energy of the current window exceeds the average value of the average peak value correlation energy of four consecutive frames or reaches the maximum value, and the target tracking result of the to-be-tracked sequence is obtained.

[0062] The application has at least the following beneficial effects:

[0063] This invention, through a feature extraction-feature fusion-target bounding box prediction-occlusion judgment-re-detection setup, effectively ensures high timeliness while accurately extracting feature information, accurately tracking targets, and improving tracking robustness. Experimental verification shows that the anti-occlusion target tracking method of this invention achieves a tracking accuracy of 0.89 and a tracking success rate of 0.67 on the public dataset OTB100. Compared to the original tracker SiamRPN, it improves accuracy and success rate by 5% and 4% respectively, and is highly competitive with other advanced tracking methods. Attached Figure Description

[0064] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a flowchart of the Siamese network anti-occlusion target tracking method with multi-level deep feature fusion and adaptive re-detection disclosed in this invention;

[0066] Figure 2 This is a network structure diagram of a multi-level feature fusion module.

[0067] Figure 3 Flowchart of the re-detection module;

[0068] Figure 4 The construction graph for adjacent windows;

[0069] Figure 5 This is a structural diagram of the target tracking system of the present invention. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0071] Example 1

[0072] The present invention will now be described in further detail with reference to the accompanying drawings.

[0073] like Figure 1 The flowchart below shows the multi-level deep feature fusion and adaptive re-detection twin network anti-occlusion target tracking method disclosed in this invention, which includes the following steps:

[0074] S1: Construct a deep feature extraction network, initialize template frames and search frames, and extract multi-layer feature information;

[0075] In specific implementation, in step S1, the video sequence is first initialized, and then multi-level feature extraction is performed, specifically as follows:

[0076] S100: Using the center coordinates of the target bounding box that achieves the given ground truth as the reference point, crop a template image of size 127×127×3 from the template frame (the first frame image is denoted as the template frame in target tracking); subsequent frames are all used as search frames, and the center coordinates of the target predicted bounding box of the previous frame are used as the reference point to crop a search region image of size 255×255×3.

[0077] S101: The original backbone network of the SiamRPN tracker is replaced with a deeper ResNet50 network. In order to make the replaced network more adaptable to the situation of small targets in distant scenes that may occur in reality, the stride of the original ResNet50 network is reduced to 8 pixels to facilitate the subsequent comparison of Siam networks.

[0078] S2: A method of feature mean concatenation and fusion is proposed to fuse the template features and search features extracted from the last three residual blocks of the ResNet50 network respectively, thereby constructing feature information at different levels;

[0079] like Figure 2 The diagram shows the structure of a multi-level feature fusion network. In specific implementation, step S2 includes the following steps:

[0080] S200: The low-level features, such as color and texture, extracted from the template features by the third residual block of the improved ResNet50 network are denoted as S200. The features extracted from the fourth residual block are denoted as The semantic features extracted from the fifth residual block are denoted as .

[0081] This invention proposes a method for feature mean cascading fusion based on the ResNet50 backbone network, which is more conducive to the extraction of features of small targets in distant scenes.

[0082] S201: The three-layer features are fused using a pixel-wise addition operation, with the resulting template feature map size being 7×7×256, where 7×7 represents the feature size and 256 represents the number of channels. The fusion method is as follows:

[0083]

[0084] Here, m is set to 0.33.

[0085] S202: Similar to S200 and S201, the pixel-by-pixel addition is performed on the three-layer search features to obtain the mean value cascade fusion, and The difference is that the size of the fused template feature map is 31x31x256, 31x31 is the feature size, and 256 is the channel number.

[0086] S3: Feature input region candidate network, the next frame target box is predicted through the region candidate network;

[0087] In the implementation, in step S3, the following specific steps are included:

[0088] S300: The template feature and the search feature are fed into the classification branch and the regression branch, the target feature and the search feature are subjected to convolution operation by the two branches, and the correlation is calculated; and the calculation results are respectively represented as follows:

[0089]

[0090]

[0091] wherein, and represent the output results of the classification branch and the regression branch respectively, represents the cross-correlation operation, , represent the output template feature and the search feature respectively. k represents that after the anchor mechanism, each pixel point maps back to the original region, which can generate k candidate boxes in which the target may exist. Therefore, the classification branch finally contains 2k channel vectors, and the regression branch finally contains 4k channel vectors.

[0092] S301: After the network, the classification and regression features are mapped and represented as point sets:

[0093]

[0094]

[0095] wherein, the foreground and background information output by the classification branch is used to determine whether the target exists, that is, the possible target box is output; and the regression branch further corrects the target box, is the classification, , , , represent the center coordinates and the length and width of the corrected target box respectively.

[0096] S4: The adaptive multi-window re-detection method is proposed, the adjacent window is adaptively constructed after the target occlusion is determined, and the re-detection is performed, so as to realize the SiamRPN anti-occlusion target tracking method.

[0097] In implementation, in step S4, the following specific steps are included:

[0098] S400: Take the oscillation degree of the response map of the first four frames of images, i.e., the average value of the average peak correlation energy, as the judgment criterion for whether the target is occluded.

[0099]

[0100] wherein the average peak correlation energy of each frame is expressed as:

[0101]

[0102]

[0103]

[0104] wherein, , respectively correspond to the responses at the highest and lowest positions, is the image block obtained according to the previous frame, is the target position, is the classifier parameter, represents the response value at the target position with a width of and a height of .

[0105] S401: Calculate the current frame When the average value of the four consecutive frames is less than , it is considered that the response map is oscillating and the target is occluded. Otherwise, it is considered that the target is not occluded.

[0106] S402: When the target is not occluded, it is considered that the current search area is correct, and the target center frame obtained by the region candidate network is the target center frame of the next frame. At this time, the flow chart is as shown in Figure 3 .

[0107] S403: When the target is occluded, eight adjacent windows are constructed around the current detection window, as shown in Figure 4 (b). As shown in Table 1, different weights are assigned to the eight adjacent windows to reduce the search time. The weight of the center window is higher than that of the four windows above, below, left and right, which is higher than that of the upper diagonal and lower diagonal.

[0108] Table 1: Window weight correspondence table

[0109] Window Weight 5 3.0 2, 4, 6, 8 2.0 1, 3, 5, 7 1.0

[0110] S404: preferentially select a window with a high weight for detection, and randomly select between windows with the same weight. Mark the selected window, and re-detect in the selected search window to obtain a new target frame, calculate the average peak correlation energy corresponding to the new target frame, and mark it as .

[0111] S405: compare and , if is smaller, then compare and the size corresponding to the previous detection window , and update to the larger value, that is: .

[0112] Repeat the re-detection operation of S404-S405 until is greater than . If all windows are detected and are smaller than , select the tracking frame corresponding to the largest after updating.

[0113] In specific implementation, verify the multi-level deep feature fusion and adaptive re-detection twin network anti-occlusion target tracking method disclosed in the application, and compare and verify the tracking accuracy and success rate on the open source dataset OTB100. Table 2 shows the accuracy and success rate comparison results of the method of the application and other five advanced methods under different properties.

[0114] From the positioning error in Table 2, it can be seen that the multi-level deep feature fusion and adaptive re-detection twin network anti-occlusion target tracking method described in this paper has a tracking accuracy of 0.89 and a tracking success rate of 0.67 on the public dataset OTB100. Compared with the original tracker SiamRPN, the accuracy and success rate are improved by 5% and 4% respectively, and compared with other advanced tracking methods, it has strong competitiveness.

[0115] Table 2: Tracking accuracy and success rate on OTB100 dataset

[0116] Method Accuracy Success rate SiamFC 0.70 0.52 CFNet 0.77 0.58 SiamDW 0.82 0.62 SiamRPN 0.84 0.63 DaSiamRPN 0.88 0.66 The method herein 0.89 0.67

[0117] Embodiment 2

[0118] The application also provides an anti-occlusion target tracking system, as shown in Figure 5 , comprising:

[0119] An acquisition module is configured to acquire and initialize frame information.

[0120] A feature extraction module is configured to acquire different layer features of a template frame and a search frame through a backbone network.

[0121] A feature fusion module is configured to construct multi-level fusion features;

[0122] A target frame prediction module is configured to predict a next frame target frame position through a candidate network;

[0123] A shielding judgment module is configured to judge whether shielding exists or not;

[0124] A re-detection module is configured to construct a nearby window re-detection until an average peak value correlation energy of a current window exceeds an average value of a continuous four frame average peak value correlation energy or reaches a maximum value, and obtain a target tracking result of the to-be-tracked sequence.

[0125] The basic principle, main features and advantages of the present application are shown and described above. It should be understood by those skilled in the art that the present application is not limited by the above examples, and the above examples and descriptions in the specification are only the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection required by the present application is defined by the appended claims and their equivalents.

Claims

1. A target tracking method with multi-level deep feature fusion and adaptive re-detection, characterized in that, Includes the following steps: S1. Initialize the video sequence and perform multi-level feature extraction on it to extract template features and search features; S2. Perform feature mean concatenation and fusion on the template features and search features to construct feature information at different levels; S3. Integrate the fused feature information into the region candidate network to predict the target box in the next frame; S4. The target occlusion is determined by the adaptive multi-window re-detection method, and then the nearest window is constructed for re-detection to realize the anti-occlusion target tracking of SiamRPN tracker; In S1, the initialization of the video sequence and the multi-level feature extraction thereof include: Using the center coordinates of the target bounding box of ground_truth as a reference point, a template image with a size of 127×127×3 is cropped from the template frame; All subsequent frames of the template frame are used as search frames, and the center coordinates of the target prediction bounding box of the previous frame are used as reference points to crop out a search region image with a size of 255×255×3. The stride of the ResNet50 network was reduced to 8 pixels, and the original feature extraction network of the SiamRPN tracker was replaced to construct a deep feature extraction network. Template features and search features of template frames and search frames in the video are extracted using a deep feature extraction network. In S2, the template features and search features are subjected to feature mean concatenation and fusion, including: The feature information extracted from the three residual blocks of the ResNet50 network is denoted as follows: , , ; The three-layer features are fused by mean concatenation using a pixel-wise addition operation. The fusion method is as follows: ; Where m is 0.33, The low-level features extracted from the template features by the third residual block. Features extracted for the fourth residual block Let the semantic features extracted from the fifth residual block be denoted as follows; The fused template feature map is 7×7×256, where 7×7 is the feature size and 256 is the number of channels; The fused search feature map is 31×31×256, where 31×31 is the feature size and 256 is the number of channels; In S4, target occlusion is determined using an adaptive multi-window re-detection method, including: The average peak-to-peak energy of the oscillations in the response maps of the first four frames is taken as the average value. This serves as the criterion for determining whether something is obscured. The calculation method is as follows: ; in, The average peak correlation energy (PSE) for each frame is calculated as follows: ; ; ; in, , These correspond to the responses at the highest and lowest positions, respectively. It is an image patch obtained from the previous frame. It is the target location. These are classifier parameters. Indicates the target width is Gao Wei Response value; Calculate the current frame When less than four consecutive frames average If the response graph oscillates, the target is considered to be occluded; otherwise, the target is considered not to be occluded. In S4, a neighboring window is constructed for re-detection to achieve occlusion-resistant target tracking of the SiamRPN tracker. When the target is considered to be unobstructed, the current search area is considered to be correct, and the target box obtained by the region candidate network is the target center box in the next frame. When the target is considered to be obscured: Eight neighboring windows are constructed around the current detection window, and different weights are assigned to each of the eight neighboring windows. Detection is performed on each of the eight neighboring windows according to its weight. The selected window is marked, and re-detection is performed within the selected search window to obtain a new target box. The average peak correlation energy corresponding to the new target box is calculated and denoted as . ; Compare and Size, if smaller Then the comparison continues. Corresponding to the previous detection window The size, and Update to the larger value, i.e.: ; Repeat the above steps until... Greater than If the detection window is detected, the tracking box corresponding to that window will be used as the target center box in the next frame; if all windows have been detected, All less than Then select the one with the largest updated value. The corresponding tracking box serves as the target center box in the next frame.

2. The target tracking method with multi-level deep feature fusion and adaptive re-detection according to claim 1, characterized in that, In S3, the fused feature information is fed into the region candidate network to predict the target bounding box for the next frame, including: The template features and search features are respectively fed into the classification branch and regression branch of the region candidate network, and convolution operations are performed on the target features and search features respectively to calculate the correlation. For the classification branch, the correlation calculation formula is: ; For the regression branch, the correlation calculation formula is: ; in, and These represent the output results of the classification branch and the regression branch, respectively. This indicates a cross-correlation operation. , These represent the output template features and search features, respectively. The classification branch ultimately contains 2,000 channel vectors, and the regression branch ultimately contains 4,000 channel vectors.

3. The target tracking method with multi-level deep feature fusion and adaptive re-detection according to claim 2, characterized in that, After passing through the region candidate network, the feature mapping between the classification branch and the regression branch is represented as a set of points: ; ; Specifically, the classification branch outputs foreground and background information to determine whether a target exists, i.e., it outputs possible target boxes; the regression branch further refines the target boxes. It's a classification. , This indicates the center coordinates of the corrected target bounding box. , This indicates the width and height of the target bounding box to be corrected.

4. The target tracking method with multi-level deep feature fusion and adaptive re-detection according to claim 1, characterized in that, The step of assigning different weights to the eight neighboring windows and detecting the eight neighboring windows according to their weights includes: The center window has a higher weight than the four windows above, below, left, and right, and the four windows above, below, left, and right have a higher weight than the four windows diagonally above and diagonally below. Window with higher weight is selected for detection first, and among windows with the same weight, a window is randomly selected for detection.

Images