A method for target tracking against occlusion

Abstract

The application discloses a target tracking anti-occlusion method, which is used for solving target tracking in the case of occlusion. Firstly, a tracking target in two adjacent images in an image sequence is selected, and then the target is input into a full convolution feature extraction neural network for feature extraction. Then, a convolution neural network is used for position tracking of the tracking target. Finally, the target is re-detected until a tracking result is output. The application adopts a multi-method result strategy fusion mode to solve the problem of tracking loss after target occlusion. The convolution neural network is used as a tracking network main body structure, and a fast and efficient tracking confidence parameter is combined to determine whether the tracked target is occluded. Once the target is occluded, the current image is quickly detected based on historical information. Based on the detection mechanism, the position of the re-detected target is used as an initial coordinate, and target tracking is restarted.

Description

Technical Field

[0001] This invention belongs to the field of image target tracking, and specifically relates to a method for target tracking with anti-occlusion. Background Technology

[0002] In the field of image target tracking, when continuously tracking a target of interest, situations such as occlusion can occur, causing the target to be lost. The target visual tracking process generally involves selecting the target object in the initial frame of a given video and obtaining the target's initial state (such as position and size), and then automatically estimating the state of the moving object in subsequent frames.

[0003] Despite decades of research into visual tracking, target tracking remains a challenging problem due to various factors such as partial occlusion, rapid and sudden object movement, changes in lighting, and large variations in viewpoint and pose. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing an anti-occlusion method for tracking targets that are occluded.

[0005] The technical solution adopted by this invention to solve its technical problem is: a method for target tracking with anti-occlusion, comprising the following steps:

[0006] S1, continuously track and capture images of the target object, select the target object in two adjacent frames of the image sequence, select a rectangular area and scale it to a fixed size, and input it into a fully convolutional feature extraction neural network for target feature extraction;

[0007] S2 uses multiple convolutional layers to perform convolution operations on the target image, followed by processing with a non-linear activation layer. Each convolutional layer integrates multi-scale convolutions. The entire network does not use pooling, but instead replaces the pooling operation used in the fully convolutional feature extraction neural network with a stride of 2 convolution operations, and saves the output as the feature template of the current target.

[0008] The fully convolutional feature extraction neural network consists of five feature layers:

[0009] The first, second, and third feature layers each contain three sub-layers. The first sub-layer has a 5×5 kernel size and a stride of 2. The second sub-layer has four different kernel sizes: 1×1, 3×3, 5×5, and 7×7, with a stride of 1. These four kernel sizes are convolved with the results of the previous sub-layer in parallel, and the results of the four scales are then concatenated. The third sub-layer uses LeakyReLU as the non-linear activation function.

[0010] The fourth feature layer consists of two sub-layers. The first sub-layer has four convolutional kernel sizes: 1×1, 3×3, 5×5, and 7×7, with a stride of 1. The four scale convolutional kernels are convolved with the results of the previous sub-layer in parallel, and the results of the four scales are concatenated. The second sub-layer uses LeakyReLU as a non-linear activation function.

[0011] The fifth feature layer includes a sub-layer with a kernel size of 3×3 and a stride of 1, which performs a convolution operation with the result of the previous sub-layer;

[0012] S3, Calculate the tracking confidence parameter

[0013] S31, the n channels output by the feature extraction network are fused according to the following weighting method: calculate the maximum value of the n output channels of the feature extraction network, and n channels will have n values. Normalize these n maximum values, and use these n normalized values ​​as weights to perform weighting operations on each channel. Finally, sum all channels to output a two-dimensional template image: Here we define a gradient trend distribution map of the template image. The calculation process is to calculate the absolute value of the difference between each pixel in the two-dimensional template image and the pixel values ​​of the surrounding 8 neighborhood pixels, select the largest absolute value as the gradient value of the current pixel, and obtain the gradient trend distribution map of each pixel in this two-dimensional template image;

[0014] S32, after inputting a new image, it is first sent to the feature extraction network for feature extraction to obtain a two-dimensional template image. The gradient trend distribution map of the template image of the current image is calculated according to the following steps.

[0015] S33, calculate the mean square error between the current frame and the first frame using the following formula and use it as the tracking confidence level: ,

[0016] I(i,j) represents the gray value at point (i,j) in the current frame, K(i,j) represents the gray value at point (i,j) in the first frame, and m and n correspond to the length and width of the image, respectively.

[0017] S34. Calculate 10 consecutive frames of input image according to steps S31 to S33 to obtain 9 tracking confidence scores to form a confidence score queue and calculate the median value of the queue. If the tracking confidence score obtained in the current frame exceeds ±3 times the median value, it is considered that the target in the current image is occluded and the tracking is no longer reliable.

[0018] S4, Target Re-detection Step

[0019] S41. If the target is occluded, take the target coordinates calculated in the previous frame as the center and select 8 neighborhoods with the same size as the tracking box around them. The size of each neighborhood is the same as the tracking box obtained in the previous frame. Here, the resolution of the entire image and the size of the target tracking box can be used to comprehensively determine whether the 8 neighborhood regions overlap or not.

[0020] S42, the tracking confidence scores obtained from the feature extraction networks of these 8 neighborhoods are compared with the median value of the confidence score queue respectively;

[0021] S43, if the tracking confidence obtained in the current neighborhood is less than ±3 times the value in the confidence queue, then mark it as a matching item;

[0022] S44. Finally, among all the matching items, the neighborhood closest to the confidence queue is selected as the area where the target appears in the current image, and target tracking is restarted.

[0023] The target tracking anti-occlusion method includes a target selection method that allows manual selection of a rectangular area to select the target or automatic selection of the target using a moving object detection algorithm.

[0024] The beneficial effects of this invention are as follows: Based on the target tracking feature extraction network structure, this invention employs multi-layer convolution followed by non-linear activation operations, with multiple sizes of convolution kernels fused in each convolution operation; based on the tracking confidence calculation method, the prediction result of the current frame can be judged. If the target is occluded and the prediction is unreliable, a simple re-detection method is used. This anti-occlusion target tracking method effectively solves the problem of tracking loss after target occlusion. Implementation

[0025] The principles and features of the present invention are described below with reference to embodiments. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0026] The present invention discloses a target tracking anti-occlusion method, comprising the following steps.

[0027] S1, Input.

[0028] S11, continuously track and capture images of the target object. Select the tracking target in two adjacent frames of the image sequence. For a target object that you want to continuously track and capture images of, you can manually select a rectangular area, or automatically select the tracking target after detecting a moving object.

[0029] S12: After selecting a rectangular region, scale the region to a fixed size and input it into a fully convolutional feature extraction neural network to extract features from the target.

[0030] S2, Feature Extraction.

[0031] S21. To improve the efficiency of feature extraction, unlike the common backbone network structure, multiple convolutional layers are used before applying non-linear activation layers. Multi-scale convolutions are fused in each convolutional layer. Pooling is not used in the entire network. Instead, convolutions with a stride of 2 are used to replace pooling operations, and the output is saved as the feature template of the current target.

[0032] S22, the fully convolutional feature extraction neural network consists of five feature layers: The first layer includes three sub-layers. The first sub-layer has a 5×5 kernel size and a stride of 2. The second sub-layer contains four kernel sizes: 1×1, 3×3, 5×5, and 7×7, with a stride of 1. These four kernel sizes are convolved with the results of the previous sub-layer in parallel, and the results of the four scales are concatenated. The third sub-layer uses LeakyReLU as a non-linear activation function. The operations of the second and third feature layers are the same as those of the first feature layer. The fourth feature layer includes two sub-layers. The first sub-layer contains four kernel sizes: 1×1, 3×3, 5×5, and 7×7, with a stride of 1. These four kernel sizes are convolved with the results of the previous sub-layer in parallel, and the results of the four scales are concatenated. The second sub-layer uses LeakyReLU as a non-linear activation function. The fifth feature layer includes one sub-layer, using a 3×3 kernel with a stride of 1, which is convolved with the results of the previous sub-layer.

[0033] S23. After the above operations, the output result is the feature template of the current target, which is then saved.

[0034] S3, calculates the confidence parameter.

[0035] S31, the n channels output by the feature extraction network are fused according to the following weighting method: calculate the maximum value of the n output channels of the feature extraction network, and n channels will have n values. Normalize these n maximum values, and use these n normalized values ​​as weights to perform weighting operations on each channel. Finally, sum all channels to output a two-dimensional template image. Here we define a gradient trend distribution map of the template image. The calculation process is to calculate the absolute value of the difference between each pixel in the two-dimensional template image and the pixel values ​​of the surrounding 8 neighborhood pixels, select the largest absolute value as the gradient value of the current pixel, and obtain the gradient trend distribution map of each pixel in this two-dimensional template image.

[0036] S32, after inputting a new image, it is first sent to the feature extraction network for feature extraction to obtain a two-dimensional template image. The gradient trend distribution map of the template image of the current image is calculated according to the following steps.

[0037] S33, calculate the mean square error between the current frame and the first frame using the following formula:

[0038] ,

[0039] I(i,j) represents the gray value at point (i,j) in the current frame, K(i,j) represents the gray value at point (i,j) in the first frame, m and n correspond to the length and width of the image, respectively, and this mean square error is the tracking confidence.

[0040] S34. Following steps S31 to S33 above, calculate 10 consecutive frames of input images to obtain 9 tracking confidence scores, which can form a confidence score queue.

[0041] Calculate the median value of the queue. If the tracking confidence obtained in the current frame exceeds ±3 times the median value, the target in the current image is considered to be occluded and the tracking is no longer reliable.

[0042] S4, Target Re-detection Step.

[0043] S41. If the target is occluded, select eight neighboring regions centered on the target coordinates calculated in the previous frame. The size of each neighboring region is the same as the tracking box size obtained in the previous frame. Here, the resolution of the entire image and the size of the target tracking box can be used to comprehensively determine whether the eight neighboring regions overlap or not.

[0044] S42, input these 8 neighborhoods into the feature extraction network respectively, and obtain their respective tracking confidence scores, and compare them with the median value of the confidence score queue.

[0045] S43, if the tracking confidence obtained in the current neighborhood is less than ±3 times the value in the confidence queue, then mark it as a conforming item.

[0046] S44. Finally, among all the matching items, the neighborhood closest to the confidence queue is selected as the area where the target appears in the current image, and target tracking is restarted.

[0047] This invention employs a multi-method result strategy fusion approach to address the tracking loss problem caused by target occlusion. It uses a convolutional neural network as the main structure of the tracking network, combined with a fast and efficient tracking confidence parameter to determine whether the tracked target is occluded. Once occlusion is detected, the current image is quickly detected based on historical information. Based on this detection mechanism, the re-detected target position is used as the initial coordinates, and target tracking restarts.

[0048] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for target tracking with anti-occlusion, characterized in that, Includes the following steps: S1, continuously track and capture images of the target object, select the target object in two adjacent frames of the image sequence, select a rectangular area and scale it to a fixed size, and input it into a fully convolutional feature extraction neural network for target feature extraction; S2 uses multiple convolutional layers to perform convolution operations on the target image, followed by processing with a non-linear activation layer. Each convolutional layer integrates multi-scale convolutions, replaces the pooling operation used in the fully convolutional feature extraction neural network with a stride of 2 convolutional operation, and saves the output as the feature template of the current target. The fully convolutional feature extraction neural network consists of five feature layers: The first, second, and third feature layers each contain three sub-layers. The first sub-layer has a 5×5 kernel size and a stride of 2. The second sub-layer has kernel sizes of 1×1, 3×3, 5×5, and 7×7, with a stride of 1. These four kernel sizes are convolved with the results of the previous sub-layer in parallel, and the results of the four scales are then concatenated. The third sub-layer is a LeakyReLU non-linear activation layer. The fourth feature layer consists of two sub-layers; the first sub-layer has convolutional kernels with sizes of 1×1, 3×3, 5×5, and 7×7, with a stride of 1. The four scale convolutional kernels are convolved with the results of the previous sub-layer in parallel, and the results of the four scales are concatenated at the end; the second sub-layer is a LeakyReLU non-linear activation layer. The fifth feature layer includes a sub-layer with a kernel size of 3×3 and a stride of 1, which performs a convolution operation with the result of the previous sub-layer; S3, Calculate the tracking confidence parameter S31, calculate the maximum value of the n output channels of the feature extraction network and normalize these n maximum values. Then, use the n normalized values ​​as weights to perform a weighted operation on each channel. Finally, sum all the channels and output a two-dimensional template image. S32, after inputting a new image, it is first sent to the feature extraction network for feature extraction. The resulting two-dimensional template image is used to calculate the gradient trend distribution map of the current image according to the following steps: calculate the absolute value of the difference between each pixel in the two-dimensional template image and the pixel values ​​of the surrounding 8 pixels, select the largest absolute value as the gradient value of the current pixel, and obtain the gradient trend distribution map of each pixel in this two-dimensional template image. S33, calculate the mean square error between the current frame and the first frame of the rectangular region according to the following formula, and use it as the tracking confidence: ， I(i,j) represents the gray value at point (i,j) in the current frame of the gradient trend distribution map, K(i,j) represents the gray value at point (i,j) in the first frame of the gradient trend distribution map, and m and n correspond to the length and width of the image, respectively. S34. Calculate 10 consecutive frames of input image according to steps S31 to S33 to obtain 9 tracking confidence scores to form a confidence score queue and calculate the median value of the queue. If the tracking confidence score obtained in the current frame exceeds ±3 times the median value, it is considered that the target in the current image is occluded and the tracking is no longer reliable. S4, Target Re-detection: S41, if the target is occluded, then take the target coordinates calculated in the previous frame as the center and select 8 neighboring regions of the same size as the surrounding tracking boxes to determine whether they overlap. S42, the tracking confidence scores obtained from the feature extraction networks of these 8 neighborhoods are compared with the median value of the confidence score queue respectively; S43, if the tracking confidence obtained in the current neighborhood is less than ±3 times the value in the confidence queue, then mark it as a matching item; S44. Finally, among all the matching items, the neighborhood closest to the confidence queue is selected as the area where the target appears in the current image, and target tracking is restarted.

2. The target tracking anti-occlusion method according to claim 1, characterized in that, The methods for selecting the tracking target include manually selecting a rectangular area to select the tracking target or automatically selecting the tracking target using a moving object detection algorithm.

Images