An online updating target tracking method based on mask information

Abstract

This invention discloses an online target tracking method based on mask information, relating to computer vision and target tracking technology. The method mainly consists of a feature extraction module, a target localization module, a target segmentation module, and a refinement module. To maintain the real-time performance of the tracking algorithm, this method selects the first three layers of ResNet-18 as the backbone feature network. The refinement module takes the image itself as input and directly extracts features from the image using a fine-grained module, aiming to preserve the target's detailed information to the greatest extent. The network first extracts features from the target image, inputs the obtained features into the target localization module to obtain robust target localization information, and updates the network parameters of this module online every 25 frames. Then, the localization information is input into the target segmentation module to obtain the target's predicted mask. Finally, the fine-grained module upsamples and restores the mask to the input size to obtain the target tracking result.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision-target tracking technology, and specifically is a method for accurately tracking a non-rigidly deformable target in a complex scene. Background Art

[0002] In today's society where the level of science and technology is changing with each passing day, artificial intelligence technology has entered the homes of ordinary people. Everyone has more or less felt the convenience brought by artificial intelligence technology. With the vigorous development of artificial intelligence technology, computer vision technology has become an indispensable part of it. Computer vision technology refers to a processing technology that uses intelligent terminals, cameras and other image acquisition devices to simulate biological vision. The main task is to analyze and process the collected images or videos, thereby completing some tasks that are inconvenient for humans to complete, thereby improving the quality of life of the entire society.

[0003] In recent years, with the rise of artificial intelligence technology and image processing technology, visual target tracking technology has played an important role in large-scale intelligent systems. With the development of computer computing power, machine learning, deep learning and other technologies have also been widely used in the tracking field, injecting new vitality into the field of target tracking.

[0004] At present, the target tracking algorithm based on the twin network is developing rapidly, and has achieved certain results in tracking speed and accuracy. For example, twin network trackers such as SiamFC and SiamRPN have achieved certain performance improvements but have always had limitations. Since the focus has always been on studying classification networks with strong performance, the problem of accurate target state estimation is easily overlooked. First, the above network uses a shallow AlexNet network to extract convolutional features for similarity discrimination, and the feature representation ability learned by the shallow network is insufficient; secondly, the twin network tracking algorithm adopts an online training and offline tracking mechanism, and does not update the model parameters during the tracking process, resulting in weak generalization of the algorithm and weak ability to cope with complex scenes, especially the interference of similar objects; finally, as the network of the existing tracking algorithm continues to deepen, the feature map receptive field is large, excessive attention is paid to semantic information, and there is a lack of fine-grained information, resulting in poor results for the tracker for non-rigid deformation of the target. For example, the target is elongated, rotated, deformed, or a person with his hands spread out. In these cases, the previous algorithms have poor results. Summary of the invention

[0005] The purpose of the present invention is to provide an online updated target tracking method based on mask information, which is committed to solving the problems of tracking drift and inaccurate target state estimation caused by complex scenes and non-rigid deformation of targets in actual scenes, and further improving the performance of target tracking algorithms.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] An online updating target tracking method based on mask information comprises the following steps:

[0008] S1: Target area selection: Select the target area to be tracked from the initial frame of the video sequence, mainly by human-computer interaction or automatic extraction by target detection method;

[0009] S2: Preparation of training data set: This method uses the training set in the Youtube-VOS data set for training. First, the image pairs and their corresponding real segmentation masks are uniformly sampled within the range of 80 frames in the same video sequence for training. The initial sampling size is 384×384 pixels, and feature extraction is performed within 4 times the target size.

[0010] S3: Construction of target positioning module: Figure 1 As shown in the figure, the extracted features are firstly reduced in dimension using 1×1 convolution, and then input into the positioning module. Since the function of the target positioning module is to provide the precise center position of the target, the goal of this module is to hope that it will not be affected by the size and shape of the target, and it can enhance the foreground and background discrimination by minimizing the classification error. The target positioning module of this algorithm is a two-layer full convolution layer, defined as:

[0011] f(x;w)=φ2(w2*φ1(w1*x))

[0012] In the above formula, x is the feature map of the input image obtained by the backbone network, w = {w1, w2} is the parameter of the two-layer convolutional network, φ is the activation function, PELU is used here, and * represents the standard 64-channel convolution operation, so the objective function of the target positioning module learning is defined as follows, similar to L2 regularization:

[0013]

[0014] Among them, w is the parameter of the convolution kernel learned online, y j Represents the confidence of the target position. If the traditional stochastic gradient descent is used, it will be difficult to meet the real-time requirements. Therefore, this module adopts a second-order optimization algorithm to unify the expression of the regular term and the residual term. Let:

[0015]

[0016]

[0017] Among them, j from 1 to m represents each frame of the input image, and the k values ​​1 and 2 here represent two layers of convolution respectively, so the Loss function can be written in a simpler form:

[0018] L(w)=||γ(w)|| 2

[0019] According to the principle of Gauss-Newton approximation to ignore the second-order differential, firstly, the first-order Taylor expansion of γ(w) at w+Δw can be obtained:

[0020] γ(w+Δw)≈γ w +J w Δw

[0021] Then, substituting into L(w+Δw) gives the following formula:

[0022]

[0023] Among them, J w Represents the Jacobian matrix of γ with respect to w. Now all we need to do is to update w with Δw. We can take the derivative of L(w+Δw)-L(w) with respect to Δw to get the analytical solution of Δw, that is, because Since it is positive definite, the conjugate gradient descent can be used to solve it, thereby realizing the fast online update strategy of the classification module, and finally outputting a 24×24×1 positioning score response map, where the position with the highest score is the center position of the tracked target.

[0024] S4: Construct target segmentation module: Figure 2 As shown in the figure, the target segmentation module is responsible for predicting the binary mask information of the target. Since the target mask information is introduced for target tracking, the task becomes a pixel-by-pixel tracking task, which will inevitably introduce a large amount of background clutter and easily generate false positive predictions, making it difficult to identify the target's foreground and background information. To address this problem, the SiamMask algorithm uses a hard spatial constraint method to alleviate it, that is, during tracking, a bounding box is first generated for the target, and then the mask information of the target is predicted within the bounding box, and then a more refined bounding box is fitted. However, this method is flawed. If the bounding box constraint generated during the tracking process is wrong, the subsequent target mask information prediction will be wrong, which will limit the accuracy that the algorithm can achieve. Therefore, this paper proposes a soft spatial constraint method, which guides the network to perform precise segmentation by establishing a mask information space probability map. First, the mask information of the target initial frame is extracted and downsampled and transformed, denoted as M∈R 1×(24×24) , and then the initial frame feature T0 and the current frame feature T obtained from the backbone network are n Transform the dimensions to T0∈R (24×24)×64 , Tn ∈R 64×(24×24) , the generated spatial probability map is recorded as S. Its specific operation can be expressed by the formula:

[0025] S=M⊙T0⊙T n

[0026] Specifically, the initial frame mask information M is dot-producted with the initial frame feature T0 to obtain the foreground and background information of the initial frame feature, and then combined with the current frame feature T n Perform a dot product operation, then use the Softmax activation function to obtain a spatial probability map of the target's mask information, and finally output a 24*24 target binary mask information map.

[0027] After the target segmentation network is constructed, the target segmentation network is trained using the training set generated in step S2. In order to further optimize the network training, the Adam optimizer is used in the training phase. The initial learning rate is set to 0.001, and the learning rate is decayed every 10 iterations with a decay amplitude of 0.1. The BatchSize size is 32, and a total of 40 iterations are trained, with 1000 training iterations per iteration. The device used this time is NVIDIA GeForce GTX 1080, and the loss function used is the cross entropy between the predicted target mask and the true target mask.

[0028] S5: Construction of fine-grained information module: This paper chooses to use 6 3×3 convolutions and 1×1 convolutions to construct this module. The number of channels is set to 64. The final output feature map is 1 / 8 of the original image size. Finally, the output feature map is added to the obtained segmentation feature map and then restored to the original size of the image 384*384 through upsampling. Finally, the target segmentation result map containing fine-grained information is obtained.

[0029] S6: Generation of target bounding box: The output representation of the target tracking task is a bounding box that wraps the target. The output of this network is a segmentation probability map of the target, so the network output needs to be simply processed. First, the predicted target segmentation probability map is binarized with a threshold of 0.5 to obtain the binary segmentation mask of the target. Then, the largest connected area in the mask is retained, and the ellipse is fitted to its contour using the least squares method. Finally, a rotated target bounding box consisting of the center, major and minor axes of the ellipse is generated.

[0030] S7: Network online update: After successfully tracking the target, the target positioning information obtained in step S6 is input into the target positioning module, and the network parameters are updated according to the update algorithm in S6. The network is set to be updated online every 25 frames, and training samples are generated using data enhancement technology, including cropping, scale transformation, etc.

[0031] The beneficial effects of the present invention are:

[0032] This online update target tracking algorithm based on mask information analyzes the low algorithm performance caused by the rapid deformation, non-rigid deformation and background interference of the target, and divides the entire target tracking task into two sub-tasks: target positioning and segmentation tracking. In the positioning module, a new online update strategy is proposed to update the parameters of the positioning network, so as to robustly locate the target to adapt to the ever-changing complex scenes; in the segmentation and tracking module, an accurate target mask information segmentation sub-network is constructed to accurately obtain the target mask information while maintaining robust positioning of the target to solve the problem of non-rigid deformation of the target. At the same time, in order to overcome the interference of background clutter in the segmentation module, it is used to enhance the foreground and background discrimination ability of the segmentation network; finally, the target mask information obtained by the segmentation module is used to perform ellipse fitting to generate an accurate bounding box that wraps the target. The performance of this algorithm is compared with Figure 3 shown. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic structural diagram of the positioning module of the present invention.

[0034] Figure 2 Schematic diagram of the segmentation module structure of the present invention.

[0035] Figure 3 Performance comparison chart of the tracker of the present invention. DETAILED DESCRIPTION

[0036] The method of the present invention can be used in various fields where single target tracking is applicable, mainly including intelligent security, automatic driving, drone cruise and virtual reality technology which has been popular in recent years. Take virtual reality technology as an example: virtual reality is an emerging technology that can create a virtual world, allowing users to immerse themselves in a simulated environment and interact with the surrounding environment in an immersive way. In order to allow users to move freely in a virtual environment to increase the flexibility of interaction, it is very important to capture the user's position and direction information in real time and accurately. Generally, motion capture is performed using camera-based image processing and positioning, which places more stringent requirements on the target tracking algorithm, requiring both accuracy and robustness. The method proposed in the present invention can be used for implementation. Specifically, an algorithm model of the present invention is first constructed, which is mainly composed of a target positioning module, a target tracking module and a fine-grained module, and then the commonly used object targets are manually annotated to obtain a training data set, and then the network is trained using the Adam gradient descent method using the data set. After convergence, the network has accurate tracking capabilities. In the tracking process, the target object information of the initial frame is first input, and after feature extraction, it is sent to the target positioning module to obtain robust target positioning information and rough appearance state information. Then the obtained features are sent to the target segmentation module to obtain the target prediction mask information and the accurate target appearance state. The target prediction mask information and positioning information are sent to the fine-grained module together, and up-sampled and fused to restore the original input size and obtain an accurate target segmentation binary mask. Finally, the rotating rectangular frame that wraps the target is obtained through ellipse fitting, so as to better adapt to the stringent requirements of motion capture in virtual reality technology and solve the problems of tracking drift and target deformation in the tracking process.

[0037] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. The method can be implemented by programming on any platform, whether it is a computer or various embedded devices. There is no restriction on the programming language, and it can be implemented in mainstream languages.

Claims

1. An online updating target tracking method based on mask information comprises the following steps: S1: Target area selection: Select the target area to be tracked from the initial frame of the video sequence, mainly by human-computer interaction or automatic extraction by target detection method; S2: Preparation of training dataset: This method uses the training set in the Youtube-VOS dataset for training. First, it trains by uniformly sampling image pairs and their corresponding true segmentation masks within the range of 80 frames of the same video sequence. The initial sampling size is 384×384 pixels, and feature extraction is performed within a range of 4 times the target size. S3: Construction of the target positioning module: First, the extracted features are reduced in dimension using 1×1 convolution, and then input into the positioning module. The target positioning module of this method is a two-layer full convolution layer, defined as: f(x;w)=φ2(w2*φ1(w1*x)) In the above formula, x is the feature map of the input image obtained by the backbone network, w = {w1, w2} is the parameter of the two-layer convolutional network, φ is the activation function, PELU is used here, and * represents the standard 64-channel convolution operation, so the objective function of the target positioning module learning is defined as follows, similar to L2 regularization: Among them, w is the parameter of the convolution kernel learned online, y j Represents the confidence of the target position. Using a second-order optimization algorithm, first unify the expression of the regular term and the residual term, and let: Among them, j from 1 to m represents each frame of the input image, and the k values ​​1 and 2 here represent two layers of convolution respectively, so the Loss function can be written in a simpler form: L(w)=||γ(w)|| 2 According to the principle of Gauss-Newton approximation to ignore the second-order differential, firstly, the first-order Taylor expansion of γ(w) at w+Δw can be obtained: γ(w+Δw)≈γ w +J w Δw Then, substituting into L(w+Δw) gives the following formula: Among them, J w Represents the Jacobian matrix of γ with respect to w. Now all we need to do is to update w with Δw. We can take the derivative of L(w+Δw)-L(w) with respect to Δw to get the analytical solution of Δw, that is, because Since it is positive definite, the conjugate gradient descent method can be used to solve it, thereby realizing the fast online update strategy of the classification module. The final output is a 24×24×1 positioning score response map, in which the position with the highest score is the center position of the tracked target. S4: Construct target segmentation module: The target segmentation module is responsible for predicting the binary mask information of the target and establishing a mask information space probability map to guide the network for accurate segmentation. First, the mask information of the target initial frame is extracted and downsampled, denoted as M∈R 1×(24×24) , and then the initial frame feature T0 and the current frame feature T obtained from the backbone network are n Transform the dimensions to T0∈R (24×24)×64 , T n ∈R 64×(24×24) , the generated spatial probability map is recorded as S, and its specific operation can be expressed by the formula: S=M⊙T0⊙T n Specifically, the initial frame mask information M is dot-producted with the initial frame feature T0 to obtain the foreground and background information of the initial frame feature, and then combined with the current frame feature T n Perform a dot product operation, then use the Softmax activation function to obtain a spatial probability map of the target's mask information, and finally output a 24*24 target binary mask information map; After the target segmentation network is constructed, the training set generated in step S2 is used to train the target segmentation network. In order to further optimize the network training, the Adam optimizer is used in the training phase. The initial learning rate is set to 0.001, and the learning rate is decayed every 10 iterations with a decay amplitude of 0.

1. The BatchSize size is 32, and a total of 40 iterations are trained, with 1000 training iterations per iteration. The device used this time is NVIDIA GeForce GTX 1080, and the loss function used is the cross entropy between the predicted target mask and the true target mask. S5: Construction of fine-grained information module: This application chooses to use 6 3×3 convolutions and 1×1 convolutions to construct this module. The number of channels is set to 64. The final output feature map is 1 / 8 of the original image size. Finally, the output feature map is added to the obtained segmentation feature map and then restored to the original size of the image 384*384 through upsampling, and finally the target segmentation result map containing fine-grained information is obtained; S6: Generation of target bounding box: The output representation of the target tracking task is a bounding box that wraps the target. The output of this network is a segmentation probability map of the target. First, the predicted target segmentation probability map is binarized with a threshold of 0.5 to obtain the binary segmentation mask of the target. Then, the largest connected area in the mask is retained, and the least squares method is used to fit an ellipse to its contour. Finally, a rotated target bounding box consisting of the center, major and minor axes of the ellipse is generated. S7: Network online update: After successfully tracking the target, the target positioning information obtained in step S6 is input into the target positioning module, and the network parameters are updated according to the update algorithm in S6. The network is set to be updated online every 25 frames, and samples are generated using data enhancement technology.

Images