An anchor-free target tracking method based on a twin network

Abstract

The application discloses an Anchor-free target tracking method based on a twin network, which is used in a system comprising a twin network and a contour segmentation network, the twin network is responsible for extracting convolutional neural network (CNN) features of a target template and a search area and calculating a score map of the search area, and outputs an initial rectangular frame, the contour segmentation network is responsible for regression of a target contour, the twin network is composed of a twin backbone network and a classification module, takes an image pair composed of a template image and a search area image as input, and outputs a score map of the search area; the twin backbone network is composed of a template branch and a search area branch with the same structure, and the template branch and the search area branch share network parameters; the application proposes a twin target tracking network, the network does not need to define anchor frames according to prior knowledge, reduces hyperparameters, and can be trained in an end-to-end mode.

Description

Technical Field

[0001] This invention relates to the field of target tracking technology, specifically to an anchor-free target tracking method based on Siamese networks. Background Technology

[0002] Target tracking utilizes image sequences to estimate the position and scale of a target, providing essential information for target behavior analysis and understanding. It is a crucial research area in computer vision. It has broad application prospects in civilian fields such as video surveillance, vehicle navigation, human-computer interaction, and intelligent transportation, as well as military fields such as visual guidance, target localization, and fire control. In recent years, with the development of deep learning technology, target tracking has made significant progress, and numerous methods have been proposed. However, it still faces many challenges, including complex backgrounds, target changes, and speed requirements, and remains a hot research topic in computer vision.

[0003] Due to the motion of the target or imaging device, the target's scale constantly changes during tracking, making target scale estimation a crucial factor affecting target tracking performance. Target scale estimation methods mainly include multi-scale search, anchor-box regression, and anchor-free regression. Tracking methods such as SiamFC use multi-scale search to obtain the target scale, resulting in high computational complexity and inaccurate scale estimation. Methods like SiamRPN utilize anchor-box regression to estimate the target scale, requiring the placement of anchor boxes based on prior knowledge, leading to numerous hyperparameters. Methods like SiamBAN use anchor-free regression to calculate the target scale, reducing hyperparameters but resulting in lower accuracy. Currently, mainstream target tracking methods include correlation filter tracking and Siamese network tracking. These methods use rectangular boxes to represent the target; when the target rotates or deforms, the rectangular representation introduces errors, leading to reduced tracking accuracy. Compared to rectangular boxes, masks provide more accurate target descriptions. Inspired by video target segmentation, some researchers have integrated segmentation into target tracking, proposing segmentation-based target tracking methods, with representative methods including SiamMask, D3S, and AFOD. These methods can simultaneously obtain the target bounding box and the mask, improving tracking accuracy. However, they use region-based segmentation methods, which require, on the one hand, normalizing the target region feature map to the same scale and then classifying the pixels within the target region point by point; on the other hand, post-processing operations such as mask upsampling are required. These two aspects lead to problems such as sensitivity to the tracking bounding box and slow speed. Summary of the Invention

[0004] The purpose of this invention is to provide an anchor-free target tracking method based on Siamese networks, which aims to improve the problems of high computational complexity and mask accuracy limited by the target tracking box when using multi-scale search, anchor box regression or anchorless regression to obtain the target rectangle and perform region segmentation on the target rectangle.

[0005] The present invention is implemented as follows: an anchor-free target tracking method based on Siamese network, which is used in a system including Siamese network and contour segmentation network. The Siamese network is responsible for extracting the convolutional neural network (CNN) features of the target template and search region and calculating the score map of the search region, and outputting the initial bounding box. The contour segmentation network is responsible for the regression of the target contour. Anchor-free target tracking methods include the following steps: Input: Initial target bounding box and image sequences ; Output: target box at time and target outline ; Step 1: Based on the initial target bounding box Crop the template image to a size of 127×127. ; Step 2: Calculate the template image using a twin backbone network Features ; , Step 3: Calculate the search area. ; Step 4: Crop the search area to a size of 255×255. ; Step 5: Calculate the search region using a twin backbone network. Features ; Step 6: Calculate the score graph. ; Step 7: Calculation Target location at all times ; Step 8: Initialization Target scale at any time ; Step 9: Obtain Initial bounding box of target at any given time. ; Step 10: Input the contour segmentation network and obtain Target outline at any time ; Step 11: Based on the target contour Fitting target rectangle at any time ; Step 12: Output Time target box and target outline .

[0006] Preferred, Denotes the initial bounding box of the target at time t, where These are the x-coordinate, y-coordinate, width, and height of the initial bounding box of the target at time t, where... It is the center of the target location.

[0007] Preferred, Represents the coordinates of the target contour at time t, where , and These are the x and y coordinates of the i-th point on the target contour at time t, respectively.

[0008] Preferred, express The minimum bounding rectangle of the target at any given time. These are its x-coordinate, y-coordinate, width, and height, respectively, where ( , ) is the center position of the target.

[0009] x1, y1, w1, h1: represent the x-coordinate, y-coordinate, width, and height of the target bounding box at the initial moment, respectively; : respectively represent The x-coordinate, y-coordinate, width, and height of the target bounding box at any given time.

[0010] Preferred, This represents the coordinate transformation from the feature map to the original map.

[0011] Preferred target contour It is an ordered set of points on an image, where p i = (x i y i () are the coordinates of the point. It is the number of points, points The features are represented as ,in It is a point exist Bilinear interpolation on These are feature maps obtained from the original input image using a CNN. yes The translation-invariant coordinates, where .

[0012] Preferably, the Siamese network consists of two parts: a Siamese backbone network and a classification module. It takes image pairs consisting of a template image and a search region image as input and outputs a score map of the search region. The Siamese backbone network consists of template branches and search region branches with the same structure, and they share network parameters.

[0013] Preferably, if the template image and a certain image block in the search area represent the same target, a larger score value is output; otherwise, a smaller score value is output.

[0014] Preferably, the task of a Siamese network is to learn a metric template image. and search area similarity function ,in" "Indicates cross-relevance, and They are and CNN features.

[0015] Preferably, the process of the contour segmentation network segmenting a rectangular box into a target contour is as follows: First, the midpoints of each side of the rectangular box are connected to obtain a rhombus contour; second, the rhombus contour is used to obtain extreme points through the contour segmentation network, and then an octagonal contour is obtained; finally, the octagonal contour is iteratively transformed into the target contour through the contour segmentation network, with an iteration count of 3.

[0016] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention proposes a twin target tracking network that does not require pre-defining anchor boxes based on prior knowledge, reduces hyperparameters, and can be trained end-to-end.

[0017] 2. This invention implements an anchor-free target tracking method. This method uses region classification and contour regression to model target tracking, and can obtain the target bounding box and contour at the same time, thereby improving the speed and accuracy of target tracking. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the method and system design of the present invention; Figure 2 This is a schematic diagram of the contour segmentation network flowchart of the present invention; Figure 3 These are comparative diagrams of the experimental structures of this invention; Figure 4 This is a performance metric curve of the target tracking method of the present invention on the OTB-2015 dataset. Detailed Implementation

[0019] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0020] The following description, in conjunction with the accompanying drawings and specific embodiments, provides further details: Example 1

[0021] This invention designs an anchor-free target tracking system based on Siamese networks within a deep learning framework, such as... Figure 1 As shown, the system consists of two parts: a Siamese network and a contour segmentation network. The Siamese network is responsible for extracting the convolutional neural network (CNN) features of the target template and the search region, calculating the score map of the search region, and outputting the initial bounding box; the contour segmentation network is used for target contour regression.

[0022] The working principle of a Siamese network is as follows: A Siamese network consists of a Siamese backbone network and a classification module. It takes image pairs consisting of a template image and a search region image as input and outputs a score map of the search region. The Siamese backbone network consists of template branches and search region branches with identical structures, sharing network parameters. If the template image and a certain image patch in the search region represent the same target, a larger score value is output; otherwise, a smaller score value is output. The task of the Siamese network is to learn a metric for the template image. and search area similarity function ,in" "Indicates cross-relevance, and They are and CNN features.

[0023] To obtain the similarity function To use images The neural network is trained offline, and the loss function used for training the neural network is as follows: , (1) in It's a score chart. These are points on the score graph. It uses logistic loss, as shown below: , (2) in These are the labeled values ​​of the score map. It is the predicted value of the score plot. The definition is as follows: , (3) in For network step size, These are the center coordinates of the target on the feature map. It is the radius of the circular region.

[0024] The working principle of contour segmentation networks is as follows: target contour It is an ordered set of points on an image, where p i = (x i y i () are the coordinates of the point. It refers to the number of dots. The features are represented as ,in It is a point exist Bilinear interpolation on These are feature maps obtained from the original input image using a CNN. yes The translation-invariant coordinates, where .like Figure 2 As shown, the process by which the contour segmentation network segments a rectangular box into the target contour is as follows: First, connecting the midpoints of each side of the rectangular box yields a rhombus contour; second, the rhombus contour is processed by the contour segmentation network to obtain extreme points, thus yielding an octagonal contour; finally, the octagonal contour is iteratively deformed by the contour segmentation network into the target contour. In this invention, the number of iterations is 3. The contour segmentation network uses an extreme point prediction loss function and a contour deformation loss function for supervised learning. The extreme point prediction loss function is: (4) in These are the predicted coordinates of the extreme points. These are the coordinates of the extreme points. The contour deformation loss function is: , (5) in These are the predicted coordinates of the contour points. These are the coordinates of the outline points. The number of points (in this invention) Take 128).

[0025] Based on equations (1), (4), and (5), a multi-task loss function for training a twin target tracking network is defined. As shown below: , (6) in, These are hyperparameters, which are set in this invention. .

[0026] The network training process and details are as follows: The twin target tracking network was trained for 20 epochs using the COCO, ImageNet-VID, and YouTube-VOS datasets. The optimization algorithm was SGD, and the initial learning rate was 10. -3 In the first 5 training cycles, the learning rate increases linearly to 5 × 10. -3 In the subsequent 15 training cycles, the learning rate was reduced logarithmically to 5 × 10⁻⁶. -4 .

[0027] Assume the target bounding box at time t is ,in( , ) is the center location of the target. and Here, denoted by , are the width and height of the target, respectively. The number of frames in the image sequence is T, and N is the number of target contour points. Based on the forward pass of the Siamese target tracking network, an anchor-free target tracking method based on the Siamese network is designed.

[0028] Based on the above system, this invention designs a Siamese target tracking network for anchor-free target tracking. This network consists of two parts: a Siamese network and a contour segmentation network. It does not require pre-defining anchor boxes based on prior knowledge, thus reducing hyperparameters. On the other hand, addressing the problems of slow speed and mask accuracy limited by the target tracking box in region segmentation-based target tracking methods, this invention implements an anchor-free target tracking method based on a Siamese network. This method uses region classification and contour regression to model target tracking, and can simultaneously obtain the target bounding box and contour.

[0029] To test the performance of the aforementioned network, testing was conducted on a SYS-7048GR-TR desktop computer (CPU model: Intel Xeon(R) ES-2630v4 @ 2.20GHz × 20, RAM: 64GB, GPU: RTX2080Ti 11G). The software environment consisted of Ubuntu 18.04, Python 3.7, cuda 10.0, and cuDNN 7.5. The proposed target tracking method was experimentally validated using the OTB-2015 dataset. The OTB-2015 dataset contains 100 video sequences with 11 challenging attributes: illumination variation (IV), scale variation (SV), occlusion (OCC), deformation (DEF), motion blur (MB), fast motion (FM), in-plane rotation (IPR), out-of-plane rotation (OPR), target out of bounds (OV), background clutter (BC), and low resolution (LR). The dataset utilizes the OPE (One-Pass Evaluation) method to calculate Precision and Success metrics to evaluate the tracker's accuracy and robustness. Since the labeled bounding boxes in the OTB-2015 dataset are regular rectangles, this method uses the minimum bounding box of the target contour to calculate Precision and Success. Table 1 shows the Precision, Success, and FPS metrics of 10 target tracking methods on the OTB-2015 dataset, including this method, DaSiamRPN, SiamRPN++, ECO, MDNet, C-COT, MCPF, TADT, DeepSRDCF, and SRDCF. Figure 4 Precision and success metric curves for 10 target tracking methods on the OTB-2015 dataset. Figure 3 and Figure 4It can be seen that our method has the best success rate. For the precision rate, it is similar to SiamRPN++ and slightly lower than DaSiamRPN, but still has a competitive advantage. In terms of tracking speed, our method is superior to the others. The reason our method's precision rate is lower than DaSiamRPN's is that DaSiamRPN uses ImageNet and COCO datasets to augment the training set during network training, while our method does not use this technique. Compared to DaSiamRPN, our method has two advantages: First, our method uses a contour segmentation network to deform the initial bounding box into the target contour, essentially using multi-point regression of the target's scale, making it robust to target changes. DaSiamRPN, on the other hand, is based on anchor regression, and its results are limited by the anchor settings. Therefore, our method outperforms DaSiamRPN in the success rate. Second, our method fits a regular rectangular box or a rotated rectangular box to the target contour, while DaSiamRPN can only obtain a regular rectangular box.

[0030] Example 2

[0031] This invention designs an anchor-free target tracking system based on Siamese networks within a deep learning framework, such as... Figure 1 As shown, the system consists of two parts: a Siamese network and a contour segmentation network. The Siamese network is responsible for extracting the convolutional neural network (CNN) features of the target template and the search region, calculating the score map of the search region, and outputting the initial bounding box; the contour segmentation network is used for target contour regression.

[0032] The working principle of a Siamese network is as follows: A Siamese network consists of a Siamese backbone network and a classification module. It takes image pairs consisting of a template image and a search region image as input and outputs a score map of the search region. The Siamese backbone network consists of template branches and search region branches with identical structures, sharing network parameters. If the template image and a certain image patch in the search region represent the same target, a larger score value is output; otherwise, a smaller score value is output. The task of the Siamese network is to learn a metric for the template image. and search area similarity function ,in" "Indicates cross-relevance, and They are and CNN features.

[0033] To obtain the similarity function To use images The neural network is trained offline, and the loss function used for training the neural network is as follows: , (1) in It's a score chart. These are points on the score graph. It uses logistic loss, as shown below: (2) in These are the labeled values ​​of the score map. It is the predicted value of the score plot. The definition is as follows: , (3) in For network step size, These are the center coordinates of the target on the feature map. It is the radius of the circular region.

[0034] The working principle of contour segmentation networks is as follows: target contour It is an ordered set of points on an image, where p i = (x i y i () are the coordinates of the point. It refers to the number of dots. The features are represented as ,in It is a point exist Bilinear interpolation on These are feature maps obtained from the original input image using a CNN. yes The translation-invariant coordinates, where .like Figure 2 As shown, the process by which the contour segmentation network segments a rectangular box into the target contour is as follows: First, connecting the midpoints of each side of the rectangular box yields a rhombus contour; second, the rhombus contour is processed by the contour segmentation network to obtain extreme points, thus yielding an octagonal contour; finally, the octagonal contour is iteratively deformed by the contour segmentation network into the target contour. In this invention, the number of iterations is 3. The contour segmentation network uses an extreme point prediction loss function and a contour deformation loss function for supervised learning. The extreme point prediction loss function is: , (4) in These are the predicted coordinates of the extreme points. These are the coordinates of the extreme points. The contour deformation loss function is: , (5) in These are the predicted coordinates of the contour points. These are the coordinates of the outline points. The number of points (in this invention) Take 128).

[0035] Based on equations (1), (4), and (5), a multi-task loss function for training a twin target tracking network is defined. As shown below: , (6) in, These are hyperparameters, which are set in this invention. .

[0036] The network training process and details are as follows: The twin target tracking network was trained for 20 epochs using the COCO, ImageNet-VID, and YouTube-VOS datasets. The optimization algorithm was SGD, and the initial learning rate was 10. -3 In the first 5 training cycles, the learning rate increases linearly to 5 × 10. -3 In the subsequent 15 training cycles, the learning rate was reduced logarithmically to 5 × 10⁻⁶. -4 .

[0037] Assume the target bounding box at time t is ,in( , ) is the center location of the target. and Here, denoted by , are the width and height of the target, respectively. The number of frames in the image sequence is T, and N is the number of target contour points. Based on the forward pass of the Siamese target tracking network, an anchor-free target tracking method based on the Siamese network is designed.

[0038] Based on the above system, this invention designs a Siamese target tracking network for anchor-free target tracking. This network consists of two parts: a Siamese network and a contour segmentation network. It does not require pre-defining anchor boxes based on prior knowledge, thus reducing hyperparameters. On the other hand, addressing the problems of slow speed and mask accuracy limited by the target tracking box in region segmentation-based target tracking methods, this invention implements an anchor-free target tracking method based on a Siamese network. This method uses region classification and contour regression to model target tracking, and can simultaneously obtain the target bounding box and contour.

[0039] To test the performance of the aforementioned network, testing was conducted on a SYS-7048GR-TR desktop computer (CPU model: Intel Xeon(R) ES-2630v4 @ 2.20GHz × 20, RAM: 64GB, GPU: RTX2080Ti 11G). The software environment consisted of Ubuntu 18.04, Python 3.7, cuda 10.0, and cuDNN 7.5. The proposed target tracking method was experimentally validated using the OTB-2015 dataset. The OTB-2015 dataset contains 100 video sequences with 11 challenging attributes: illumination variation (IV), scale variation (SV), occlusion (OCC), deformation (DEF), motion blur (MB), fast motion (FM), in-plane rotation (IPR), out-of-plane rotation (OPR), target out of bounds (OV), background clutter (BC), and low resolution (LR). The dataset utilizes the OPE (One-Pass Evaluation) method to calculate Precision and Success metrics to evaluate the tracker's accuracy and robustness. Since the labeled bounding boxes in the OTB-2015 dataset are regular rectangles, this method uses the minimum bounding box of the target contour to calculate Precision and Success. Table 1 shows the Precision, Success, and FPS metrics of 10 target tracking methods on the OTB-2015 dataset, including this method, DaSiamRPN, SiamRPN++, ECO, MDNet, C-COT, MCPF, TADT, DeepSRDCF, and SRDCF. Figure 4 Precision and success metric curves for 10 target tracking methods on the OTB-2015 dataset. Figure 3 and Figure 4It can be seen that our method has the best success rate. For the precision rate, it is similar to SiamRPN++ and slightly lower than DaSiamRPN, but still has a competitive advantage. In terms of tracking speed, our method is superior to the others. The reason our method's precision rate is lower than DaSiamRPN's is that DaSiamRPN uses ImageNet and COCO datasets to augment the training set during network training, while our method does not use this technique. Compared to DaSiamRPN, our method has two advantages: First, our method uses a contour segmentation network to deform the initial bounding box into the target contour, essentially using multi-point regression of the target's scale, making it robust to target changes. DaSiamRPN, on the other hand, is based on anchor regression, and its results are limited by the anchor settings. Therefore, our method outperforms DaSiamRPN in the success rate. Second, our method fits a regular rectangular box or a rotated rectangular box to the target contour, while DaSiamRPN can only obtain a regular rectangular box.

[0040] Based on the aforementioned tracking network, an anchor-free target tracking method was designed, including the following steps:

[0041] Input: Initial target bounding box and image sequences .

[0042] Output: target box at time and target outline .

[0043] Step 1: Based on the initial target bounding box Crop the template image to a size of 127×127. .

[0044] Step 2: Calculate the template image using a twin backbone network Features .

[0045] .

[0046] Step 3: Calculate the search area. .

[0047] Step 4: Crop the search area to a size of 255×255. .

[0048] Step 5: Calculate the search region using a twin backbone network. Features .

[0049] Step 6: Calculate the score graph. .

[0050] Step 7: Calculation Target location at all times .

[0051] Step 8: Initialization Target scale at any time .

[0052] Step 9: Obtain Initial bounding box of target at any given time. .

[0053] Step 10: Input the contour segmentation network and obtain Target outline at any time .

[0054] Step 11: Based on the target contour Fitting target rectangle at any time .

[0055] Step 12: Output Time target box and target outline .

[0056] END FOR

[0057] The parameters involved in the above steps are explained below:

[0058] (1) : Initial bounding box of the target at time t, where , , and These are the x-coordinate, y-coordinate, width, and height of the initial bounding box of the target at time t, where... It is the center of the target location.

[0059] (2) The coordinates of the target contour at time step, where , and These are the x and y coordinates of the i-th point on the target contour at time t, respectively.

[0060] (3) The minimum bounding rectangle of the target at any given time. and These are its x-coordinate, y-coordinate, width, and height, respectively. It is the center of the target location.

[0061] (4) This represents the coordinate transformation from the feature map to the original map.

[0062] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An anchor-free target tracking method based on Siamese networks, characterized in that, It is used in systems that include Siamese networks and contour segmentation networks. The Siamese network is responsible for extracting convolutional neural network (CNN) features of the target template and search region and calculating the score map of the search region, outputting an initial bounding box. The contour segmentation network is responsible for regressing the target contour. The Siamese network consists of two parts: a Siamese backbone network and a classification module. It takes image pairs consisting of a template image and a search region image as input and outputs a score map of the search region. The Siamese backbone network consists of template branches and search region branches with the same structure, and they share network parameters. Anchor-free target tracking methods include the following steps: Step 1: Based on the initial target bounding box Image sequence Crop the template image to a size of 127×127. ; Step 2: Calculate the template image using a twin backbone network Features ; Steps 3-12 are executed in a loop from t=2 to T: Step 3: Calculate the search area. ; Step 4: Crop the search area to a size of 255×255. ; Step 5: Calculate the search region using a twin backbone network. Features ; Step 6: Calculate the score graph. ; Step 7: Calculation Target location at all times ; This represents the coordinate transformation from the feature map to the original image; Step 8: Initialization Target scale at any time , ; Step 9: Obtain Initial bounding box of target at any given time. ; Step 10: Input the contour segmentation network and obtain Target outline at any time ; Step 11: Based on the target contour Fitting target rectangle at any time ; Step 12: Output Time target box and target outline ; Denotes the initial bounding box of the target at time t, where , , and These are the x-coordinate, y-coordinate, width, and height of the initial bounding box of the target at time t, where ( , ) is the center location of the target; Represents the coordinates of the target contour at time t, where , and These are the x and y coordinates of the i-th point on the target contour at time t, respectively. express The minimum bounding rectangle of the target at any given time. , , and These are its x-coordinate, y-coordinate, width, and height, respectively, where ( , ) is the center position of the target.

2. The anchor-free target tracking method based on Siamese networks according to claim 1, characterized in that, The target contour is an ordered set of points on an image. The set of, where = (A i B i () are the coordinates of the point. It is the number of points, points The features are represented as ,in It is a point exist Bilinear interpolation on These are feature maps obtained from the original input image using a CNN. yes The translation-invariant coordinates, where .

3. The anchor-free target tracking method based on Siamese networks according to claim 1, characterized in that, If the template image and a certain image patch in the search area represent the same target, a larger score value is output; otherwise, a smaller score value is output.

4. The anchor-free target tracking method based on Siamese networks according to claim 3, characterized in that, The task of a Siamese network is to learn a metric template image. and search area similarity function ,in" "Indicates cross-relevance, and They are and CNN features.

5. The anchor-free target tracking method based on Siamese networks according to claim 1, characterized in that, The process by which the contour segmentation network segments a rectangular box into a target contour is as follows: First, the midpoints of each side of the rectangular box are connected to obtain a rhombus contour; second, the rhombus contour is processed by the contour segmentation network to obtain extreme points, and then an octagonal contour is obtained; finally, the octagonal contour is iteratively transformed into the target contour by the contour segmentation network, with an iteration count of 3.

Images