Method and apparatus for performing target object tracking

Abstract

A method and device for performing target object tracking are disclosed. The method comprises: obtaining a first depth feature of a target object region image in an image and a second depth feature of a search region image; obtaining a global response map according to the first depth feature and the second depth feature, and obtaining a first-stage target object bounding box prediction result according to the global response map; updating the second depth feature according to the first-stage target object bounding box prediction result; obtaining a plurality of local feature blocks based on the first depth feature, obtaining a local response map according to the plurality of local feature blocks and the updated second depth feature, and obtaining a second-stage target object bounding box prediction result according to the local response map.

Description

Technical Field

[0001] The present invention generally relates to the field of computer vision, and more particularly, to a method and apparatus for performing target object tracking. Background Art

[0002] Visual object tracking is an important area of ​​computer vision. Its specific task is to continuously predict the bounding box of the target object in subsequent frames of a video sequence based on the first frame image and a given bounding box of the target object. The target object can be an object or part of an object. Due to the limited information and complex scenes, the task of visual object tracking is extremely challenging. Complex scenes mainly include situations where the target object is partially or completely occluded, the shape of the target object changes, the target object moves rapidly, the frame image is blurred, the scene lighting changes, and the video perspective changes significantly. The target tracking method can establish the association between the target object at different times, and therefore has a wide range of applications in the field of computer vision, especially in some video applications, including camera tracking, action recognition, live broadcast of events, security monitoring, human-computer interaction, etc.

[0003] However, existing target tracking methods all use only a one-stage network, and only perform a single correlation operation on the target object's features and the search area's features in the one-stage network, considering only the global feature correlation of the target object. This will lead to a decrease in target tracking accuracy. In view of this, a target tracking method and device that can improve target tracking accuracy is needed. Summary of the Invention

[0004] To address the problem of low target tracking accuracy, the present invention proposes a target object tracking method and system that combines block correlation and global correlation in a two-stage framework.

[0005] According to one aspect of the present invention, a method for tracking a target object using a cascade network is provided, and the method may include: obtaining a first depth feature of a target object area image and a second depth feature of a search area image in an image; obtaining a global response map based on the first depth feature and the second depth feature, and obtaining a first-stage target object bounding box prediction result based on the global response map; updating the second depth feature based on the first-stage target object bounding box prediction result; obtaining multiple local feature blocks based on the first depth feature, obtaining a local response map based on the multiple local feature blocks and the updated second depth feature, and obtaining a second-stage target object bounding box prediction result based on the local response map.

[0006] According to an exemplary embodiment, obtaining multiple local feature blocks based on the first depth feature may include: dividing the first depth feature or the third depth feature obtained by further feature extraction of the first depth feature into blocks to obtain multiple local feature blocks; obtaining a local response map based on the multiple local feature blocks and the updated second depth feature may include: performing block correlation on the multiple local feature blocks and the updated second depth feature or the fourth depth feature obtained by further feature extraction of the updated second depth feature to obtain a local response map.

[0007] According to an exemplary embodiment, performing block correlation on the multiple local feature blocks with the updated second depth feature or the fourth depth feature obtained by further feature extraction on the updated second depth feature to obtain a local response map may include: performing block correlation on each local feature block in the multiple local feature blocks with the updated second depth feature or the fourth depth feature to obtain a plurality of local sub-response maps, and fusing the multiple local sub-response maps to obtain the local response map.

[0008] According to an exemplary embodiment, obtaining a second-stage target object bounding box prediction result based on a local response map may include: predicting a position offset and a size offset of the second-stage target object bounding box based on the local response map, and obtaining the second-stage target object bounding box prediction result based on the predicted position offset and size offset.

[0009] According to an exemplary embodiment, fusing the multiple local sub-response maps to obtain the local response map includes: classifying each local feature block of the multiple local feature blocks as a target object feature block or a background feature block; and fusing the local sub-response map corresponding to the target object feature block and the sub-response map corresponding to the background feature block to obtain the local response map.

[0010] According to an exemplary embodiment, classifying each local feature block among the multiple local feature blocks as a target object feature block or a background feature block includes: using an initial target object bounding box calibrated on the target object sub-image as a classification basis, and classifying each local feature block as a target object feature block or a background feature block according to the ratio of the overlapping area between each local feature block and the initial target object bounding box to the each local feature block.

[0011] According to an exemplary embodiment, the first-stage target object bounding box prediction result and the second-stage target object bounding box prediction result both include position information and size information of the target object bounding box, the position offset may be the coordinate offset between the center position coordinates of the second-stage target object bounding box and the center position coordinates of the first-stage target object bounding box, and the size offset may be the size offset between the second-stage target object bounding box and a pre-specified target object bounding box, wherein obtaining the second-stage target object bounding box prediction result based on the predicted position offset and size offset may include: when the sum of the absolute values ​​of the coordinate offsets is greater than a preset threshold, using the first-stage target object bounding box prediction result as the second-stage target object bounding box prediction result; when the sum of the absolute values ​​of the coordinate offsets is less than or equal to the preset threshold, obtaining the second-stage target object bounding box prediction result by adding the center position of the first-stage target object bounding box to the predicted position offset and adding the size of the pre-specified target object bounding box to the predicted size offset.

[0012] According to an exemplary embodiment, obtaining a first-stage target object bounding box prediction result based on a global response map may include: using the position with the largest score in the global response map as the position information included in the first-stage target object bounding box prediction result, and using the size of the target object bounding box predicted on the image before the current frame image as the size information included in the first-stage target object bounding box prediction result.

[0013] According to an exemplary embodiment, dividing the first depth feature or the third depth feature obtained by further feature extraction of the first depth feature into blocks to obtain multiple local feature blocks may include: dividing the first depth feature or the third depth feature into blocks according to one of the following three division methods: a division method in which the local feature blocks do not overlap with each other; a division method in which adjacent local feature blocks overlap with each other; and a division method based on a predetermined block distribution.

[0014] According to another aspect of the present invention, a device for performing target object tracking is provided, which may include: a first-stage tracker, configured to: obtain a first depth feature of a target object area image and a second depth feature of a search area image in an image; obtain a global response map based on the first depth feature and the second depth feature, and obtain a first-stage target object bounding box prediction result based on the global response map; update the second depth feature based on the first-stage target object bounding box prediction result; a second-stage tracker, configured to: obtain a plurality of local feature blocks based on the first depth feature, obtain a local response map based on the plurality of local feature blocks and the updated second depth feature, and obtain a second-stage target object bounding box prediction result based on the local response map.

[0015] According to another aspect of the present application, there is provided an electronic device, which can include a processor and a memory, wherein the memory has stored therein program instructions, wherein the program instructions, when executed by the processor, cause the processor to perform the target tracking method as described above.

[0016] According to another aspect of the present application, there is provided a computer readable recording medium, wherein the computer readable recording medium has recorded thereon program instructions, wherein the program instructions, when executed by a processor, cause the processor to perform the target tracking method as described above.

[0017] According to the target tracking method and the target tracking system of the present application, since the global correlation and the block correlation are combined by using a cascade network for two-stage tracking, not only the target tracking accuracy can be effectively improved, but also the target object can be tracked in real time with high precision and stability, and with light weight and low computation consumption. BRIEF DESCRIPTION OF DRAWINGS

[0018] These and / or other aspects and advantages of the present application will become apparent and more readily appreciated from the following detailed description, taken in conjunction with the accompanying drawings, in which:

[0019] Figure 1 is a schematic diagram of a process of target object tracking by a conventional target tracking device;

[0020] Figure 2 is a schematic diagram of a conventional target tracking method;

[0021] Figure 3 is a brief schematic diagram showing the concept of the target tracking method according to the present application;

[0022] Figure 4 is a detailed schematic diagram showing the concept of the target tracking method according to the present application;

[0023] Figure 5 is a flowchart showing the target tracking method according to the present application;

[0024] Figure 6 is a schematic diagram of a global correlation operation according to the present application;

[0025] Figure 7 is a schematic diagram of a first stage operation of the target tracking method according to the present application;

[0026] Figure 8 is a schematic diagram of a block division manner according to the present application;

[0027] Figure 9 is a schematic diagram of a block correlation operation according to the present application;

[0028] Figure 10 is a schematic diagram of interference suppression and response fusion according to the present application;

[0029] Figure 11 is a schematic diagram of adaptive prediction according to the present application;

[0030] Figure 12 is a schematic diagram of the second stage operation of the target tracking method according to the present application;

[0031] Figure 13 is a schematic diagram of network training according to the present application;

[0032] Figure 14 shows the difference between the effect of the block correlation method combined with interference suppression and the global correlation method and the block correlation method proposed by the present application;

[0033] Figure 15 is a block diagram of a target tracking device according to the present application. DETAILED DESCRIPTION

[0034] Before describing the inventive concept and the exemplary embodiments of the present application, for better understanding of the present application, first of all, a brief description of the target tracking system and the target tracking method in the prior art is given.

[0035] Figure 1 is a schematic diagram of the process of target object tracking by a commonly used target tracking device. As shown in Figure 1 , generally, a monocular camera can be used to collect the first frame image of the video sequence, and the bounding box of the target object is obtained by manual annotation or visual object detection technology (Visual object detection) to initialize the target tracking device. In the subsequent frame images of the video sequence, the search region is selected according to the prediction result of the previous frame image, and the target object is tracked by using image features.

[0036] The feature representation of the target object is the key to the performance of the target tracking system. Existing target tracking methods mainly use hand-crafted features and deep features. Commonly used hand-crafted features include histogram of oriented gradients (HOG), scale-invariant features (SIFT), grayscale features (Gray), etc. Deep features are obtained by training and learning a large number of samples using a parameter model, and have stronger distinguishability and robustness than hand-crafted features. In recent years, with the breakthrough of target tracking methods based on deep features, they have surpassed traditional methods based on hand-crafted features in terms of robustness and accuracy, especially the deep features obtained using convolutional neural networks (CNN).

[0037] Among the existing target tracking methods based on deep features, some use a Siamese network as the basic framework of the target tracking network. The Siamese network uses the same network parameters to extract the image features of the target object in the first frame image and the image features of the search area in the current frame image, ensuring that the extracted features are in the same feature space. Then, the correlation between the two features is used to obtain the response map of the target object in the search area image. After training on a large-scale dataset, good results in accuracy and robustness have been achieved. Existing target tracking methods based on Siamese networks can be divided into two categories: without bounding box regression and with bounding box regression.

[0038] The bounding box-free regression method uses a Siamese network to extract features from the same feature space between the target object image in the first frame and the search area image in the current frame. Correlation operations are then performed to generate a response map representing the degree of match between the target object image and the search area image. The location corresponding to the maximum value in the response map is taken as the center position of the target object in the current frame. To account for object size variations, a multi-scale test is used, and the bounding box of the target object at the scale with the largest response is used as the bounding box of the target object in the current frame.

[0039] The bounding box regression method improves the non-bounding box regression method. Figure 2 is a schematic diagram of an existing target object tracking method using bounding box regression. Figure 2As shown in Figure 2, this method further expands upon the positional response information obtained from related operations. For example, it utilizes a region proposal network (RPN) to simultaneously classify multiple candidate boxes (distinguishing between target and non-target objects) and obtain bounding box coordinate regression results. The candidate box with the highest classification probability is selected as the target object in the current frame image. Due to the learning of bounding box coordinate regression, this method achieves high target tracking accuracy.

[0040] Target tracking without bounding box regression does not require an additional network for bounding box coordinate regression learning, resulting in a lighter model with fewer parameters, but lower accuracy. Furthermore, multi-scale testing reduces the model's lightweight advantage (a test is required at each scale). Target tracking with bounding box regression, such as incorporating a region generation network, can significantly improve tracker performance, but the large number of parameters reduces the system's real-time performance.

[0041] Furthermore, both of the aforementioned target tracking methods utilize only a single-stage network, and only perform a single correlation operation on the target object's features and the search area's features within the single-stage network, considering only the global feature correlation of the target object. This approach presents the following technical drawbacks: Firstly, due to the imbalance between positive and negative samples during network training, the single-stage network tends to suppress the influence of negative samples and insufficiently utilizes the information of positive samples, resulting in reduced tracking accuracy. Furthermore, since tracking of the target object occurs between video frames, the tracker is susceptible to cumulative errors. In situations with complex backgrounds, significant changes in lighting, or object shape, tracking performance deteriorates without exploiting information surrounding the target. Secondly, by only using the target object's global features for overall tracking, this approach ignores local information, leading to inaccurate tracking when the target is deformed.

[0042] Therefore, the present invention proposes a new target tracking method to improve the accuracy of target tracking.

[0043] Below, we will refer to Figures 3 to 14 The concept and exemplary embodiments of the present invention for tracking a target object are described in detail.

[0044] Figure 3 A schematic diagram showing the concept of the target tracking method according to the present invention is shown. The target tracking method of the present invention includes two stages. Figure 3 As shown in FIG, after the target object region image and the search region image are determined, the first stage tracking is performed to obtain a rough preliminary target tracking result. Then, the second stage tracking is performed to obtain an accurate final target tracking result.

[0045] Figure 4 Detailed schematic diagram showing the concept of the target tracking method according to the present invention. Specifically, Figure 4 As shown, the target tracking method of the present invention includes two stages. The first stage is coarse tracking. In this stage, after the target object area image and the search area image are determined, the global features of the target object area image and the search area image are extracted respectively, and the extracted global features are subjected to global correlation calculation to obtain a global response map, and then a rough matching is performed to obtain a rough prediction result. The second stage is fine tracking, and a local response map is further obtained based on the local image features of the target object and the updated search area image features (for example, the local image features of the target object and the updated search area image features are subjected to block correlation to obtain the local response map), and then the final target tracking result is obtained based on the local response map.

[0046] Next, we will refer to Figures 5 to 13 The target tracking method according to the present invention is described in detail. Figure 5 : is a flow chart showing a target tracking method according to the present invention. Figure 5 , in step S510, the first depth feature of the target object area image and the second depth feature of the search area image in the image are obtained. For example, first, a video sequence can be obtained, and then, using a first neural network, the first depth feature of the target object area image in the first frame image of the video sequence is extracted, and the second depth feature of the search area image in the current frame image of the video sequence is extracted. Here, the target object area image can be obtained by cropping the first frame image according to the manually calibrated initial target object bounding box, or the target object sub-image can also be obtained by cropping the first frame image according to the initial target object bounding box determined by target object detection, and the present invention is not limited to this. In addition, the first depth feature here is a global feature of the target object area image, and the second depth feature is a global feature of the search area image. As an example, the first neural network mentioned above can be a twin convolutional network, but is not limited to this.

[0047] In step S520, a global response map is obtained based on the first depth feature and the second depth feature, and the first-stage target object bounding box prediction result is obtained based on the global response map. Specifically, the global response map is obtained by performing a global correlation calculation on the first depth feature and the second depth feature. For ease of understanding, the relevant operations are briefly introduced first. In the image task, the application of the relevant operation can obtain a response map Y representing the degree of similarity between the two images, where the larger the value, the higher the degree of similarity between the corresponding position in the search area image Z and the target object area image X. The relevant calculation is shown in the following formula:

[0048] Y=corr(X,Z)

[0049]

[0050] Where h, w represent the size of the image X, and i, j, u, v are the coordinates in the image respectively. Figure 6 This is a schematic diagram of the global correlation operation according to the present invention. Figure 6 The global related operations of the present invention are briefly described. Figure 6 As shown, global correlation is the whole image feature of the target object area image (in Figure 6 It is called “target object image feature F T ”) and the entire image features of the search area sub-image (in Figure 6 It is called “search area image feature F St ”) performs correlation operations. A global response map can be obtained through global correlation operations. In the present invention, the first-stage target object bounding box prediction result may include position information and size information of the target object bounding box. Specifically, in step S520, obtaining the first-stage target object bounding box prediction result according to the global response map may include: using the position with the largest score in the global response map as the position information included in the first-stage target object bounding box prediction result, and using the size of the target object bounding box predicted on the image before the current frame image as the size information included in the first-stage target object bounding box prediction result. In step S530, the second depth feature is updated according to the first-stage target object bounding box prediction result to obtain an updated second depth feature. Specifically, the search area sub-image can be cropped according to the first-stage target object bounding box prediction result to obtain a reduced search area sub-image, and the second depth feature of the reduced search area sub-image can be extracted using the first convolutional network as the updated second depth feature.

[0051] The above steps S510 to S530 are operations performed in the first stage tracking according to the target tracking method of the present invention.

[0052] In order to more intuitively understand the operation of the target tracking method according to the present invention in the first stage, refer to Figure 7 The operation of the first stage of the target tracking method is briefly described. Figure 7As shown, in the first stage tracking, there are mainly three parts: feature extraction (corresponding to step S510 above), global correlation (corresponding to step S520 above), and feature map cropping (corresponding to step S530 above). According to an exemplary embodiment, first, in the feature extraction part, feature extraction can be performed on the target object region image and the search region image respectively by using, for example, a convolutional neural network. In the feature extraction, a light-weight convolutional neural network φ1 can be used to extract image features. For the input target object region image Z and the search region image X, the convolutional neural network φ1 can be used to obtain the deep features of the target object region image φ1(Z) and the deep features of the search region image φ1(X). In the adopted twin convolutional network, the parameters of the two branches are shared to ensure that the images are mapped to the same feature space.

[0053] Then, in the global correlation part, global correlation operation is performed on the extracted features to obtain a global response map, and the response map is processed to obtain the first stage prediction result (i.e., the first stage target object bounding box prediction result mentioned above). The global correlation operation can obtain a global position (similarity) response map f of the entire target object region image and the search region image, which can be represented as follows:

[0054] f = corr(φ1(Z), φ1(X))

[0055] After obtaining the global response map, the position with the maximum response score in the global response map can be selected as the first stage prediction position of the target object bounding box, and the size of the bounding box is selected as the size of the target object bounding box predicted in the previous frame of the current frame image, so that the first stage target object bounding box prediction result P1 = (x1, y1, w1, h1) can be obtained, where x1 and y1 are the horizontal and vertical coordinates of the center position of the first stage target object bounding box, and w1 and h1 are the width and height of the first stage target object bounding box.

[0056] Finally, in the feature map cropping part, according to the center position and size of the target bounding box predicted in the first stage, the search region image X is cropped to obtain a search region image X' with a smaller region range, and the second deep features of the search region image are extracted to obtain updated second deep features φ1(X') for use in the second stage.

[0057] Next, referring back to Figure 5, continue to describe the operation of the second stage of the target tracking method according to the present invention. In step S540, a plurality of local feature blocks are obtained based on the first depth feature, a local response map is obtained according to the plurality of local feature blocks and the updated second depth feature, and a second stage target object bounding box prediction result is obtained according to the local response map. For example, the first depth feature or the third depth feature obtained by further feature extraction of the first depth feature can be divided into blocks to obtain a plurality of local feature blocks. As an example, a second neural network can be used to perform further feature extraction on the first depth feature and the updated second depth feature to obtain a third depth feature of the target object image and a fourth depth feature of the search area image. Here, the third depth feature is a depth feature obtained by further convolution operation on the first depth feature, and the fourth depth feature is a depth feature obtained by further convolution operation on the updated second depth feature.

[0058] As an example, dividing the first depth feature or the third depth feature obtained by further feature extraction of the first depth feature into blocks to obtain multiple local feature blocks can be dividing the first depth feature or the third depth feature into blocks according to one of the following three division methods: a division method in which the local feature blocks do not overlap with each other; a division method in which adjacent local feature blocks overlap with each other; a division method based on a predetermined block distribution. Here, the predetermined block distribution can be either a manually specified block distribution or a learned block distribution. The block distribution based on the manually specified block distribution can be block division according to a specified specific distribution. For example, when the specified block distribution is a Gaussian distribution, the distribution of the blocks is closer to the center. In addition, the learned block distribution mentioned here can be a block distribution obtained in the following way: taking the parameters of a specific distribution (such as the mean and variance of a Gaussian distribution) as the optimization target, and continuously adjusting them during training until the most suitable parameters are found, and then learning the corresponding block distribution. Figure 8 FIG. 4 is a schematic diagram of a block division method according to the present invention, which specifically shows the above three division methods.

[0059] As described above, after obtaining a plurality of local feature blocks, a local response map can be obtained based on the plurality of local feature blocks and the updated second depth feature, and a second-stage target object bounding box prediction result can be obtained based on the local response map. According to an exemplary embodiment, the local response map can be obtained by performing block correlation on the plurality of local feature blocks and the updated second depth feature or the fourth depth feature obtained by further feature extraction of the updated second depth feature. Specifically, first, each local feature block in the plurality of local feature blocks can be block correlated with the updated second depth feature or the fourth depth feature to obtain a plurality of local sub-response maps. Then, the plurality of local sub-response maps are fused to obtain the local response map. Figure 9FIG. 1 is a schematic diagram of block-related operations according to the present invention. Figure 9 As shown, block correlation is performed after the target object feature (the first depth feature or the third depth feature mentioned above) is divided into blocks, and then a correlation operation is performed on each local feature block in the multiple local feature blocks and the search area image feature (the updated second depth feature or the fourth depth feature mentioned above) to obtain multiple local sub-response maps, which can then be fused to obtain a fused local response map. According to an exemplary embodiment of the present invention, fusing the multiple local sub-response maps to obtain the local response map may include: classifying each local feature block in the multiple local feature blocks into a target object feature block or a background feature block, and fusing the local sub-response map corresponding to the target object feature block and the sub-response map corresponding to the background feature block to obtain the local response map. According to such a fusion method, the stability and accuracy of the target tracking method can be further improved. This is because, in addition to the target object, the target object sub-image also contains part of the background area, and the characteristics of the background area will affect the stability and accuracy of the target tracking method. By classifying the multiple local feature blocks into target object feature blocks and background feature blocks before fusing, the interference of the background can be effectively reduced.

[0060] Figure 10 FIG. 1 is a schematic diagram of interference suppression and response graph fusion according to the present invention. Figure 10 As shown, as an example, when performing local feature block classification, the initial target object bounding box calibrated on the target object area image can be used as the classification basis. Each local feature block can be classified as a target object feature block or a background feature block based on the ratio of the overlapping area between each local feature block and the initial target object bounding box to the local feature block. For example, using the initial target object bounding box calibrated on the target object area image as the classification basis, when more than p% of the area of ​​the local feature block is within the bounding box, it is a target object feature block, and when the overlap between the local feature block and the bounding box is less than p%, it is a background feature block, where p can be a pre-set threshold. After classification, the local sub-response map corresponding to the target object feature block and the sub-response map corresponding to the background feature block can be fused using the following equation to obtain the local response map:

[0061] in,

[0062] Where S is the local response map, s o is the local sub-response map corresponding to the target object feature block, s b is the sub-response graph corresponding to the background feature block, n o is the number of target object feature blocks, n b is the number of background feature blocks.

[0063] After obtaining the local response map, the second-stage target object bounding box prediction result is then obtained based on the local response map. Specifically, the position offset and size offset of the second-stage target object bounding box can be predicted based on the local response map, and the second-stage target object bounding box prediction result is obtained based on the predicted position offset and size offset. For example, the local response map can be processed using a third neural network to predict the position offset and size offset of the second-stage target object bounding box. The third neural network may be different from the first and second neural networks mentioned above. Here, the second target object bounding box prediction result may include position information and size information of the target object bounding box. Hereinafter, the above process may be referred to as an adaptive prediction process. Figure 11 Schematic diagram of adaptive prediction according to the present invention. Figure 11 As shown in the adaptive prediction process, the convolutional network can be used to process the local response map S first, and the offset D of the second stage bounding box is predicted. x ,d y ,d w ,d h ), the offset includes a position offset and a size offset. According to an exemplary embodiment, the position offset may be a coordinate offset between the center position coordinates of the second-stage target object bounding box and the center position coordinates of the first-stage target object bounding box, and the size offset may be a size offset between the second-stage target object bounding box and a pre-specified target object bounding box. After obtaining the offset, the second-stage target object bounding box prediction result is obtained based on the predicted position offset and size offset. Specifically, when the sum of the absolute values ​​of the coordinate offsets is greater than a preset threshold, the first-stage target object bounding box prediction result is used as the second-stage target object bounding box prediction result; however, when the sum of the absolute values ​​of the coordinate offsets is less than or equal to the preset threshold, the second-stage target object bounding box prediction result is obtained by adding the center position of the first-stage target object bounding box to the predicted position offset and adding the size of the pre-specified target object bounding box to the predicted size offset. For example, if the first-stage target object bounding box prediction result is P1=(x1, y1, w1, h1) and the pre-specified target object bounding box size is (w0, h0) (i.e., width is w0, height is h0), then the second-stage target object bounding box prediction result can be P2=(x 1+ d x ,y 1+ d y ,w 0+ d w ,h 0+ d h ).

[0064] At this point, the second stage of the target tracking method according to the present invention is completed. Figure 12 The second stage of the target tracking method is briefly described. Figure 12 As shown in the figure, after obtaining the depth features φ1(Z) and the updated depth features φ1(X') of the target object sub-image through the first stage operation, they can be input into the convolutional network for further feature extraction, and then the extracted features can be subjected to block correlation operations. After that, interference suppression and sub-response map fusion can be performed to obtain the local response map, and finally, adaptive prediction is performed to obtain the second stage target object bounding box prediction result P2.

[0065] According to the present invention, a cascade network (including the first neural network, the second neural network and the third neural network mentioned above) can be used to Figure 5The method shown is used to track the target object, and the cascade network can be trained using multiple supervisory signals. Here, the multiple supervisory signals include a global response map, a local response map, and a target bounding box. Below, a brief introduction to the training of the cascade network is given. It is clear to those skilled in the art that the operations performed in the training process and the prediction process are the same, the difference being that the input during training is the target object area image (also referred to as a "template image"), the search area image, and the known bounding box on the search area image, and the output can be the target bounding box, the global response map, and the local response map predicted on the search area image. The above three are used as supervisory signals, and the parameters in the network are learned by iterative cycle learning and optimizing the loss function until convergence. Specifically, when training using multiple supervisory signals, the first stage of tracking is first performed to obtain a global response map, wherein, in the global response map, the distance from the center position less than a certain threshold is set to +1, and the distance greater than a certain threshold is set to -1. Next, the second stage of tracking is performed. Specifically, first, the segmentation result of the target on the search area image can be obtained (segmentation algorithm or manual annotation can be used), and secondly, distance transformation is performed on the segmentation result, and the distance change map is numerically normalized to obtain the supervisory signal of the local response map. Finally, a fine prediction result is obtained through adaptive position prediction. During the training process, the global response map, the local response map and the target bounding box are used as supervisory signals, and the loss function is continuously optimized to convergence through iterative loop learning to learn the parameters in the cascade network. For example, the training process can be: first, the image pair extracted from the same video sequence (including the template image and the search area image) is input into the one-stage tracking network to obtain the output result of the one-stage (global response map prediction and coarse prediction box). The binary cross entropy loss function (Binary Cross Entroy) is used to calculate the loss Loss0 (loss can be understood as the difference, and the loss is zero, which means there is no difference) between the global response map prediction and the true value. Then, based on the prediction result of the coarse prediction box, the one-stage features are cropped (in Figure 13 The two layers are called "shared features" (referred to as "shared features") as the input of the second-stage tracking network and generate the true value of the local response map. Then, the output of the second stage (local response map prediction and fine prediction box) is obtained. After that, the KL divergence (Kullback-leibler Divergence) is used to measure the loss Loss1 between the local response map prediction and the true value. Next, the L1 distance is used to measure the loss Loss2 between the fine prediction box and the true box. Finally, the loss (Loss = Loss0 + (a1) * Loss1 + (a2) * Loss2) is optimized until convergence to learn the parameters of the network, where a1 and a2 are the weights of each loss.

[0066] After training, the cascade network can be tested. Specifically, after the first frame image and the target object bounding box are given, they are input into the cascade network for initialization to obtain the target image features. After that, subsequent frame images are continuously input to obtain the tracking results of the target object in real time.

[0067] The target tracking method of the present invention has been introduced above. This method proposes a two-stage target object tracking method that combines block correlation with global correlation. This method, for the first time, employs block correlation to extract local information for adjusting the target object's bounding box. Compared to existing target tracking methods, this method is lightweight and computationally inefficient. The target tracking method of the present invention enables high-precision, stable, and real-time tracking of target objects on mobile devices.

[0068] In addition, as mentioned above, the target tracking method of the present invention also adopts a block correlation method combined with interference suppression. Figure 14 The difference between the block correlation method combined with interference suppression and the global correlation method and the block correlation method used in the present invention is shown. Figure 14 As shown in the figure, by comparing and analyzing the global correlation response graph (i.e., the global response graph mentioned above), the block correlation response graph (i.e., the local response graph mentioned above) and the results of combining interference suppression on the basis of block correlation in the target tracking process, it can be found that the block correlation method combined with interference suppression adopted in the present invention can effectively extract the detailed information of the tracked target, thereby further improving the tracking accuracy.

[0069] Figure 15 1 is a block diagram of a device for performing target object tracking according to the present invention (hereinbefore, for convenience of description, it is referred to as a "target tracking device"). Figure 15 , the target tracking device 1500 may include a first-stage tracker 1510 and a second-stage tracker 1520. Specifically, the first-stage tracker 1510 may be configured to: obtain a first depth feature of the target object area image and a second depth feature of the search area image in the image; obtain a global response map based on the first depth feature and the second depth feature, and obtain a first-stage target object bounding box prediction result based on the global response map; update the second depth feature based on the first-stage target object bounding box prediction result. The second-stage tracker 1530 may be configured to: obtain multiple local feature blocks based on the first depth feature, obtain a local response map based on the multiple local feature blocks and the updated second depth feature, and obtain a second-stage target object bounding box prediction result based on the local response map. Since the contents involved in the above operations have been introduced in the process of describing the target tracking method according to the present invention, they will not be repeated here for the sake of brevity.

[0070] The above has been referenced Figures 1 to 15The target tracking method and target tracking device according to the embodiments of the present invention are described. However, it should be understood that: Figure 15 Each unit in the apparatus shown (e.g., the first stage tracker 1510 and the second stage tracker 1520) can be configured as software, hardware, firmware, or any combination thereof to perform specific functions. For example, these units can correspond to dedicated integrated circuits, pure software codes, or modules that combine software and hardware. As an example, refer to Figure 15 The device described may be a PC computer, a tablet device, a personal digital assistant, a smartphone, a web application, or other device capable of executing program instructions, but is not limited thereto.

[0071] It should be noted that, although the target tracking device 1500 is divided into units for performing corresponding processing in the above description, it is clear to those skilled in the art that the processing performed by each unit can also be performed without any specific unit division or clear demarcation between the units in the target tracking device. Figure 15 The described apparatus is not limited to include the units described above, but other units (eg, storage unit, data processing unit, etc.) may be added as needed, or the above units may be combined.

[0072] In addition, the target tracking method according to the present invention can be recorded in a computer-readable recording medium. Specifically, according to the present invention, a computer-readable recording medium having program instructions recorded thereon can be provided, which, when executed by a processor, can cause the processor to execute the target tracking method as described above. Examples of computer-readable recording media may include magnetic media (such as hard disks, floppy disks, and tapes); optical media (such as CD-ROMs and DVDs); magneto-optical media (such as optical disks); and hardware devices specially configured to store and execute program instructions (such as read-only memory (ROM), random access memory (RAM), flash memory, etc.). In addition, according to the present invention, an electronic device including a processor and a memory can be provided, in which program instructions are stored, wherein when the program instructions are executed by a processor, the processor is caused to execute the target tracking method as described above. Examples of program instructions include, for example, machine code generated by a compiler and files containing high-level code that can be executed by a computer using an interpreter.

[0073] In addition, according to the exemplary embodiment of the present application, some operations in the target tracking method can be implemented by software, some operations can be implemented by hardware, and furthermore, these operations can be implemented by a combination of software and hardware.

[0074] While the invention has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the claims.

Claims

1. A method for performing target object tracking, comprising: Acquire a first depth feature of a target object area image and a second depth feature of a search area image in the image; Obtaining a global response map according to the first depth feature and the second depth feature, and obtaining a first-stage target object bounding box prediction result according to the global response map; Update the second depth feature according to the target object bounding box prediction result in the first stage; A plurality of local feature blocks are obtained based on the first depth feature, a local response map is obtained according to the plurality of local feature blocks and the updated second depth feature, and a second-stage target object bounding box prediction result is obtained according to the local response map.

2. The method according to claim 1, wherein Obtaining a plurality of local feature blocks based on the first depth feature includes: dividing the first depth feature or a third depth feature obtained by further extracting features from the first depth feature into blocks to obtain a plurality of local feature blocks, Obtaining a local response map based on the multiple local feature blocks and the updated second depth feature includes: performing block correlation on the multiple local feature blocks and the updated second depth feature or a fourth depth feature obtained by further feature extraction of the updated second depth feature to obtain a local response map.

3. The method according to claim 2, wherein: Performing block correlation on the multiple local feature blocks with the updated second depth feature or the fourth depth feature obtained by further feature extraction on the updated second depth feature to obtain the local response map includes: performing block correlation on each local feature block in the multiple local feature blocks with the updated second depth feature or the fourth depth feature to obtain multiple local sub-response maps, and fusing the multiple local sub-response maps to obtain the local response map; Among them, obtaining the second-stage target object bounding box prediction result according to the local response map includes: predicting the position offset and size offset of the second-stage target object bounding box according to the local response map, and obtaining the second-stage target object bounding box prediction result according to the predicted position offset and size offset.

4. The method according to claim 3, wherein: Fusing the multiple local sub-response maps to obtain the local response map includes: classifying each local feature block among the plurality of local feature blocks as a target object feature block or a background feature block; The local response map is obtained by fusing the local sub-response map corresponding to the target object feature block and the sub-response map corresponding to the background feature block.

5. The method according to claim 4, wherein: Classifying each of the plurality of local feature blocks as a target object feature block or a background feature block comprises: The initial target object bounding box calibrated on the target object sub-image is used as the classification basis. According to the ratio of the overlapping area between each local feature block and the initial target object bounding box to the local feature block, each local feature block is classified as a target object feature block or a background feature block.

6. The method of claim 3, wherein: The prediction results of the target object bounding box in the first stage and the prediction results of the target object bounding box in the second stage both include the position information and size information of the target object bounding box. The position offset is the coordinate offset between the center position coordinates of the target object bounding box in the second stage and the center position coordinates of the target object bounding box in the first stage. The size offset is the size offset between the target object bounding box in the second stage and the pre-specified target object bounding box. The second-stage target object bounding box prediction results obtained based on the predicted position offset and size offset include: When the sum of the absolute values ​​of the coordinate offsets is greater than a preset threshold, the target object bounding box prediction result of the first stage is used as the target object bounding box prediction result of the second stage; When the sum of the absolute values ​​of the coordinate offsets is less than or equal to the preset threshold, the second-stage target object bounding box prediction result is obtained by adding the center position of the first-stage target object bounding box to the predicted position offset and adding the pre-specified size of the target object bounding box to the predicted size offset.

7. The method according to claim 6, wherein: The first-stage target object bounding box prediction results obtained based on the global response map include: The position with the largest score in the global response map is used as the position information included in the first-stage target object bounding box prediction result, and the size of the target object bounding box predicted on the image before the current frame image is used as the size information included in the first-stage target object bounding box prediction result.

8. The method of claim 2, wherein: Dividing the first depth feature or the third depth feature obtained by further feature extraction of the first depth feature into blocks to obtain a plurality of local feature blocks includes: dividing the first depth feature or the third depth feature into blocks according to one of the following three division methods: The division method between local feature blocks does not overlap with each other; A division method in which adjacent local feature blocks overlap with each other; A partitioning method based on a predetermined block distribution.

9. An apparatus for performing target object tracking, comprising: The first stage tracker is configured to: obtain a first depth feature of a target object area image and a second depth feature of a search area image in the image; Obtaining a global response map according to the first depth feature and the second depth feature, and obtaining a first-stage target object bounding box prediction result according to the global response map; Update the second depth feature according to the target object bounding box prediction result in the first stage; The second stage tracker is configured to: obtain multiple local feature blocks based on the first depth feature, obtain a local response map according to the multiple local feature blocks and the updated second depth feature, and obtain a second stage target object bounding box prediction result according to the local response map.

10. The device according to claim 9, wherein The second stage tracker is further configured as follows: The first depth feature or the third depth feature obtained by further feature extraction of the first depth feature is divided into blocks to obtain a plurality of local feature blocks, Obtaining a local response map based on the multiple local feature blocks and the updated second depth feature includes: performing block correlation on the multiple local feature blocks and the updated second depth feature or a fourth depth feature obtained by further feature extraction of the updated second depth feature to obtain a local response map.

11. The device according to claim 9, wherein The second stage tracker is further configured as follows: performing block correlation on each of the plurality of local feature blocks and the updated second depth feature or the fourth depth feature to obtain a plurality of local sub-response maps, and fusing the plurality of local sub-response maps to obtain the local response map; The position offset and size offset of the second-stage target object bounding box are predicted according to the local response map, and the second-stage target object bounding box prediction result is obtained according to the predicted position offset and size offset.

12. The device according to claim 11, wherein The second stage tracker is further configured as follows: classifying each local feature block among the plurality of local feature blocks as a target object feature block or a background feature block; The local response map is obtained by fusing the local sub-response map corresponding to the target object feature block and the sub-response map corresponding to the background feature block.

13. The device of claim 12, wherein: The second stage tracker is further configured as follows: The initial target object bounding box calibrated on the target object sub-image is used as the classification basis. According to the ratio of the overlapping area between each local feature block and the initial target object bounding box to the local feature block, each local feature block is classified as a target object feature block or a background feature block.

14. The device according to claim 11, wherein The prediction results of the target object bounding box in the first stage and the prediction results of the target object bounding box in the second stage both include the position information and size information of the target object bounding box. The position offset is the coordinate offset between the center position coordinates of the target object bounding box in the second stage and the center position coordinates of the target object bounding box in the first stage. The size offset is the size offset between the target object bounding box in the second stage and the pre-specified target object bounding box. The second stage tracker is further configured as follows: When the sum of the absolute values ​​of the coordinate offsets is greater than a preset threshold, the target object bounding box prediction result of the first stage is used as the target object bounding box prediction result of the second stage; When the sum of the absolute values ​​of the coordinate offsets is less than or equal to the preset threshold, the second-stage target object bounding box prediction result is obtained by adding the center position of the first-stage target object bounding box to the predicted position offset and adding the pre-specified size of the target object bounding box to the predicted size offset.

15. The apparatus of claim 14, wherein: The first stage tracker is further configured as follows: The position with the largest score in the global response map is used as the position information included in the first-stage target object bounding box prediction result, and the size of the target object bounding box predicted on the image before the current frame image is used as the size information included in the first-stage target object bounding box prediction result.

16. The apparatus of claim 10, wherein: The second stage tracker is further configured to divide the first depth feature or the third depth feature into blocks according to one of the following three division methods: The division method between local feature blocks does not overlap with each other; A division method in which adjacent local feature blocks overlap with each other; A partitioning method based on a predetermined block distribution.

17. An electronic device comprising a processor and a memory, wherein: Program instructions are stored in the memory, wherein when the program instructions are executed by the processor, the processor is prompted to perform the method according to any one of claims 1 to 8.

18. A computer-readable recording medium, wherein: The computer-readable recording medium has program instructions recorded thereon, wherein when the program instructions are executed by a processor, the processor is prompted to execute the method according to any one of claims 1 to 8.

Images