Method and system for tracking arbitrary quadrilateral plane targets in video based on depth and shallowness model integration

Abstract

The present invention discloses a method for tracking arbitrary quadrilateral plane targets in a video based on the integration of deep and shallow models, comprising: firstly, integrating two shallow target tracking methods and a deep general target tracker into a unified plane target tracking framework, wherein the shallow target tracking method includes a plane target tracker and a general target tracker; finally, refining the fused tracking results using a shot boundary detection method based on grid motion statistics. Also disclosed is a system, device and storage medium for tracking arbitrary quadrilateral plane targets in a video based on the integration of deep and shallow models. The present invention proposes an integration method that does not require manual intervention and fuses the results of different trackers by comparing the differences between different trackers. Different tracking algorithms can simultaneously give play to their respective advantages, overcome the shortcomings of other algorithms, and greatly improve the performance of plane target tracking.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision target tracking, and in particular to a method and a system for tracking arbitrary quadrilateral plane targets in a video based on depth and shallowness model integration. Background Art

[0002] The purpose of tracking arbitrary quadrilateral planar targets in videos is to accurately and robustly track planar targets without any prior knowledge except the four corner coordinates of the first frame, which is of great significance for virtual advertising or other realistic applications.

[0003] Visual object tracking is a research hotspot in the field of computer vision. According to the tracking target, existing visual tracking methods can be divided into general object tracking and planar object tracking. For general object tracking, the tracking target is the rectangular area contained in the minimum bounding box of the object, and the visual content of the object itself does not change much in adjacent frames. However, for planar object tracking, the target is usually an irregular quadrilateral area surrounded by four arbitrary points. Sometimes, the tracking target can be a computer screen with constantly changing content, or even not a "real object", including the following situations: (1) any object, even a randomly specified area, can be the tracking target; (2) the size of the target gradually decreases; (3) the view outside the target window is constantly changing. Therefore, although deep learning-based visual object tracking technology has made great progress, most research focuses on general object tracking and is not suitable for planar object tracking.

[0004] Therefore, there is an urgent need to provide a new method for tracking arbitrary quadrilateral planar targets in videos to solve the above problems. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method, system, device and storage medium for tracking arbitrary quadrilateral planar targets in a video based on depth and shallow model integration, which can greatly improve the performance of planar target tracking.

[0006] In order to solve the above technical problems, the first technical solution adopted by the present invention is to provide a method for tracking an arbitrary quadrilateral plane target in a video based on depth model integration, comprising:

[0007] Firstly, two shallow target tracking methods and a deep general target tracker are integrated into a unified planar target tracking framework. The shallow target tracking method includes a planar target tracker and a general target tracker.

[0008] Finally, the fused tracking results are refined using a shot boundary detection method based on grid motion statistics.

[0009] In a preferred embodiment of the present invention, a method for tracking an arbitrary quadrilateral plane target in a video based on depth and shallow model integration specifically includes the following steps:

[0010] Assume that the four corner coordinates of any planar quadrilateral target R in a certain frame of the video in the coordinate system t are The coordinates of the four corners of R in the coordinate system t+1 are

[0011] S1: Use the planar target tracker Gracker to track R, and the result is expressed as where i∈[1,4];

[0012] S2: With each corner of R as the center, a rectangle with a length of r is established, and the four small rectangular areas are tracked using fDSST. Similarly, the center of the tracked rectangle in frame t+1 is also the corner of the tracked plane object, denoted as where i∈[1,4];

[0013] S3: Use UDT to track the bounding box of the established rectangle, and use the bounding boxes of frames t and t+1 to calculate the homography matrix P u , the UDT tracking result is

[0014] S4: Calculate the Euclidean distance between all pairs of different tracking results: d GF ,d FU ,d GU :

[0015]

[0016]

[0017]

[0018] If d GF >T1 and d FU >T1 and d GU >T1, T1 is a predefined threshold, then the UDT tracking result is regarded as the integrated result; otherwise, the comparison d GF ,d FU ,d GU The smallest value is selected and the average of the two algorithm results constituting the value is used as the final result.

[0019] S5: If there is a lens change in frame t+1, collect the tracking results will be replaced by in It is the result of tracking any quadrilateral planar target using the SIFT matching tracking method.

[0020] In order to solve the above technical problems, the second technical solution adopted by the present invention is to provide a system for tracking arbitrary quadrilateral plane targets in a video based on depth model integration, comprising:

[0021] Shallow object tracking module, used to track any quadrilateral in the video using a shallow object tracker;

[0022] Deep general object tracking module, used to track any quadrilateral in the video using a deep general object tracker;

[0023] A tracking result integration fusion module is used to fuse the tracking results obtained by the shallow target tracking module and the deep general target tracking module into a unified planar target tracking framework;

[0024] The integrated result refinement module is used to refine the fused tracking results using a shot boundary detection method based on grid motion statistics.

[0025] In a preferred embodiment of the present invention, the shallow target tracking module includes a planar target tracker and a general target tracker.

[0026] In a preferred embodiment of the present invention, the judgment condition for the integrated result refinement module to refine the fused tracking result is to first detect whether there is a shot change in the t+1 frame.

[0027] Further, the method for detecting whether there is a lens change in frame t+1 is:

[0028] Given frames t-1, t, t+1, use the GMS algorithm to calculate the matching key points of [t-1, t] and [t, t+1], and the number of matches is recorded as N t-1,t and N t,t+1 ; Set the predefined threshold T2, if |N t-1,t -N t,t+1 |>T2, indicating that there is a lens change in frame t+1.

[0029] To solve the above technical problems, the third technical solution adopted by the present invention is: to provide a device for tracking arbitrary quadrilateral plane targets in a video based on depth and shallow model integration, comprising a memory and a processor, the memory being used to store at least one program, and the processor being used to load the at least one program to execute any of the methods described above.

[0030] The fourth technical solution adopted by the present invention is: to provide a storage medium, in which processor-executable instructions are stored, and the processor-executable instructions are used to execute any of the methods described above when executed by the processor.

[0031] The beneficial effects of the present invention are:

[0032] (1) The present invention proposes an integration method that does not require manual intervention and fuses the results of different trackers by comparing the differences between them. Different tracking algorithms can simultaneously play their respective advantages, overcome the shortcomings of other algorithms, and greatly improve the performance of planar target tracking.

[0033] (2) Since high-quality labeled planar target tracking data is difficult to obtain, all tracking methods selected in the framework of the present invention do not require supervised training;

[0034] (3) A large number of qualitative and quantitative experimental results on a large-scale real TV program dataset verify the effectiveness of the integration method. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of a method for tracking an arbitrary quadrilateral plane target in a TV program based on depth and shallowness model integration of the present invention;

[0036] Figure 2 is a schematic diagram of comparing different trackers used in the integrated framework of the present invention on the constructed TV program dataset through ablation study;

[0037] Figure 3 It is a comparative schematic diagram of target tracking on a constructed TV program data set using the present invention and two most advanced general target tracking methods based on supervised deep learning, SiamRPN and SiamMask;

[0038] Figure 4 It is a block diagram of the system for tracking any quadrilateral plane target in a TV program based on the integration of depth and shallow models. DETAILED DESCRIPTION

[0039] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0040] See also Figure 1 , the embodiment of the present invention includes:

[0041] A method for tracking arbitrary quadrilateral planar targets in videos based on deep and shallow model integration is firstly to integrate two shallow target tracking methods and a deep general target tracker into a unified planar target tracking framework, wherein the shallow target tracking method includes a planar target tracker and a general target tracker; finally, the fused tracking results are refined using a shot boundary detection method based on grid motion statistics. The specific steps include:

[0042] Given a TV program V, it has m frames {I1, ..., I m}, and in the first frame I1 there is an arbitrary quadrilateral plane object of interest. It consists of four corner coordinates {(x i ,y i )} i∈ [ 1,4 ], the goal of tracking any quadrilateral plane target is to predict the coordinates of R represented in all frames, such as where j is the frame index.

[0043] Assume that the four corner coordinates of any planar quadrilateral target R in a certain frame of the video in the coordinate system t are The coordinates of the four corners of R in the coordinate system t+1 are

[0044] S1: Use the planar target tracker Gracker to track R, and the result is expressed as , where i∈[1,4];

[0045] Gracker uses graphs to model planar objects. The vertices of the graph are generated by a reliable automatic selection mechanism. First, the SIFT response is calculated for each pixel in the video frame R, and then R is divided into N grids. The largest SIFT response is taken from each grid as the key point of the grid. These key points are used as nodes of the graph, and then their descriptors are calculated as the attributes of the node. The edges of the graph are constructed using Delaunay triangulation, which has the advantages of translation, scaling and rotation invariance.

[0046] Gracker is based on graphics and transforms the task of plane objects into a unified geometric matching framework. In general, the algorithm can track the target stably, but the tracking box will be greatly deformed in fast camera switching.

[0047] S2: With each corner of R as the center, a rectangle with a length of r is established, and the four small rectangular areas are tracked using fDSST. Similarly, the center of the tracked rectangle in frame t+1 is also the corner of the tracked plane object, denoted as , where i∈[1,4];

[0048] The correlation filter-based tracker used in the present invention is the fast resolution scale space tracker fDSST, which is a real-time version of the DSST tracker that won the VOT2014 award. fDSST is a general object bounding box tracker that cannot directly track planar targets. To address this problem, this step uses planar target tracking as a corner tracking task.

[0049] fDSST uses PCA-HOG for image representation and learns separate correlation filters online for explicit translation and scale estimation. Specifically, given a new frame, the center position of the target is first evaluated using a standard position filter, and then features are extracted using pixel blocks of different sizes centered on the target to accurately estimate the target scale. Unlike the exhaustive strategy, the size of the search area is reduced due to the determination of the position (if the target center is determined, there is no need to search for areas that deviate from the target center). The algorithm is prone to offset in videos with large lens changes.

[0050] S3: Use UDT to track the bounding box of the established rectangle, and use the bounding boxes of frames t and t+1 to calculate the homography matrix P u , the UDT tracking result is

[0051] The example of the present invention adopts the excellent unsupervised deep learning-based tracker UDT, builds a framework based on the twin correlation filter network, and uses unlabeled original video sequences for training. At the same time, it proposes a multi-frame verification and a cost-sensitive loss to improve the unsupervised learning performance. UDT achieves the baseline accuracy of a fully supervised tracker without labeled training videos.

[0052] UDT is also a general object bounding box tracker. Unlike fDSST, this step constructs a bounding box based on the four corners of the first frame, and then uses UDT to track this bounding box.

[0053] S4: Calculate the Euclidean distance between all pairs of different tracking results: d GF ,d FU ,d GU :

[0054]

[0055]

[0056]

[0057] If d GF >T1 and d FU >T1 and d GU >T1, where T1 is a predefined threshold, then the UDT tracking results are considered as integrated results, for example, i∈[1,4]. This indicates that there is a large gap between the tracking results. Therefore, the UDT result is selected as the final result because the tracking effect of UDT is relatively stable compared with other algorithms. Otherwise, compare d GF ,d FU ,d GUThe smallest value is selected and the average of the two algorithm results constituting the value is used as the final result.

[0058] S5: Refinement through shot boundary detection: All the tracking methods mentioned in the above integration steps will perform worse if there is a shot change. For example, if the shot changes quickly, some algorithms may lose tracking in subsequent frames, or the tracking frame may shift away from the original target, which seriously affects the tracking effect. To solve this problem, first detect whether there is a shot change in frame t+1. Specifically, given frames t-1, t, t+1, use the GMS algorithm to calculate the matching key points of [t-1, t] and [t, t+1]. The number of matches is recorded as N. t-1,t and N t,t+1 ; Set the predefined threshold T2, if |N t-1,t -N t,t+1 |>T2, indicating that there is a lens change in frame t+1.

[0059] If there is a lens change in frame t+1, the tracking results are collected will be replaced by in It is the result of tracking any quadrilateral plane target using SIFT matching tracking method. This algorithm rarely loses frames or has serious deviations in tracking position. The steps of SIFT matching tracking method are: first detect SIFT key points of frame t and frame t+1 respectively; then use RANSAC algorithm to calculate homography matrix based on SIFT pairs matched between two frames; finally, the tracking result can be calculated as:

[0060]

[0061] The following experimental process was carried out using the method described in the present invention:

[0062] A. Experimental Environment

[0063] 1) Dataset: The evaluation dataset comes from Hunan Satellite TV, with 200 1080p videos covering different scenes. All frames are manually marked with four points, and the irregular quadrilateral formed by them is the tracking target.

[0064] 2) Evaluation Metrics: According to the official protocol, the tracking method is evaluated by calculating the mean square error (MSE), which is defined as follows:

[0065]

[0066] Where N is the number of test TV program video clips, It's a video V n The number of frames in the clip; and are the predicted coordinates and true coordinates of the planar object R. Obviously, a smaller MSE means better tracking performance.

[0067] B. Implementation Details

[0068] All experiments are conducted on a Lenovo PC with 64G RAM. For Gracker, fDSST, and UDT trackers, the frames are resized to 720p. To speed up the refinement process, the frames are resized to 360p. In fDSST, the side length of the square is set to 25 pixels. The UDT model is pre-trained on ILSVRC 2015 without labels. Based on experience, the thresholds st1 and t2 are set to 1000 and 100, respectively.

[0069] C. Ablation studies

[0070] In this section, an ablation study is performed to analyze the effectiveness of the proposed components. The results are shown in Tables 1 and Figure 2 The experimental results show that the proposed integrated method can make different tracking algorithms play their respective advantages while overcoming the shortcomings of other algorithms, thus achieving the best performance. The results obtained by graphic matching in some TV videos (the surface of the tracking target is smooth and there is no clear texture feature) are completely wrong.

[0071] Table 1: Ablation study, tracking error on TV show dataset

[0072]

[0073] D. Comparison with State-of-the-Art Deep Learning-Based General Object Tracking Methods

[0074] The proposed method is compared with two state-of-the-art supervised deep learning based general object tracking methods SiamRPN and SiamMask. The backbone is ResNet50, and the pre-trained model is downloaded from https: / / github.com / STVIR / pysot. The results are shown in Table 2 and Figure 3 As shown in the figure, it can be seen that the performance of SiamMask and SiamRPN is much worse than that of the method described in the present invention. The reason is that the training data of SiamMask and SiamRPN are independent objects (such as cars, people, balls, etc.), and the tracking target in the TV program of the present invention may be an arbitrarily specified irregular area. Therefore, there is a serious semantic gap between these two models and the TV scene of the present invention.

[0075] Table 2: Comparison with the state-of-the-art, tracking error on the constructed TV show dataset

[0076] Methods MSE SiamRPN 148.80 SiamMask 186.10 Ours 2.89

[0077] This paper proposes a shallow and deep model integration method to solve the problem of arbitrary quadrilateral planar target tracking. The graph-based tracker, the correlation filter-based tracker and the unsupervised deep learning-based tracker are integrated into a unified tracking framework. Different tracking algorithms play their respective advantages while suppressing the shortcomings of other algorithms. In addition, a practical shot boundary detection method is used to refine the tracking results. Ablation studies verify the effectiveness of the proposed integration strategy, and the present invention achieves state-of-the-art results on a real TV program dataset.

[0078] See also Figure 4 In an example of the present invention, a system for tracking an arbitrary quadrilateral plane target in a video based on depth model integration is also provided, comprising:

[0079] Shallow object tracking module, used to track any quadrilateral in the video using a shallow object tracker;

[0080] Deep general object tracking module, used to track any quadrilateral in the video using a deep general object tracker;

[0081] A tracking result integration fusion module is used to fuse the tracking results obtained by the shallow target tracking module and the deep general target tracking module into a unified planar target tracking framework;

[0082] The integrated result refinement module is used to refine the fused tracking results using a shot boundary detection method based on grid motion statistics.

[0083] Specifically, the shallow target tracking module includes a planar target tracker and a general target tracker.

[0084] This example provides a system for tracking arbitrary quadrilateral plane targets in videos based on the integration of deep and shallow models. This system can execute a method for tracking arbitrary quadrilateral plane targets in videos based on the integration of deep and shallow models provided by this invention, and can execute any combination of implementation steps of the method example, and has the corresponding functions and beneficial effects of the method.

[0085] An embodiment of the present invention also provides a device for tracking arbitrary quadrilateral plane targets in a video based on depth and shallow model integration, comprising a memory and a processor, wherein the memory is used to store at least one program, and the processor is used to load the at least one program to execute any of the methods described above.

[0086] An embodiment of the present invention further provides a storage medium, in which processor-executable instructions are stored. When the processor-executable instructions are executed by the processor, they are used to execute any of the methods described above.

[0087] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0088] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0089] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0090] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for tracking arbitrary quadrilateral plane targets in a video based on depth and shallow model integration, characterized in that: Firstly, two shallow target tracking methods and a deep general target tracker are integrated into a unified planar target tracking framework. The shallow target tracking method includes a planar target tracker and a general target tracker. Finally, a shot boundary detection method based on grid motion statistics is used to refine the fused tracking results. The method includes the following steps: Assume that the four corner coordinates of any planar quadrilateral target R in a certain frame of the video in the coordinate system t are The coordinates of the four corners of R in the coordinate system t+1 are S1: Use the planar target tracker Gracker to track R, and the result is expressed as where i∈[1,4]; S2: With each corner of R as the center, a rectangle with a length of r is established, and the four small rectangular areas are tracked using fDSST. Similarly, the center of the tracked rectangle in frame t+1 is also the corner of the tracked plane object, denoted as where i ∈ [1, 4]; S3: Use UDT tracking to establish a rectangular bounding box, and use the bounding boxes of frames t and t+1 to calculate the homography matrix P u , the UDT tracking result is i∈[1,4]; S4: Calculate the Euclidean distance between all pairs of different tracking results: d GF d FU d GU : If d GF >T1 and d FU >T1 and d GU >T1, T1 is a predefined threshold, then the UDT tracking result is regarded as an integrated result; otherwise, the comparison d GF d FU d GU The smallest value is selected and the average of the two algorithm results constituting the value is used as the final result. S5: If there is a lens change in frame t+1, collect the tracking results will be replaced by in It is the result of tracking any quadrilateral planar target using the SIFT matching tracking method.

2. The method for tracking arbitrary quadrilateral plane targets in a video based on depth and shallow model integration according to claim 1, characterized in that: The method to detect whether there is a lens change in frame t+1 is: Given frames t-1, t, t+1, use the GMS algorithm to calculate the matching key points of [t-1, t] and [t, t+1], and the number of matches is recorded as N t-1，t and N t，t+1 ; Set the predefined threshold T2, if |N t-1，t -N t，t+1 |>T2, indicating that there is a lens change in frame t+1.

3. The method for tracking arbitrary quadrilateral plane targets in a video based on depth and shallow model integration according to claim 1, characterized in that: The steps of SIFT matching tracking method include: First, the SIFT key points of frame t and frame t+1 are detected respectively; then the homography matrix is ​​calculated based on the matched SIFT pairs between the two frames using the RANSAC algorithm; finally, the tracking result can be calculated as:

4. A system for tracking arbitrary quadrilateral plane targets in a video based on depth and shallow model integration, using the method for tracking arbitrary quadrilateral plane targets in a video based on depth and shallow model integration as claimed in any one of claims 1 to 3, characterized in that: include: Shallow object tracking module, used to track any quadrilateral in the video using a shallow object tracker; Deep general object tracking module, used to track any quadrilateral in the video using a deep general object tracker; A tracking result integration fusion module is used to fuse the tracking results obtained by the shallow target tracking module and the deep general target tracking module into a unified planar target tracking framework; The integrated result refinement module is used to refine the fused tracking results using a shot boundary detection method based on grid motion statistics.

5. The system for tracking arbitrary quadrilateral plane targets in videos based on depth and shallow model integration according to claim 4, characterized in that: The shallow target tracking module includes a planar target tracker and a general target tracker.

6. The system for tracking arbitrary quadrilateral plane targets in videos based on depth and shallow model integration according to claim 4, characterized in that: The judgment condition for the integrated result refinement module to refine the fused tracking result is to first detect whether there is a shot change in the t+1 frame.

7. The system for tracking arbitrary quadrilateral plane targets in videos based on depth and shallow model integration according to claim 6, characterized in that: The method to detect whether there is a lens change in frame t+1 is: Given frames t-1, t, t+1, use the GMS algorithm to calculate the matching key points of [t-1, t] and [t, t+1], and the number of matches is recorded as N t-1，t and N t，t+1 ; Set the predefined threshold T2, if |N t-1，t -N t，t+1 |>T2, indicating that there is a lens change in frame t+1.

8. A device for tracking arbitrary quadrilateral plane targets in a video based on depth and shallow model integration, comprising a memory and a processor, wherein the memory is used to store at least one program, and the processor is used to load the at least one program to execute the method described in any one of claims 1 to 3.

9. A storage medium storing instructions executable by a processor, characterized in that: The processor-executable instructions are used to perform the method according to any one of claims 1 to 3 when executed by the processor.

Images