A video multi-plane tracking method based on occlusion reasoning

Abstract

The application discloses a video multi-plane tracking method based on occlusion reasoning, and tracks multiple plane targets in a video, and obtains the positions of the planes according to the appearance features of the planes in the occlusion area, predicts the positions and postures of the plane targets at the current time according to the historical trajectories of the plane targets, models the occlusion area by combining the two prediction results, and further utilizes a double-flow attention network to fuse the prediction results of the two to obtain a final tracking result. Compared with the prior art, the application realizes high-precision tracking of multiple high-degree-of-freedom plane targets.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a video multi-plane tracking method based on occlusion reasoning. Background Technology

[0002] Video tracking, also known as motion analysis, is a technique involving the automatic detection and tracking of moving objects in a video stream. The history of video tracking dates back to the 1960s, when researchers began developing algorithms to track objects in surveillance videos. In the 1970s and 80s, video tracking technology was primarily used for military applications, such as tracking enemy vehicles and aircraft. However, as the technology became more advanced and affordable, it began to be used in a wider range of applications, including traffic monitoring, motion analysis, and animal behavior research. Today, video tracking is used in various fields such as security monitoring, scientific research, and entertainment. It has become an important tool for analyzing human and animal behavior, detecting and preventing crime, and improving the efficiency of industrial processes.

[0003] Early video tracking efforts primarily focused on tracking single targets. Bolme et al. proposed a method based on discriminative correlation filters to locate targets in tracking videos, achieving optimal localization by finding the minimum difference in filter features between the search image and the template image. SiamFC further proposed a deep tracking network based on convolutional neural networks, which utilizes stochastic gradient descent to optimize the network and minimize the differences between convolutional features.

[0004] Besides tracking single targets, multi-target tracking is also an important area of ​​video object tracking. Due to the need for downstream applications to track multiple targets simultaneously, such as vehicles, aircraft, and people, multi-target tracking has attracted considerable research attention. Current mainstream methods employ a detection-then-tracking framework. First, a detector locates targets belonging to a specific category in each frame of the image. Then, multiple targets are matched across different frames of the video to obtain the spatiotemporal trajectory of each target. A core issue is how to correlate targets when the location / appearance of the same person changes at different locations in the video. DeepSort proposes using an association network to extract features for each target, constructing an N×M matching matrix by calculating the similarity between features, and finally using the Hungarian matching algorithm to match the N×M targets detected in two images respectively. FairMOT, considering that the training process of existing algorithms couples the detection and association networks, proposes a multi-task learning framework that simultaneously trains the detection and association networks and designs an inconsistency loss based on the difficulty of task optimization.

[0005] However, existing tracking algorithms are limited to locating common targets, such as those located by horizontal and vertical rectangular boxes. They aim to implicitly locate targets by mining their appearance information, which limits their ability to understand the mutual motion between planes in a scene.

[0006] Unlike video object tracking, tracking multiple planar objects (MPOT) requires tracking not only the position of planar objects but also the orientation of the planes. Unlike a rectangle, which can be described by two nodes (top left and bottom right), the orientation of a plane requires at least four independent nodes, thus offering greater degrees of freedom. Furthermore, MPOT introduces motion between multiple planes, necessitating the handling of complex occlusion between these objects, which further increases the challenge of estimating multiple nodes on the planes. Summary of the Invention

[0007] The purpose of this invention is to propose a video multi-plane tracking method based on occlusion reasoning. It uses the occlusion region obtained by fusing position and pose reasoning as a priori, and adaptively uses a dual-stream attention network to fuse the prediction results of the two to obtain the final multi-plane target tracking result.

[0008] This invention is achieved using the following technical solution:

[0009] A video multi-plane tracking method based on occlusion inference, characterized in that the method includes:

[0010] Step 1, construct the memory pool. The image and the corresponding prediction results are stored in the database. For the result of the o-th planar target after filtering, all prediction results in the T frames of the video are merged, where T is the number of video frames processed;

[0011] Step 2: Use a feature extractor to extract image features f of the current frame and the previous T frames of the video from the memory pool, respectively. i Mask features f of the first T frames of the video m And extracting features f from the heatmap of ordered nodes p The above features are fused into initial appearance features using a pre-trained ResNet50, which are then used in subsequent steps 3 and 4 to obtain prediction results based on appearance features.

[0012] Step 3: Apply a two-stream network containing a mask prediction branch and a planar node prediction branch to the prediction range to obtain the mask prediction result and the heatmap prediction result of four ordered nodes respectively; wherein, the mask prediction result represents the predicted position of multiple planar targets, and the ordered nodes represent the predicted pose of the planar targets.

[0013] Step 4: Based on the two initial tracking results obtained in Step 3 and the occluded part inferred from the historical motion trajectory of the planar target, model the occlusion area of ​​the planar target;

[0014] Step 5: Using the obtained occlusion region as guidance, a multi-head attention mechanism is employed to refine the features and obtain occlusion-resistant features X from the fused features. occ ;

[0015] Step 6: In the mask and ordered node heatmap of the occluded region of the given planar target, feature X is... occ The two-stream network from the appearance feature perception stage is input again to obtain a refined prediction result for the 0th planar target, including ordered nodes. and mask

[0016] Step 7: Based on the prediction results of the current frame image obtained in Step 6, for each mask... With each heatmap Calculate the confidence level M of the mask conf The confidence level P of the heatmap conf Based on the confidence level, select credible prediction results and update the memory pool; return to step 1 until all frames of the video have been processed.

[0017] The beneficial effect of this invention is that it achieves high-precision tracking of multiple high-degree-of-freedom planar targets. Attached Figure Description

[0018] Figure 1 This is an overall flowchart of a video multi-plane tracking method based on occlusion inference according to the present invention;

[0019] Figure 2 This is a detailed flowchart of a video multi-plane tracking method based on occlusion reasoning according to the present invention; (2a) detailed flowchart of step 1, (2b) detailed flowchart of step 2, (2c) detailed flowchart of step 3, (2d) detailed flowchart of step 4, and (2e) detailed flowchart of step 5.

[0020] Figure 3 This is a diagram of the target planar ordered node encoder architecture according to an embodiment of the present invention;

[0021] Figure 4 This is a diagram of the reasoning-tracking framework according to an embodiment of the present invention;

[0022] Figure 5 This is an example diagram showing the overall tracking metrics results on the POT210 and POT280 benchmark datasets and the algorithm performance under different challenge conditions in this embodiment of the invention.

[0023] Figure 6It is a schematic diagram of a planar target, including (a) the target mask on the POT210 benchmark dataset, (b) the target mask on the POT280 benchmark dataset, and (c) planar nodes. Detailed Implementation

[0024] The present invention will be further described in detail below with reference to the accompanying drawings.

[0025] This invention addresses the limitation of existing video planar tracking methods, which are confined to locating common targets. It aims to implicitly locate targets by mining their appearance information, which restricts the ability to understand the mutual motion between planes in a scene. This invention primarily models occluded regions and locates planar targets by analyzing the differences between predictions based on their historical trajectories and appearance features. Guided by prior knowledge of the occluded region's location, it further utilizes a two-stream attention network to fuse these predictions, enabling the tracking of multiple planes in a video.

[0026] Example

[0027] like Figure 1 The diagram shows the overall process of a recyclable item detection method for complex scenarios according to the present invention, which specifically includes the following steps:

[0028] Step 1: For N planar targets in a video containing T frames, use a tracker to continuously track the position changes of the planar targets relative to the initial position, and obtain a quadrilateral mask describing the predicted range of the planar targets. Heatmap of four ordered nodes h and w are the height and width of the image, respectively. A memory pool is constructed within the memory pool. The image and the corresponding prediction results are stored in the database. For the result of the o-th planar target after filtering, all prediction results in the T frames of the video are merged, where T is the number of video frames processed;

[0029] Step 2: Use a feature extractor to extract image features f of the current frame and the previous T frames of the video from the memory pool, respectively. i Mask features f of the first T frames of the video m And extracting features f from the heatmap of ordered nodes p The above features are fused into initial appearance features using a pre-trained ResNet50, which are then used in subsequent steps 3 and 4 to obtain prediction results based on appearance features.

[0030] like Figure 2 As shown, (2a) is the detailed process of step 2, which specifically includes the following steps:

[0031] Step 2-1: Extract image features f of the current frame using an image feature extractor. cur ∈R CHW Then, the image features f of the first T frames of the video in the memory pool are extracted using an image feature extractor, a mask feature extractor, and a planar node feature extractor, respectively. i Mask features f m and ordered node features f p The three features are added together to obtain the predicted target feature f in the memory pool. pre ∈R T×CHW Where H, W, and C represent the height, width, and dimension of the feature, respectively.

[0032] like Figure 3 The figure shows an ordered node encoder for a planar target according to an embodiment of the present invention. To avoid positional bias, the ordered node heatmap features p obtained in step 2.1 are first randomly shuffled by the shift operation of the node encoder. l ∈R HW The order ensures that the encoder processes all nodes; the shift operation is shown in the following formula:

[0033] shift(P, +r) = [P (1+r)％L , ..., P (L+r)％L (1)

[0034] Where L is the total number of ordered nodes in the planar target, and r is a random number and r∈{0,...,L-1};

[0035] Then, using a node encoder, single-layer convolution is applied to extract features from the four ordered nodes of each planar target, and the position encoding features f based on the cosine function are used. emb ∈R L×d The relative position information of the ordered nodes is introduced by summing the features of different ordered nodes; the position encoding feature contains the order information of L ordered nodes, and is represented by a d-dimensional vector respectively.

[0036] Finally, the features of the four ordered nodes are fused by four groups of 7×7 grouped convolutions to capture the long-distance spatial interactions between ordered nodes;

[0037] Step 3: Apply mask prediction branch and planar node prediction branch to the prediction range to obtain mask prediction results and heatmap prediction results of four ordered nodes respectively; where the mask prediction results represent the predicted positions of multiple planar targets, and the ordered nodes represent the predicted poses of the planar targets.

[0038] (2b) is the detailed process of step 3, which specifically includes the following steps:

[0039] Step 3-1: Use the pre-trained ResNet50 neural network to extract the current frame features f from Step 2-1. cur ∈R CHW and predict target features f pre ∈R T×CHW Preliminary predicted appearance features were obtained. Furthermore, regarding appearance features and its high-dimensional appearance feature decoding results;

[0040] Step 3-2: Based on the high-dimensional appearance feature decoding results obtained in Step 3-1, predict ordered nodes: predict the heatmap of ordered nodes, and restore the shuffled ordered node predictions by shift(·, -r); obtain the heatmap of the o-th planar target as the initial tracking result of the tracker by upsampling to the same resolution as the original image; use a convolutional network and a sigmoid activation function to predict the heatmap of ordered nodes.

[0041] Step 3-3: Based on the appearance feature decoding results obtained in Step 3-1, perform mask prediction: Perform 1×1 convolution and softmax function mapping in the foreground / background dimension to accurately locate the mask of the planar object at the pixel level. By aggregating the output of the model, obtain the probability map m of the background. bkg and the probability graph m of each foreground object o As shown in the following formula:

[0042]

[0043]

[0044] Among them, l o This is the model's output for the o-th planar target;

[0045] After passing through the softmax function, the probability segmentation mask of the 0th planar target is obtained. Then, the probability background map and the probability foreground map are merged together as the initial tracking result of the tracker.

[0046] Step 4: Based on the two initial tracking results obtained in Step 3 and the occluded part inferred from the historical motion trajectory of the planar target, model the occlusion area of ​​the planar target;

[0047] (2c) is the detailed process of step 4, which specifically includes the following steps:

[0048] Step 4-1: First, obtain the coordinates of the ordered nodes on the ordered node heatmap, and calculate the homography matrix between the current frame coordinates and the previous frame coordinates: using the motion parameter h p and position parameter h lAs input, a Kalman filter is used for prediction based on historical motion trajectories. In the Kalman filter, the input homography matrix H is decomposed into a similarity transformation H. s and residual transformation H Λ According to h p =(t x , t y The eight stable motion parameters (γ, θ, k1, k2, v1, v2) eliminate the interference caused by the sensitivity problem of the homography matrix, where t x t y Let γ be the translation offset, θ be the scale and angle changes, and k1, k2, v1, v2 be the control matrices H. Λ Parameters;

[0049] Step 4-2: Based on the position of the planar target in the current frame obtained in Step 4-1, and the homography matrix of the 0th planar target in the current frame. Mask the planar target in frame T. Distorted by perspective transformation

[0050] Step 4-3: Based on the appearance features of the two initial tracking results obtained in Step 3-3 and the historical trajectory of the planar target obtained in Step 4-1, model the occlusion region φ of the target according to the difference between the two. o As shown in the following formula:

[0051]

[0052] Where ||·|| is the distance function calculated pixel by pixel, δ i To determine the confidence level of the mask, where i is the index of the planar target, o is the index of the currently predicted planar target, and T is the Tth frame;

[0053] Step 5: Using the obtained occlusion region as guidance, a multi-head attention mechanism is used to refine the features and obtain occlusion-resistant features from the fused features.

[0054] (2d) is the detailed process of step 5, which specifically includes the following steps:

[0055] Step 5-1: Analyze the image features of the current frame. and the distortion features of previous frames To achieve refinement, the two refined features are denoted as f. a and f m Flatten along spatial dimensions The fused feature X is obtained using a multi-head sub-attention mechanism, and then split into appearance features X along the dimension. a and motion characteristics X m As shown in the following formula:

[0056] X = Concat(h1, ..., h) H (5)

[0057]

[0058]

[0059] f = Concat(f) m f a (8)

[0060] in, Let be the learnable matrix for the i-th attention head;

[0061] Step 5-2: Based on the fused feature X, obtain feature X according to the guidance of the occlusion region. occ As shown in the following formula:

[0062] X occ =F ens (F φ (X a ), X m (9)

[0063] Among them, F ens and F φ All are 1×1 convolutional layers, X occ For features robust to occlusion; F φ Additionally, a stitching method is used to model the occluded area φ o As initial weights, they guide feature extraction; specifically, the guidance based on the occlusion region means that during feature processing, the occlusion region is compared with X. a First, they are pieced together, and then fed into the convolutional layer;

[0064] Step 6: In the mask and ordered node heatmap of the occluded region of the given planar target, feature X is... occ The two-stream network from the appearance feature perception stage is input again to obtain a refined prediction result for the 0th planar target, including ordered nodes. and mask

[0065] Step 7: Based on the prediction results of the current frame image obtained in Step 6, for each mask... With each heatmap Calculate the confidence level M of the mask conf The confidence level P of the heatmap conf Based on the confidence level, select credible prediction results and update the memory pool; return to step 1 until all frames of the video have been processed.

[0066] (2e) is the detailed process of step 7, which specifically includes the following steps:

[0067] Step 7-1, calculate each mask Confidence level M conf With each heatmap confidence level P conf As shown in the following formula:

[0068]

[0069]

[0070] Where O represents the total number of planar targets in the current frame;

[0071] Based on the confidence level of the prediction results, each mask is... and each heatmap Planar targets whose confidence scores all exceed a threshold of 0.5 are considered reliable prediction results.

[0072] like Figure 4 As shown in the diagram, the tracking framework of this invention, as a "reasoning-tracking" framework focusing on occlusion regions between planar targets, is summarized as including 1) a memory pool, 2) visual appearance detection, and 3) an occlusion inference stage. This framework is used to implement the overall process of this invention, tracking multiple planar targets in a video. For occlusion regions between planar targets, this invention obtains the position of the planar target based on its appearance information and predicts its current position based on its historical trajectory. The occlusion region is located based on these two prediction results. Furthermore, this invention uses the position of the occlusion region as a priori and adaptively fuses the prediction results using a two-stream attention network to obtain the final tracking result.

[0073] It should be noted that although the present invention has been shown and described with reference to specific exemplary embodiments thereof, those skilled in the art should understand that the present invention is not limited to the above embodiments, and all modifications to the present invention fall within the scope of protection of the present invention.

Claims

1. A method for video multiplane tracking based on occlusion reasoning, the method comprising: The method comprises: Step 1, constructing a memory pool, saving images and corresponding prediction results in the memory pool , wherein, the results of screening the last plane target are combined with all prediction results in the frame images of the video, the number of processed video frames;​ Step 2, image features of the current frame and the previous T frames of the video are extracted from the memory pool respectively by using a feature extractor mask features of the previous T frames of the video and heat map extraction features of the ordered nodes the above features are fused into initial appearance features of the prediction target by using a pre-trained ResNet50, which are used in subsequent Step 3 and Step 4 to obtain the prediction result based on the appearance features; Step 3, the predicted range adopts a double-flow network containing a mask prediction branch and a planar node prediction branch to obtain a mask prediction result and a heat map prediction result of four ordered nodes respectively; wherein the mask prediction result represents the predicted position of multiple planar targets, and the ordered node represents the predicted attitude of the planar target; Step 4, based on the two initial tracking results obtained in step 3 and the historical motion trajectory of the planar target, infer the occluded part, and model the occluded area of the planar target; Step 5, using the obtained occlusion area as guidance, a multi-head attention mechanism is used to refine the features from the fused features to obtain occlusion-robust features ; Step 6: obtaining the feature in the given plane target's occluded region mask and ordered node heat map Again input the double flow network of the appearance feature perception stage, and obtain the refined first plane target prediction result including the ordered node and the mask , to obtain the prediction result of all plane targets in the current frame image, including the mask of the quadrilateral describing the prediction range of the plane target and the heat map of the four ordered nodes , wherein, and are the height and width of the image, is the total number of plane targets in the current frame; Step 7, each mask in the prediction result of the current frame image obtained based on step 6 each heat map of Calculate the confidence of the mask and the confidence of the heat map According to the confidence, filter out the credible prediction result, update the memory pool; return to step 1 until all frames of the video are processed.

2. The occlusion-reasoning-based video multiplane tracking method of claim 1, wherein, The step 2 comprises the following processing: Step 2-1, extracting image features of the current frame by the image feature extractor ; and extracting image features of the video frames in the memory pool by the image feature extractor, respectively adding the four features to obtain initial appearance features of the prediction target in the memory pool where H, W and C represent height, width and dimension of the features, respectively.​​​ 3. The method of claim 2, wherein, The step 3 comprises the following processing: Step 3-1, the current frame features extracted from step 2-1 and the predicted initial appearance target features to get the preliminary predicted appearance features , and further to get the appearance features and the high-dimensional appearance feature decoding results thereof; Step 3-2: Order node prediction based on the high-dimensional appearance feature decoding result obtained in step 3-1: predict the heat map of the order node, and recover the order node prediction by up-sampling to the same resolution as the original image to obtain the heat map of the oth plane target as the initial tracking result of the tracker; Step 3-3, mask prediction based on the appearance feature decoding result obtained in step 3-1: 1x1 convolution and softmax function mapping are performed in the foreground / background dimension to accurately locate the mask of the planar object at the pixel level, and by aggregating the output of the model, a probability background map is obtained and the probability foreground map of the planar target as shown in the following formula: ； ； wherein, is the output of the model for the first planar target; Obtain the probability segmentation mask of the oth planar target, and then combine the probability background map and the probability foreground map as the initial tracking result of the tracker.

4. The occlusion-reasoning-based video multiplane tracking method of claim 3, wherein, The step 4 comprises the following processing: Step 4-1, first get the coordinates of the ordered nodes on the ordered node heat map, and calculate the homography matrix between the current frame coordinates and the previous frame coordinates: with motion parameters and position parameters as input, predict based on historical motion trajectory using Kalman filter; In the kalman filter, the input homography matrix H is decomposed into a similarity transformation S and a residual transformation R as follows: H = S * R where S is a 3x3 matrix of the form S = [1 + s x s x s x 1 + s y s y s y 0 0 0] where s x and s y are the scale changes in the x and y directions respectively, and R is a 3x3 matrix of the form R = [1 + r x r x r x r y r y r y r z r z r z] where r x, r y, and r z are the rotation angles around the x, y, and z axes respectively. The homography matrix H is decomposed into a similarity transformation S and a residual transformation R as follows: H = S * R where S is a 3x3 matrix of the form [1 + s x s x s x 1 + s y s y s y 0 0 0] where s x and s y are the scale changes in Step 4-2, get the position of the planar object in the current frame and the homography matrix of the oth planar object in the current frame according to step 4-1 , the oth planar object in the current frame is determined , the mask of the planar object in the current frame is determined , the mask of the planar object in the current frame is determined ; Step 4-3 - Appearance features based on the initial tracking results obtained by merging the probability background map and the probability foreground map obtained in Step 3-3 and the historical trajectory warped to the current frame in Step 4-2 , modeling the occluded region of the target according to the difference between the two as shown in the following equation: ； wherein, is a pixel-wise computed distance function, is a confidence of the decision mask, wherein i is the index of the planar object, o is the index of the current predicted planar object, and T is the T-th frame.

5. The occlusion-reasoning-based video multiplane tracking method of claim 1, wherein, The step 5 comprises the following processing: Step 5-1, image features of the current frame and the warping features of the previous frame Implement the refinement process, and the two features after refinement are respectively denoted as and ; along the spatial dimension , the fused features X are obtained by using the multi-head sub-attention mechanism, and are split into appearance features along the dimension, as shown in the following formula: ； ； ； ； wherein, , , is a learnable matrix for the i-th attention head; Step 5-2, obtain features from the occlusion region guidance As shown in the following formula: ； wherein, and are convolutional layers, is a feature robust to occlusion, are appearance features obtained by splitting feature X along dimensions .

6. The video multi-planar tracking method based on occlusion inference according to claim 1, wherein, The step 7 comprises the following processing: Compute the confidence of each mask The confidence of each heat map As shown in the following formula:​​ ； ； wherein, is the total number of planar objects in the current frame; Based on the confidence of the prediction results, each mask and each heat map whose confidence exceeds a threshold are considered as a confident prediction result.

Images