A method for accurately tracking moving objects in video

Abstract

The present invention discloses a method for accurately tracking moving objects in a video, which mainly comprises the following steps: dividing a video to be detected into frames; obtaining an object segmentation mask for each frame; obtaining an optical flow map for each frame; using the SLIC superpixel segmentation algorithm to segment each frame into 2000 superpixels; using the DBSCAN clustering method to automatically cluster the 2000 superpixels with average optical flow features, determining superpixels belonging to the background, and removing object masks belonging to the background; calibrating the coordinates of a pixel point inside the object to be accurately tracked in the first frame of the video; using optical flow information to bidirectionally propagate and optimize the object mask containing the specified pixel; sending the RGB, HSV and CIELAB color space images of each frame, the optical flow map and the optimized object mask to a shallow fully convolutional network to obtain a final pixel-level segmentation result; this aspect improves the accuracy of accurate tracking of specific moving objects in the video by comprehensively utilizing composite color space features and optical flow information.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and artificial intelligence, and in particular to a method for accurately tracking moving objects in a video. Background Art

[0002] To accurately track specific moving objects in a video at the pixel level, each pixel in each frame must be labeled to indicate whether it belongs to the foreground or background object being tracked. Accurately tracking moving objects in videos plays a vital role in industrial production, social governance, and security, and also serves as a foundation for a variety of subsequent video processing and computer vision applications. Therefore, improving the accuracy of tracking specific moving objects in videos is of great research significance and a pressing practical need.

[0003] Current approaches to accurately tracking video objects include building a graph model based on video pixels and classifying each pixel by minimizing an energy function. Another approach involves manually designing features empirically and extracting relevant features from the video to classify each pixel. Furthermore, with the recent development of deep learning technology, a series of accurate video object tracking techniques based on deep neural networks have also been proposed. Currently, related methods can be divided into two categories: unsupervised and supervised. Unsupervised accurate video object tracking methods primarily track foreground objects using saliency or motion characteristics, without requiring any other information. While these methods are relatively simple to apply, they cannot track specific objects within a video. Supervised accurate video object tracking methods can track specific objects within a video, but often require a complete pixel-level mask of the object in the first frame of the video, which is often difficult to achieve in practical applications.

[0004] Since existing video object precision tracking methods still have certain limitations and are easily affected by object motion, lighting changes, object occlusion, etc., they are still full of challenges in practical applications. Summary of the Invention

[0005] To address the existing shortcomings of accurate tracking of specific moving objects in videos, the present invention provides a method for accurately tracking moving objects in videos. This method utilizes the color space characteristics of the video frame's spatial domain and the optical flow information in the temporal domain. The method modifies the object mask through forward and backward propagation, and then uses a neural network to obtain the final pixel-level tracking results, achieving high accuracy.

[0006] The purpose of the present invention can be achieved by taking the following technical solutions:

[0007] A method for accurately tracking a moving object in a video, characterized by comprising the following steps:

[0008] S1, the video to be detected is divided into frames to obtain the frame images of each frame, which are recorded as I0, I1, I2, ..., I N-1 , where N is the total number of video frames;

[0009] S2. Use pixel-level object segmentation method to detect each frame image and obtain the object mask of each frame, which is recorded as M0, M1, M2, ..., M N-1 ;

[0010] S3, for the i-th frame image I i , select the adjacent I before and after it i-3 , I i-2 , I i-1 , I i+1 , I i+2 and I i+3 The entities in the frame image are respectively i The corresponding dense optical flow map is obtained by combining the calculations, and the average optical flow of each pixel is further calculated to obtain the average optical flow map corresponding to the i-th frame image, which is recorded as the horizontal component optical flow map X i And the vertical component optical flow map Y i ,i∈{0,1,2,…,N-1};

[0011] S4. Use the SLIC superpixel segmentation algorithm to segment each frame image into 2000 superpixels, and record the jth superpixel of the i-th frame image as S ij , where j∈{0, 1, 2, …, 1999};

[0012] S5. Calculate the average optical flow of all pixels in each superpixel, denoted as V ij ;

[0013] S6. For the i-th frame image, the average optical flow V of each superpixel ij The DBSCAN clustering method is used to automatically cluster 2000 superpixels. The classes are arranged in descending order of the number of superpixels within the class. The number of superpixels in the k-th class of the i-th frame is recorded as H ik ;

[0014] S7. Calculate the average optical flow of superpixels contained in each class, and record the average optical flow of the k-th class in the i-th frame as HV ik ;

[0015] S8. Take the category k=0 as the background, and then test the category k=1. If it satisfies the following formula:

[0016]

[0017] Then the category k=1 is also regarded as the background, and the category k=2 is tested. In the formula, |||| represents the length of the optical flow vector, and · represents the inner product of the two vectors. The above process is repeated until the above formula is not satisfied between the kth category and the k+1th category. Then the test is stopped, and all superpixels in the background category are set as background in the object mask of the corresponding frame to obtain new object masks M'0, M'1, M'2, ..., M' N-1 ;

[0018] S9, mark the moving object that needs to be accurately tracked in the frame image I0 of the 0th frame, use the mask of the object in M'0 as the preliminary object mask, and calculate the average optical flow OV of all pixels in the mask 00 ;

[0019] S10. Check all other object masks that are adjacent to or overlapped with the preliminary object mask, calculate their average optical flow, and record the average optical flow of the lth adjacent or overlapping object as OV 0l , if:

[0020]

[0021] It is considered that the lth object and the selected object can be merged, and finally all masks that can be used as selected objects in the 0th frame are obtained, which is recorded as M b 0, where b∈O, O represents the set of all selected objects;

[0022] S11. Calculate the mask of the object to be tracked after processing in the 0th frame, denoted as P0, where P0(x, y) represents the probability that the pixel at the coordinate (x, y) belongs to the object to be tracked, and:

[0023]

[0024] S12. After obtaining the mask of the object to be tracked in the i-1th frame, calculate the optical flow from the i-1th frame to the i-th frame, and map the mask of the object to be tracked in the i-1th frame to the i-th frame to obtain the mapped tracking mask PP. i , and there is:

[0025]

[0026] where dx x，y Represents the horizontal component of the optical flow at the coordinate (x, y), dy x，y Represents the vertical component of the optical flow at the coordinate (x, y), Indicates rounding down;

[0027] S13, M' i Zhongyu PP iThe object with the largest overlapping area is taken as the object to be tracked in the i-th frame, and then the mask P of the object to be tracked after the i-th frame is processed is calculated according to steps S9 to S11. i , in which when checking other object masks adjacent to or overlapping with the preliminary object mask, it is necessary to further check the i is greater than 0.5, but in M' i The middle is the background area, and after processing each frame, the object mask P0, P1, P2, ..., P N-1 ;

[0028] S14, the mask P of the object to be tracked after processing the N-1 frame N-1 As the input object mask P to be tracked * N-1 ;

[0029] S15. After obtaining the mask of the object to be tracked in the i-th frame, calculate the optical flow from the i-th frame to the i-1-th frame, map the mask of the object to be tracked in the i-th frame to the i-1-th frame according to step S12, and calculate the mask P of the object to be tracked in the i-1-th frame according to steps S9 to S11. * i-1 Finally, the mask P of the object to be tracked in each frame is obtained * 0.P * 1. P * 2. ..., P * N-1 ;

[0030] S16, frame images I0, I1, I2, ..., I N-1 Perform color space transformation to obtain the corresponding HSV and CIELAB color spaces, and splice them with the original RGB color space to obtain a 9-channel feature map, where the value of each channel is normalized to the interval [0, 1] by dividing by the maximum value of the channel;

[0031] S17, for the i-th frame image, the 9-channel color space feature map, the horizontal component optical flow map X i , vertical component optical flow map Y i , and the mask P of the object to be tracked * i , forming a 12-channel input feature map, inputting a shallow fully convolutional network, and finally outputting an accurate tracking result R i , where R i The size and I i The number of channels is 2, which respectively represent the probability of each pixel belonging to the object to be tracked and the background, thereby achieving pixel-level accurate tracking of specific moving objects in the video.

[0032] As a preferred technical solution, in step S2, a trained Mask R-CNN network is used to detect each frame image.

[0033] As a preferred technical solution, in step S3, a dense optical flow map is obtained using a trained Flownet 2.0 network.

[0034] As a preferred technical solution, in the S17 step, the shallow fully convolutional network structure used is: the input is a feature map of size 12×H×W, where H and W are the height and width of the frame image, respectively, followed by a 3×3 convolutional layer with a 128-channel output with a stride of 1, a BN layer, a 3×3 convolutional layer with a 256-channel output with a stride of 1, a BN layer, a 3×3 convolutional layer with a stride of 128-channel output with a stride of 1, a BN layer, a 1×1 convolutional layer with a 2-channel output with a stride of 1, and a Softmax activation function, and finally outputs a 2-classification result of whether each pixel belongs to a foreground object or background.

[0035] As a preferred technical solution, in the step S17, the shallow fully convolutional network used in the training uses uniform distribution to initialize the network parameters, uses the Adam method to optimize the parameters, sets the learning rate to 0.001, the first-order smoothing parameter β1 = 0.9, the second-order smoothing parameter β2 = 0.999, and avoids the small constant ε = 10 divided by 0. -8 , the loss function L is:

[0036]

[0037] where z x，y is the label of the object mask at the coordinate (x, y) in the training sample, 1 represents the foreground object to be tracked, 0 represents the background, z′ x，y To predict the probability that the coordinate (x, y) is the foreground object to be tracked, H and W are the height and width of the frame image respectively.

[0038] The present invention has the following advantages and effects compared to the prior art:

[0039] 1. The present invention provides a method for accurately tracking moving objects in videos based on composite color space features and optical flow information. It comprehensively utilizes multiple color space features in the spatial domain of video frame images and optical flow information reflecting the time domain of the video, thereby improving the accuracy of accurate tracking of specific moving objects in the video.

[0040] 2. The present invention uses optical flow information to forward and backward propagate and correct the object mask in the video frame image, obtaining a more accurate object mask as the input of the neural network, further improving the accuracy of precise tracking of specific moving objects in the video.

[0041] 3. The present invention uses a lightweight shallow fully convolutional network to output the final pixel-level tracking results, which reduces the computational complexity while ensuring the accuracy of the results, thereby effectively improving the application efficiency of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flowchart of the overall process of the method for accurately tracking moving objects in a video disclosed by the present invention;

[0043] Figure 2 is a structural diagram of a shallow fully convolutional network model in an embodiment of the present invention;

[0044] Figure 3 This is an example of an object segmentation mask in an embodiment of the present invention. DETAILED DESCRIPTION

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0046] This embodiment discloses a method for accurately tracking moving objects in video. The following describes the implementation of the invention in detail, using the DAVIS video database as a training and test sample. The shallow fully convolutional network described in this embodiment was trained using 30 videos from the DAVIS database training set and tested using 20 videos from the DAVIS database test set. This embodiment was implemented on a Linux system, primarily based on the deep learning framework Pytorch 1.6.1, using an NVIDIA GTX1080Ti graphics card, CUDA version 10.1.105, and cudnn version 7.6.4.

[0047] The overall implementation process is as follows Figure 1 The specific implementation steps are as follows:

[0048] The video to be detected is divided into frames to obtain the frame images of each frame, which are recorded as I0, I1, I2, ..., I N-1 , where N is the total number of video frames.

[0049] Using the trained Mask R-CNN network, each frame image is detected to obtain the object mask of each frame, which is recorded as M0, M1, M2, ..., M N-1 .

[0050] For the i-th frame image Ii , select the adjacent I before and after it i-3 , I i-2 , I i-1 , I i+1 , I i+2 and I i+3 The entities in the frame image are respectively i Combined, the corresponding dense optical flow map is obtained using the trained Flownet 2.0 network, and the average optical flow of each pixel is further calculated to obtain the average optical flow map corresponding to the i-th frame image, which is recorded as the horizontal component optical flow map X i And the vertical component optical flow map Y i ,i∈{0,1,2,…,N-1}.

[0051] Each frame image is divided into 2000 superpixels using the SLIC superpixel segmentation algorithm, and the jth superpixel of the i-th frame image is recorded as S ij , where j∈{0, 1, 2, ..., 1999}, calculate the average optical flow of all pixels in each superpixel, denoted as V ij .

[0052] For the i-th frame image, the average optical flow V of each superpixel is ij The DBSCAN clustering method is used to automatically cluster 2000 superpixels. The classes are arranged in descending order of the number of superpixels within the class. The number of superpixels in the k-th class of the i-th frame is recorded as H ik .

[0053] Calculate the average optical flow of superpixels contained in each class, and record the average optical flow of the k-th class in the i-th frame as HV ik The category with k=0 is used as the background, and then the category with k=1 is tested if it satisfies the following formula:

[0054]

[0055] Then the category k=1 is also regarded as the background, and the category k=2 is tested. In the formula, |||| represents the length of the optical flow vector, and · represents the inner product of the two vectors. The above process is repeated until the above formula is not satisfied between the kth category and the k+1th category. Then the test is stopped, and all superpixels in the background category are set as background in the object mask of the corresponding frame to obtain new object masks M'0, M'1, M'2, ..., M' N-1 ;

[0056] Mark the moving object that needs to be accurately tracked in the frame image I0 of the 0th frame, use the mask of the object in M'0 as the preliminary object mask, and calculate the average optical flow OV of all pixels in the mask 00 .

[0057] Check all other object masks that are adjacent to or overlapped with the preliminary object mask, calculate their average optical flow, and record the average optical flow of the lth adjacent or overlapping object as OV 0l , if:

[0058]

[0059] It is considered that the lth object and the selected object can be merged, and finally all the masks that can be used as selected objects in the Oth frame are obtained, which is recorded as M b 0, where b∈O, O represents the set of all selected objects.

[0060] Calculate the mask of the object to be tracked after processing in the 0th frame, denoted as P0, where P0(x, y) represents the probability that the pixel at the coordinate (x, y) belongs to the object to be tracked, and:

[0061]

[0062] After obtaining the mask of the object to be tracked in the i-1th frame, calculate the optical flow from the i-1th frame to the i-th frame, and map the mask of the object to be tracked in the i-1th frame to the i-th frame to obtain the mapped object mask PP i ,have:

[0063]

[0064] where dx x，y Represents the horizontal component of the optical flow at the coordinate (x, y), dy x，y Represents the vertical component of the optical flow at the coordinate (x, y), Indicates rounding down.

[0065] M' i Zhongyu PP i The object with the largest overlapping area is taken as the object to be tracked in the i-th frame, and then the mask P of the object to be tracked after the i-th frame is processed is calculated according to steps S9 to S11. i , in which when checking other object masks adjacent to or overlapping with the preliminary object mask, it is necessary to further check the i is greater than 0.5, but in M' i The middle is the background area, and after processing each frame, the object mask P0, P1, P2, ..., P N-1 .

[0066] The object mask P to be tracked after processing the N-1th frame N-1 As the input object mask P to be tracked * N-1After obtaining the mask of the object to be tracked in the i-th frame, calculate the optical flow from the i-th frame to the i-1-th frame, map the mask of the object to be tracked in the i-th frame to the i-1-th frame according to step S12, and calculate the mask P of the object to be tracked in the i-1-th frame according to steps S9 to S11. * i-1 Finally, the mask P of the object to be tracked in each frame is obtained * 0.P * 1. P * 2. ..., P * N-1 .

[0067] For frame images I0, I1, I2, ..., I N-1 Perform color space transformation to obtain the corresponding HSV and CIELAB color spaces, which are then concatenated with the original RGB color space to obtain a 9-channel feature map, where the value of each channel is normalized to the interval [0, 1] according to the maximum value of the channel.

[0068] For the i-th frame image, the 9-channel color space feature map, the horizontal component optical flow map X i , vertical component optical flow map Y i , and the mask P of the object to be tracked * i , forming a 12-channel input feature map, inputting a shallow fully convolutional network, and finally outputting an accurate tracking result R i , where R i The size and I i The same, the number of channels is 2, representing the probability of each pixel belonging to the object to be tracked and the background respectively.

[0069] The shallow fully convolutional network structure used above is as follows Figure 2 As shown in the figure, specifically: the input is a feature map of size 12×H×W, where H and W are the height and width of the frame image respectively, followed by a 3×3 convolutional layer with a 128-channel output with a stride of 1, a BN layer, a 3×3 convolutional layer with a 256-channel output with a stride of 1, a BN layer, a 3×3 convolutional layer with a stride of 128-channel output, a BN layer, a 1×1 convolutional layer with a 2-channel output with a stride of 1, and a Softmax activation function, and finally outputs a binary classification result of whether each pixel belongs to a foreground object or background.

[0070] During training, the shallow fully convolutional network uses uniform distribution to initialize the network parameters and uses the Adam method to optimize the parameters. The learning rate is set to 0.001, the first-order smoothing parameter β1 = 0.9, the second-order smoothing parameter β2 = 0.999, and the small constant ε = 10 is used to avoid division by 0. -8 , the loss function L is:

[0071]

[0072] where z x，y is the label of the object mask at the coordinate (x, y) in the training sample, 1 represents the foreground object to be tracked, 0 represents the background, z′ x，y To predict the probability that the coordinate (x, y) is the foreground object to be tracked, H and W are the height and width of the frame image respectively.

[0073] This example accurately tracks moving objects in 20 videos from the DAVIS database test set. The results are shown in the following table. The values ​​in the table are IoU (Intersection Over Union) coefficients, which are the quotient of the intersection of the detection result and the actual result divided by the union of the two. The value ranges from 0 to 1. The maximum value of 1 is achieved when the detection result and the actual result are completely consistent. Figure 3 shown.

[0074] Table 1. Object segmentation results of the method in this embodiment

[0075]

[0076] From the above experimental results, it can be seen that the method of the present invention has a high accuracy rate for precise tracking of moving objects in videos, which proves the effectiveness of the method of the present invention.

[0077] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for accurately tracking moving objects in a video, characterized in that: The following steps are involved: S1, the video to be detected is divided into frames to obtain the frame images of each frame, which are recorded as I0, I1, I2, ..., I N-1 , where N is the total number of video frames; S2. Use pixel-level object segmentation method to detect each frame image and obtain the object mask of each frame, which is recorded as M0, M1, M2, ..., M N-1 ; S3, for the i-th frame image I i , select the adjacent I before and after it i-3 , I i-2 , I i-1 , I i+1 , I i+2 and I i+3 The entities in the frame image are respectively i The corresponding dense optical flow map is obtained by combining the calculations, and the average optical flow of each pixel is further calculated to obtain the average optical flow map corresponding to the i-th frame image, which is recorded as the horizontal component optical flow map X i And the vertical component optical flow map Y i , i∈{0,1,2,…,N-1}; S4. Use the SLIC superpixel segmentation algorithm to segment each frame image into 2000 superpixels, and record the jth superpixel of the i-th frame image as S ij , where j∈{0,1,2,…,1999}; S5. Calculate the average optical flow of all pixels in each superpixel, denoted as V ij ; S6. For the i-th frame image, the average optical flow V of each superpixel ij The DBSCAN clustering method is used to automatically cluster 2000 superpixels. The classes are arranged in descending order of the number of superpixels within the class. The number of superpixels in the k-th class of the i-th frame is recorded as H ik ; S7. Calculate the average optical flow of superpixels contained in each class, and record the average optical flow of the k-th class in the i-th frame as HV ik ; S8. Take the category k=0 as the background, and then test the category k=1. If it satisfies the following formula: Then the category k=1 is also regarded as the background, and the category k=2 is tested. In the formula, || || represents the length of the optical flow vector, and · represents the inner product of the two vectors. The above process is repeated until the above formula is not satisfied between the kth category and the k+1th category. Then the test is stopped, and all superpixels in the background category are set as background in the object mask of the corresponding frame to obtain new object masks M'0, M'1, M'2, ..., M' N-1 ; S9, mark the moving object that needs to be accurately tracked in the frame image I0 of the 0th frame, use the mask of the object in M'0 as the preliminary object mask, and calculate the average optical flow OV of all pixels in the mask 00 ; S10. Check all other object masks that are adjacent to or overlapped with the preliminary object mask, calculate their average optical flow, and record the average optical flow of the lth adjacent or overlapping object as OV 0l , if: It is considered that the lth object and the selected object can be merged, and finally all masks that can be used as selected objects in the 0th frame are obtained, which is recorded as M b 0, where b∈O, O represents the set of all selected objects; S11. Calculate the mask of the object to be tracked after processing in the 0th frame, denoted as P0, where P0(x, y) represents the probability that the pixel at the coordinate (x, y) belongs to the object to be tracked, and: S12. After obtaining the mask of the object to be tracked in the i-1th frame, calculate the optical flow from the i-1th frame to the i-th frame, and map the mask of the object to be tracked in the i-1th frame to the i-th frame to obtain the mapped tracking mask PP. i , and there is: where dx x,y Represents the horizontal component of the optical flow at the coordinate (x, y), dy x,y Represents the vertical component of the optical flow at the coordinate (x, y), Indicates rounding down; S13, M' i Zhongyu PP i The object with the largest overlapping area is taken as the object to be tracked in the i-th frame, and then the mask P of the object to be tracked after the i-th frame is processed is calculated according to steps S9 to S11. i , in which when checking other object masks adjacent to or overlapping with the preliminary object mask, it is necessary to further check the i is greater than 0.5, but in M' i The middle is the background area, and after processing each frame, the object mask P0, P1, P2, ..., P N-1 ; S14, the mask P of the object to be tracked after processing the N-1 frame N-1 As the input object mask P to be tracked * N-1 ; S15. After obtaining the mask of the object to be tracked in the i-th frame, calculate the optical flow from the i-th frame to the i-1-th frame, map the mask of the object to be tracked in the i-th frame to the i-1-th frame according to step S12, and calculate the mask P of the object to be tracked in the i-1-th frame according to steps S9 to S11. * i-1 Finally, the mask P of the object to be tracked in each frame is obtained * 0.P * 1. P * 2. ..., P * N-1 ; S16, frame images I0, I1, I2, ..., I N-1 Perform color space transformation to obtain the corresponding HSV and CIELAB color spaces, and splice them with the original RGB color space to obtain a 9-channel feature map, where the value of each channel is normalized to the interval [0,1] by dividing by the maximum value of the channel; S17, for the i-th frame image, the 9-channel color space feature map, the horizontal component optical flow map X i , vertical component optical flow map Y i , and the mask P of the object to be tracked * i , forming a 12-channel input feature map, inputting a shallow fully convolutional network, and finally outputting an accurate tracking result R i , where R i The size and I i The number of channels is 2, which respectively represent the probability of each pixel belonging to the object to be tracked and the background, thereby achieving pixel-level accurate tracking of specific moving objects in the video.

2. The method for accurately tracking a moving object in a video according to claim 1, wherein: In step S2, each frame image is detected using the trained Mask R-CNN network.

3. The method for accurately tracking a moving object in a video according to claim 1, wherein: In the step S3, a dense optical flow map is obtained using the trained Flownet 2.0 network.

4. The method for accurately tracking a moving object in a video according to claim 1, wherein: In the S17 step, the shallow fully convolutional network structure used is: the input is a feature map of size 12×H×W, where H and W are the height and width of the frame image, respectively, followed by a 3×3 convolutional layer with a 128-channel output with a stride of 1, a BN layer, a 3×3 convolutional layer with a 256-channel output with a stride of 1, a BN layer, a 3×3 convolutional layer with a stride of 128-channel output with a stride of 1, a BN layer, a 1×1 convolutional layer with a 2-channel output with a stride of 1, and a Softmax activation function, and finally outputs a 2-classification result of whether each pixel belongs to a foreground object or background.

5. The method for accurately tracking a moving object in a video according to claim 1, wherein: In the step S17, the shallow fully convolutional network used in the training was initialized with uniform distribution for network parameters, and the parameters were optimized using the Adam method, with a learning rate of 0.001, a first-order smoothing parameter β1 = 0.9, a second-order smoothing parameter β2 = 0.999, and a small constant ε = 10 to avoid division by 0. -8 , the loss function L is: where z x,y is the label of the object mask at the coordinate (x, y) in the training sample, 1 represents the foreground object to be tracked, 0 represents the background, z′ x,y To predict the probability that the coordinate (x, y) is the foreground object to be tracked, H and W are the height and width of the frame image respectively.

Images