Multi-target tracking methods, systems, devices, and media based on video hybrid domain

Abstract

This invention provides a method, system, device, and medium for multi-target tracking based on video hybrid domain. The method comprises: MV data acquisition and processing, extracting motion vectors from the video stream and preprocessing the motion vectors, where the motion vectors are denoted as MV; pixel-domain target detection, inputting the video to be detected into a pre-trained target recognition network model for prediction to obtain target prediction boxes; joint tracking decision-making, adjusting the x and y coordinates of each target tracking box in the (n-1)th frame based on the set of target tracking boxes and the MV of the target prediction boxes, and determining whether the corresponding target prediction boxes and target tracking boxes in the set of prediction boxes and the corrected set of target tracking boxes in the nth frame are the same target; obtaining the tracking results, recording and plotting list A in each frame. n The tracking result is obtained by determining the position and number of the target tracking box.

Description

Technical Field

[0001] This invention relates to the field of video target tracking technology, and more specifically, to a multi-target tracking method, system, device, and medium based on video hybrid domain. Background Technology

[0002] Target tracking involves detecting, extracting, recognizing, and tracking moving targets in a time series of consecutive frames of images. This process yields the target's position and trajectory, enabling the understanding of the target's behavior and facilitating more advanced detection tasks. Based on the number of targets being tracked, tracking algorithms can be categorized into single-target tracking and multi-target tracking. Compared to single-target tracking, multi-target tracking is more complex and challenging. The main task of multi-target tracking is to simultaneously locate multiple targets of interest in a given video, maintain the ID of each target, and record their trajectories.

[0003] The data domain of existing tracking methods can be divided into pixel domain, compression domain, and hybrid domain;

[0004] The advantages of pixel domain trackers include their high accuracy and the fact that they do not require consideration of video codecs. However, they tend to be resource-intensive because all pixel values ​​need to be reconstructed, stored in memory, and processed.

[0005] The advantages of compressed domain trackers include efficiency and speed, as they avoid most of the video decoding, pixel value storage and processing, and typically use less input data. The disadvantages of compressed domain trackers are their reliance on the video coding method used for compressed video, and potentially lower accuracy due to the low resolution limitations of the motion sampling grid.

[0006] Hybrid domain trackers combine the advantages of pixel domain and compressed domain well, but they still suffer from tracking failures due to interference such as target occlusion, lighting changes, and motion blur. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a multi-target tracking method, system, device, and medium based on a video hybrid domain. By processing pixel-domain information, the spatial information of the video is utilized; while the motion vector (MV) in the compressed domain information represents the temporal information in the video. By combining spatiotemporal information for real-time multi-target tracking in video, the problem of tracking failure easily occurring under interference such as target occlusion, lighting changes, and motion blur is solved.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] The first aspect of this invention provides a multi-target tracking method based on video hybrid domain, the method being:

[0010] MV Data Acquisition and Processing

[0011] After extracting motion vectors from the video stream, the motion vectors are preprocessed, where the motion vectors are denoted as MV;

[0012] Pixel domain object detection

[0013] The video to be detected is input into a pre-trained target recognition network model for prediction, and the target prediction box is obtained.

[0014] Tracking joint decision-making

[0015] (1) Calculate the MV of each target prediction box based on the obtained target prediction box position and the preprocessed MV, that is, calculate the average value of the MV of all pixels in each target prediction box.

[0016] The set of MVs of the m predicted object bounding boxes in the nth frame is denoted as M. n ;

[0017] (2) For the nth frame of the video to be detected, we have A n =(a1,a2,a3,…,a m ), B n =(b1,b2,b3,…,b m ), a={x,y,w,h,c,n,t}, b={x,y,w,h,c};

[0018] Among them, A n It is the set of target tracking boxes for that frame; B n This is the set of target prediction boxes obtained from target detection in this frame; x, y, w, and h are the position information of the target prediction boxes, namely the horizontal and vertical coordinates of the upper left corner of the target prediction box and the width and height of the target prediction box; c is the prediction category of the target prediction box; n is the tracking target number of the target prediction box; t is the trajectory accuracy of the tracking target, which is set to 1 in the first frame.

[0019] For the first frame of the video to be detected, the set of target prediction boxes B1 is the set of target tracking boxes A1;

[0020] (3) Perform target tracking on the nth frame of the video to be detected, where n>1:

[0021] According to A in frame n-1 n-1 With M n-1 The x and y coordinates of each target tracking box in frame n-1 are adjusted for correction. Specifically, the horizontal and vertical components of the MV of that frame are added to the horizontal and vertical directions of each target tracking box position in frame n-1 to obtain the corrected set A of target tracking boxes in frame n-1. n-1 ';

[0022] Get the set B of prediction boxes in frame n. n =(b1,b2,b3,…,b m );

[0023] Judge B n With A n-1 'Whether the target prediction box and the target tracking box in the two sets are the same target;'

[0024] Get tracking results

[0025] Record and draw list A in each frame. n The tracking result is obtained by determining the position and number of the target tracking box.

[0026] A second aspect of the present invention provides a multi-target tracking system based on video hybrid domain, comprising:

[0027] The input module is used to input the video stream;

[0028] The MV data acquisition and processing module is used to acquire motion vectors;

[0029] The pixel-domain object detection module is used to obtain the predicted bounding box of the target.

[0030] The tracking joint decision-making module is used to obtain tracking results;

[0031] The output module is used to output the tracking results;

[0032] The input module is connected to the input of the pixel domain target detection module and the input of the pixel domain target detection module, respectively. The output of the pixel domain target detection module and the output of the pixel domain target detection module are respectively connected to the input of the tracking joint decision module. The output of the tracking joint decision module is connected to the output module, so as to complete the multi-target tracking method based on video hybrid domain.

[0033] A third aspect of the present invention provides a multi-target tracking device based on video hybrid domain, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0034] Memory, used to store computer programs;

[0035] The processor, when executing a program stored in memory, implements the multi-target tracking method based on the video hybrid domain as described above.

[0036] A fourth aspect of the present invention provides a computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the multi-target tracking method based on video hybrid domain as described above.

[0037] This invention has outstanding substantive features and significant progress compared to the prior art, specifically:

[0038] 1. In the method of the present invention, pixel domain target detection only requires obtaining the target prediction box. Therefore, the required target recognition network model is not limited to a certain recognition algorithm and can be applied to various new recognition algorithms.

[0039] 2. The method of this invention only uses motion vector (MV) information in the compressed domain, therefore, it can be widely applied to different video coding standards;

[0040] 3. The method of the present invention analyzes the target's motion trajectory during target tracking, which greatly improves the tracking robustness under target occlusion conditions;

[0041] 4. When performing target tracking, the method of the present invention introduces motion vector (MV) information, which does not depend on the appearance information of the target, and the tracking effect is less affected by target occlusion, illumination changes and motion blur.

[0042] 5. The method of the present invention utilizes both pixel domain and compressed domain data. By using MV to optimize the pixel domain detection results, the tracking performance is improved with lower resource consumption.

[0043] 6. The tracking joint decision-making algorithm proposed in this invention improves the tracking success rate by setting a dynamic threshold to compare the intersection and union ratio of the predicted box and the tracking box after MV correction as the decision basis. Attached Figure Description

[0044] Figure 1 A flowchart of the method in Embodiment 1 is shown.

[0045] Figure 2 An example of MV selection in the method of Embodiment 1 is shown.

[0046] Figure 3 An example diagram of the tracking effect in Example 2 is shown.

[0047] Figure 4 A framework diagram of the system in Embodiment 3 is shown. Detailed Implementation

[0048] The technical solution of the present invention will be further described in detail below through specific embodiments.

[0049] Example 1

[0050] like Figure 1 As shown, this embodiment provides a multi-target tracking method based on video hybrid domain, the method being:

[0051] MV Data Acquisition and Processing

[0052] After extracting motion vectors from the video stream, the motion vectors are preprocessed, where the motion vectors are denoted as MV.

[0053] Specifically, the HEVC encoder is used to preprocess the MV:

[0054] The HEVC encoder divides an image into multiple coding tree units (CTUs). Each CTU is recursively divided into multiple coding units (CUs) in a quadtree manner. The size of a CU can be 64x64, 32x32, 16x16, or 8x8, and can only be less than or equal to the size of the CTU. Each CU is further divided into one or more prediction units (PUs), and each PU independently holds a prediction mode.

[0055] For PUs holding SKIP or Intra prediction modes that do not have MV data, the following processing is performed:

[0056] SKIP prediction model PU processing:

[0057] If the SKIP prediction mode indicates that the corresponding PU is almost identical to the corresponding colocalization region in the previous frame, then 0MV is assigned to each PU holding the SKIP prediction mode.

[0058] Intra-predictive model PU processing:

[0059] Sort the i MVs from adjacent PUs according to the angle v between them and the horizontal axis to obtain a list V = (v1, v2, v3, ..., v i Each adjacent PU is processed according to a 4×4 size. For example, an 8×8 PU will be added to list V as two 4×4 units. That is, an 8×8 PU can be divided into four 4×4 PUs. The MV of each PU is the same as the MV value of the 8×8 PU. Then, the two 4×4 PUs adjacent to the PU in question are selected and added to the list.

[0060] If adjacent PUs also have the Intra prediction mode, then they are not added to list V;

[0061] like Figure 2 In the list V, we get V = (v1,v2,v3,v3,v4,v4,v5,v5,v6,v6,v6,v6);

[0062] Then select half of the continuous vectors in list V as a sublist, where the selected continuous vectors are the continuous vectors with the smallest sum of angle differences;

[0063] The average of the direction and magnitude of the continuous vectors in the sublist is taken as the direction and magnitude of the MV after processing by the Intra prediction mode PU.

[0064] Pixel domain object detection

[0065] The video to be detected is input into a pre-trained target recognition network model for prediction, and the target prediction box is obtained.

[0066] Specifically, the pre-trained target recognition network model is a pre-trained YOLOv7 recognition network model.

[0067] Building and training a YOLOv7 recognition network model:

[0068] (1) Setting up a YOLOv7 network:

[0069] The YOLOv7 object detection prediction network consists of three parts: Backbone, FPN, and YOLO Head. The Backbone is the core feature extraction network of YOLOv7. The input image is first processed for feature extraction within the backbone network; the extracted features are called feature layers, which are the feature sets of the input image. In the backbone network, three feature layers are obtained for further network construction; these three feature layers are called effective feature layers. FPN is the enhanced feature extraction network of YOLOv7. The three effective feature layers obtained in the backbone network are fused here, combining feature information from different scales. In the FPN, the already obtained effective feature layers are used for further feature extraction. YOLO Head is the classifier and regressor of YOLOv7. Through the Backbone and FPN, three enhanced effective feature layers are obtained. Each feature layer has width, height, and number of channels. The feature map can then be viewed as a set of feature points, each with three prior boxes, and each prior box has a number of channels and features. The YOLO Head essentially performs the following task: judging whether there is an object corresponding to the prior bounding box at a feature point. Therefore, the entire YOLOv7 network performs the following functions: feature extraction, feature enhancement, and prediction of the object corresponding to the prior bounding box.

[0070] (2) Creating the dataset:

[0071] Training the YOLOv7 network requires a dataset in VOC or YOLO format. Videos are captured using a camera, and each frame of the video is manually labeled with the location and category of the target being tracked (such as person, car, bicycle, etc.). 20% of the labeled images are randomly selected as the test set, and the remaining 80% are used as the test set.

[0072] (3) YOLOv7 network training:

[0073] Load the training set, test set, and YOLOv7 official pre-trained weight files, and train until the network converges.

[0074] Tracking joint decision-making

[0075] (1) Calculate the MV of each target prediction box based on the obtained target prediction box position and the preprocessed MV, that is, calculate the average value of the MV of all pixels in each target prediction box.

[0076] The set of MVs of the m predicted object bounding boxes in the nth frame is denoted as M. n .

[0077] (2) For the nth frame of the video to be detected, we have A n =(a1,a2,a3,…,a m ), B n =(b1,b2,b3,…,b m ), a={x,y,w,h,c,n,t}, b={x,y,w,h,c};

[0078] Among them, A n It is the set of target tracking boxes for that frame; B n This is the set of target prediction boxes obtained from target detection in this frame; x, y, w, and h are the position information of the target prediction boxes, namely the horizontal and vertical coordinates of the upper left corner of the target prediction box and the width and height of the target prediction box; c is the prediction category of the target prediction box; n is the tracking target number of the target prediction box; t is the trajectory accuracy of the tracking target, which is set to 1 in the first frame.

[0079] For the first frame of the video to be detected, the set of predicted bounding boxes B1 for object detection is the same as the set of tracking bounding boxes A1.

[0080] (3) Perform target tracking on the nth frame of the video to be detected, where n>1:

[0081] According to A in frame n-1 n-1 With M n-1 The x and y coordinates of each target tracking box in frame n-1 are adjusted for correction. Specifically, the horizontal and vertical components of the MV of that frame are added to the horizontal and vertical directions of each target tracking box position in frame n-1 to obtain the corrected set A of target tracking boxes in frame n-1. n-1 ';

[0082] Get the set B of prediction boxes in frame n. n =(b1,b2,b3,…,b m );

[0083] Judge B n With A n-1 'Whether the target prediction box and the target tracking box in the two sets are the same target.'

[0084] Specifically, the B is determined by comparing whether the Intersection over Union (IoU) of the two bounding boxes is greater than a preset threshold. n With A n-1 'Whether the target prediction box and the target tracking box in the two sets are the same target;'

[0085] The intersection-union ratio (IoU) is obtained from formula (1).

[0086]

[0087] Where A∩B is the area of ​​the overlapping portion of the two boxes, and A∪B is the sum of the areas of the two boxes minus the area of ​​the overlapping portion; the preset threshold for each target tracking box is 0.5 + 0.2t, and t is updated with each target tracking box in each frame, calculated as follows:

[0088]

[0089] Among them, M x M y Δx and Δy are the horizontal and vertical components of the target tracking box's MV, respectively, and Δx and Δy are the horizontal and vertical displacements of the target in adjacent frames.

[0090] Judge B n With A n-1 When considering whether the predicted bounding box and the tracking bounding box in the two sets represent the same target, if B... n There is a failure to connect with A n-1 If the predicted bounding box matches the target tracking bounding box in ', then treat it as a new tracking target, add a tracking target number n and an initial trajectory accuracy t = 1 to it, and add it to list A. n middle.

[0091] Get tracking results

[0092] Record and draw list A in each frame. n The tracking result is obtained by determining the position and number of the target tracking box.

[0093] Example 2

[0094] The difference between this embodiment and Embodiment 1 is that it provides a specific tracking joint decision-making algorithm.

[0095]

[0096]

[0097] Record and draw list A in each frame. n By tracking the position and number of the bounding box, the tracking result is obtained, and the tracking effect is as follows. Figure 3 As shown.

[0098] Example 3

[0099] This embodiment provides a multi-target tracking system based on video hybrid domain, including:

[0100] The input module is used to input the video stream;

[0101] The MV data acquisition and processing module is used to acquire motion vectors;

[0102] The pixel-domain object detection module is used to obtain the predicted bounding box of the target.

[0103] The tracking joint decision-making module is used to obtain tracking results;

[0104] The output module is used to output the tracking results;

[0105] The input module is connected to the input of the pixel domain target detection module and the input of the pixel domain target detection module, respectively. The output of the pixel domain target detection module and the output of the pixel domain target detection module are connected to the input of the tracking joint decision module, respectively. The output of the tracking joint decision module is connected to the output module, so as to complete the multi-target tracking method based on video hybrid domain described in Embodiment 1.

[0106] Example 4

[0107] This embodiment provides a multi-target tracking device based on video hybrid domain, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0108] Memory, used to store computer programs;

[0109] When the processor executes the program stored in the memory, it implements the multi-target tracking method based on the video hybrid domain as described in Example 1.

[0110] Example 5

[0111] This embodiment provides a computer-readable storage medium storing instructions that, when executed by one or more processors, cause the processors to perform the multi-target tracking method based on video hybrid domain as described in Embodiment 1.

[0112] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-non-transitory readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer program code.

[0113] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0114] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of the present invention or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in the present invention.

Claims

1. A multi-target tracking method based on video hybrid domain, characterized in that, The method is as follows: MV Data Acquisition and Processing After extracting motion vectors from the video stream, the motion vectors are preprocessed, where the motion vectors are denoted as MV; The MV is preprocessed using an HEVC encoder: The HEVC encoder divides an image into multiple coding tree units (CTUs), and each CTU is recursively divided into multiple coding units (CUs) in a quadtree manner. Each CU is further divided into one or more prediction units (PUs), and each PU independently holds a prediction mode. For PUs holding SKIP or Intra prediction modes that do not have MV data, the following processing is performed: SKIP prediction model PU processing: If the SKIP prediction mode indicates that the corresponding PU is almost identical to the corresponding colocalization region in the previous frame, then 0MV is assigned to each PU holding the SKIP prediction mode. Intra-predictive model PU processing: Sort the i MVs from adjacent PUs according to the angle v between them and the horizontal axis to obtain a list V = (v1, v2, v3, ..., v i Each adjacent PU is processed in a 4×4 size; If adjacent PUs also have the Intra prediction mode, then they are not added to list V; Then select half of the continuous vectors in list V as a sublist, where the selected continuous vectors are the continuous vectors with the smallest sum of angle differences; The average of the direction and magnitude of the continuous vectors in the sublist is taken as the direction and magnitude of the MV after processing by the Intra prediction mode PU; Pixel domain object detection The video to be detected is input into a pre-trained target recognition network model for prediction, and the target prediction box is obtained. Tracking joint decision-making (1) Calculate the MV of each target prediction box based on the obtained target prediction box position and the preprocessed MV, that is, calculate the average value of the MV of all pixels in each target prediction box. The set of MVs of the m predicted object bounding boxes in the nth frame is denoted as M. n ; (2) For the nth frame of the video to be detected, we have A n =(a1,a2,a3,…,a m ), B n =(b1,b2,b3,…,b m ), a={x,y,w,h,c,n,t}, b={x,y,w,h,c}; Among them, A n It is the set of target tracking boxes for that frame; B n This is the set of target prediction boxes obtained from target detection in this frame; x, y, w, and h are the position information of the target prediction boxes, namely the horizontal and vertical coordinates of the upper left corner of the target prediction box and the width and height of the target prediction box; c is the prediction category of the target prediction box; n is the tracking target number of the target prediction box; t is the trajectory accuracy of the tracking target, which is set to 1 in the first frame. For the first frame of the video to be detected, the set of target prediction boxes B1 is the set of target tracking boxes A1; (3) Perform target tracking on the nth frame of the video to be detected, where n>1: According to A in frame n-1 n-1 With M n-1 The x and y coordinates of each target tracking box in frame n-1 are adjusted for correction. Specifically, the horizontal and vertical components of the MV of that frame are added to the horizontal and vertical directions of each target tracking box position in frame n-1 to obtain the corrected set A of target tracking boxes in frame n-1. n-1 '; Get the set B of prediction boxes in frame n. n =(b1,b2,b3,…,b m ); Judge B n With A n-1 'Whether the target prediction box and the target tracking box in the two sets are the same target;' Get tracking results Record and draw list A in each frame. n The tracking result is obtained by determining the position and number of the target tracking box.

2. The multi-target tracking method based on video hybrid domain according to claim 1, characterized in that: The B is determined by comparing whether the Intersection over Union (IoU) of the two bounding boxes is greater than a preset threshold. n With A n-1 'Whether the target prediction box and the target tracking box in the two sets are the same target;' The intersection-union ratio (IoU) is obtained from formula (1). Where A∩B is the area of ​​the overlapping portion of the two boxes, and A∪B is the sum of the areas of the two boxes minus the area of ​​the overlapping portion; the preset threshold for each target tracking box is 0.5 + 0.2t, and t is updated with each target tracking box in each frame, calculated as follows: in, M x , M y These are the horizontal and vertical components of the target tracking bounding box's MV, △ x , △ y It is the horizontal and vertical displacement of the target in adjacent frames.

3. The multi-target tracking method based on video hybrid domain according to any one of claims 1-2, characterized in that: If B n There is a failure to connect with A n-1 If the predicted bounding box matches the target tracking bounding box in ', then treat it as a new tracking target, add a tracking target number n and an initial trajectory accuracy t=1 to it, and add it to list A. n middle.

4. The multi-target tracking method based on video hybrid domain according to claim 1, characterized in that, The pre-trained target recognition network model is a pre-trained YOLOv7 recognition network model.

5. A multi-target tracking system based on video hybrid domain, characterized in that, include: The input module is used to input the video stream; The MV data acquisition and processing module is used to acquire motion vectors; The pixel-domain object detection module is used to obtain the predicted bounding box of the target. The tracking joint decision-making module is used to obtain tracking results; The output module is used to output the tracking results; The input module is connected to the input of the pixel domain target detection module and the input of the pixel domain target detection module, respectively. The output of the pixel domain target detection module and the output of the pixel domain target detection module are respectively connected to the input of the tracking joint decision module. The output of the tracking joint decision module is connected to the output module, so as to complete the multi-target tracking method based on video hybrid domain as described in any one of claims 1-4.

6. A multi-target tracking device based on video hybrid domain, comprising a processor, a communication interface, a memory, and a communication bus, wherein, The processor, communication interface, and memory communicate with each other via a communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the multi-target tracking method based on video hybrid domain as described in any one of claims 1-4.

7. A computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the video hybrid domain-based multi-target tracking method as described in any one of claims 1-4.

Images