Multi-target tracking method and device

Abstract

The invention discloses a multi-target tracking method and device, and the method comprises the steps: carrying out the target detection of a video frame of an obtained target video, and obtaining a to-be-tracked target object; dividing the video frame into a plurality of sub-regions, and counting the number of to-be-tracked target objects in each sub-region to obtain a target density distribution diagram; performing secondary division on the sub-regions according to the target density distribution diagram, and performing feature extraction on the to-be-tracked target object according to region information subjected to secondary division to obtain category features and position features of the to-be-tracked target object; predicting a predicted position feature of the to-be-tracked target object at the next moment based on the position feature; and acquiring a real-time position feature and a real-time category feature of the target object to be tracked at the next moment, and calculating a category feature distance and a position feature distance of the target object to be tracked to determine a matching relationship. According to the method provided by the invention, a multi-target tracking task can be more efficiently and accurately realized in a complex and changeable actual scene.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition and target tracking, and in particular to a multi-target tracking method and device. Background Art

[0002] Multi-object tracking (MOT) is a core task in the field of computer vision, which aims to simultaneously detect and track multiple objects from video or sensor data.

[0003] Existing multi-target tracking methods usually rely only on geometric distance or appearance similarity for matching in the matching stage, which leads to mismatching in complex scenes, thereby increasing computational overhead and decision time.

[0004] It can be seen that how to optimize the multi-target tracking method has become a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0005] The present invention provides a multi-target tracking method and device to solve the technical problem of optimizing the multi-target tracking method.

[0006] In order to solve the above technical problems, an embodiment of the present invention provides a multi-target tracking method, including:

[0007] Perform target detection on the acquired video frames of the target video based on the target detection network to obtain the target object to be tracked;

[0008] Dividing the video frame into a plurality of sub-regions, and counting the number of the target objects to be tracked in each sub-region to obtain a target density distribution map of the target objects to be tracked;

[0009] The sub-region is divided twice according to the target density distribution map, and features of the target object to be tracked are extracted according to the regional information of the secondary division to obtain the category features and position features of the target object to be tracked; wherein the secondary division is designed to further divide the target object to be tracked according to the density of the sub-region;

[0010] Predicting the predicted position feature of the target object to be tracked at the next moment based on the Kalman filter algorithm and the position feature;

[0011] The real-time position feature and real-time category feature of the target object to be tracked at the next moment are obtained, and the category feature distance and position feature distance of the target object to be tracked are calculated based on the Hungarian algorithm to determine the matching relationship, so as to complete the tracking of the target object to be tracked.

[0012] As one preferred solution, dividing the video frame into a plurality of sub-regions, and counting the number of the target objects to be tracked in each sub-region to obtain a target density distribution map of the target objects to be tracked includes:

[0013] Dividing the video frame into a plurality of sub-regions using a fixed grid partitioning strategy;

[0014] Traversing the target detection network to output bounding box information of all the target objects to be tracked in each sub-region, and counting the number of target objects to be tracked in each sub-region by determining the positional relationship between the bounding box information and the sub-region;

[0015] The quantity information is visualized to obtain the target density distribution map.

[0016] As one of the preferred solutions, the secondary division of the sub-area according to the target density distribution map includes:

[0017] When the target density distribution map shows that the density of the target object to be tracked in the first sub-region is higher than a preset high-density threshold, further dividing the first sub-region;

[0018] And when the target density distribution map shows that the density of the target object to be tracked in the second sub-region is lower than a preset low-density threshold, merging adjacent sub-regions of the second sub-region.

[0019] As one of the preferred solutions, the predicted position feature of the target object to be tracked at the next moment is predicted based on the Kalman filter algorithm and the position feature, which is expressed as:

[0020]

[0021] Where F is the state transfer matrix, Q is the process noise, and x t is the position feature of the target at time t, The predicted value of the target at time t+1 based on the position feature of the state at time t.

[0022] As one of the preferred solutions, the method of calculating the category feature distance and the position feature distance of the target object to be tracked based on the Hungarian algorithm to determine the matching relationship includes:

[0023] Constructing a bipartite graph according to the category feature, location feature, real-time location feature and real-time category feature of the target object to be tracked;

[0024] Calculating a category feature distance according to the category feature and the real-time category feature, and calculating a location feature distance according to the location feature and the real-time location feature;

[0025] A matching scheme with the minimum weighted sum of the category feature distance and the position feature distance in the bipartite graph is calculated based on the Hungarian algorithm, and a matching relationship between the current moment and the next moment of the target object to be tracked is determined according to the matching scheme.

[0026] Another embodiment of the present invention provides a multi-target tracking device, comprising:

[0027] A detection module is used to perform target detection on the acquired video frames of the target video based on a target detection network to obtain a target object to be tracked;

[0028] A first division module is used to divide the video frame into a plurality of sub-areas, and count the number of the target objects to be tracked in each sub-area to obtain a target density distribution map of the target objects to be tracked;

[0029] A second division module is used to perform secondary division on the sub-area according to the target density distribution map, and extract features of the target object to be tracked according to the regional information of the secondary division to obtain the category features and position features of the target object to be tracked; wherein the secondary division is designed to further divide the target object to be tracked according to the density of the sub-area;

[0030] A prediction module, used for predicting the predicted position feature of the target object to be tracked at the next moment based on the Kalman filter algorithm and the position feature;

[0031] The matching module is used to obtain the real-time position feature and real-time category feature of the target object to be tracked at the next moment, calculate the category feature distance and position feature distance of the target object to be tracked based on the Hungarian algorithm to determine the matching relationship, and complete the tracking of the target object to be tracked.

[0032] As one preferred solution, the first division module is specifically used to:

[0033] Dividing the video frame into a plurality of sub-regions using a fixed grid partitioning strategy;

[0034] Traversing the target detection network to output bounding box information of all the target objects to be tracked in each sub-region, and counting the number of target objects to be tracked in each sub-region by determining the positional relationship between the bounding box information and the sub-region;

[0035] The quantity information is visualized to obtain the target density distribution map.

[0036] As one preferred solution, the second division module is specifically used to:

[0037] When the target density distribution map shows that the density of the target object to be tracked in the first sub-region is higher than a preset high-density threshold, further dividing the first sub-region;

[0038] And when the target density distribution map shows that the density of the target object to be tracked in the second sub-region is lower than a preset low-density threshold, merging adjacent sub-regions of the second sub-region.

[0039] As one of the preferred solutions, the predicted position feature of the target object to be tracked at the next moment is predicted based on the Kalman filter algorithm and the position feature, which is expressed as:

[0040]

[0041] Where F is the state transfer matrix, Q is the process noise, and x t is the position feature of the target at time t, The predicted value of the target at time t+1 based on the position feature of the state at time t.

[0042] As one preferred solution, the matching module is specifically used for:

[0043] Constructing a bipartite graph according to the category feature, location feature, real-time location feature and real-time category feature of the target object to be tracked;

[0044] Calculating a category feature distance according to the category feature and the real-time category feature, and calculating a location feature distance according to the location feature and the real-time location feature;

[0045] A matching scheme with the minimum weighted sum of the category feature distance and the position feature distance in the bipartite graph is calculated based on the Hungarian algorithm, and a matching relationship between the current moment and the next moment of the target object to be tracked is determined according to the matching scheme.

[0046] Compared with the prior art, the beneficial effects of the present invention are at least one of the following:

[0047] 1) In the target detection and region division stage, the present invention processes the target video frames through an advanced target detection network, which can accurately identify various target objects to be tracked. Compared with the traditional method relying on simple threshold segmentation or template matching, it greatly improves the accuracy and robustness of target recognition.

[0048] 2) The present invention performs secondary division of sub-areas based on the target density distribution map, showing strong adaptive capabilities. For target-dense areas, such as crowded squares and traffic-congested intersections, further subdividing sub-areas can effectively reduce the complexity of targets in each sub-area, avoid confusion when associating data, and greatly improve processing efficiency.

[0049] 3) In the feature extraction link, the present invention adopts different strategies for different density areas according to the regional information after secondary division, and uses advanced methods such as multi-scale feature fusion in dense areas to ensure that even if the targets occlude each other, effective category features and position features can still be extracted; a lightweight feature extraction network is used in sparse areas to speed up the processing while ensuring accuracy, thereby achieving reasonable allocation and efficient utilization of resources.

[0050] 4) The present invention uses position features combined with the Kalman filter algorithm to predict the next moment position features of the target object to be tracked, and fully considers the motion continuity of the target. Moreover, after obtaining the real-time position features and real-time category features of the target at the next moment, the matching relationship is determined by calculating the category feature distance and the position feature distance. This dynamic matching mechanism based on feature distance can not only utilize the discriminability of the target appearance features, but also take into account the continuity of the motion trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of a multi-target tracking method in one embodiment of the present invention;

[0052] Figure 2 It is a structural schematic diagram of a multi-target tracking device in one embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] In the description of this application, the terms "first", "second", "third", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", "third", etc. may explicitly or implicitly include one or more of the feature. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0055] In the description of the present application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. The terms "vertical", "horizontal", "left", "right", "upper", "lower" and similar expressions used herein are only for illustrative purposes, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0056] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those commonly understood by those skilled in the art. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood by specific circumstances.

[0057] An embodiment of the present invention provides a multi-target tracking method. For details, see Figure 1 , Figure 1 The flowchart of the multi-target tracking method in one embodiment of the present invention is shown, which includes steps S1-S5:

[0058] S1: Perform target detection on the acquired video frame of the target video based on the target detection network to obtain the target object to be tracked;

[0059] Among them, the target videos of this application are widely derived from actual scenarios where different industries need to track and monitor specific targets, which can specifically include surveillance videos of city streets, videos shot by surveillance cameras at traffic intersections, recorded videos of various sports competitions, and surveillance videos in automated assembly line workshops.

[0060] Among them, the target detection network of this application adopts Faster R-CNN, and its core includes two parts: the region generation network (RPN) and Fast R-CNN. RPN is responsible for quickly generating a series of candidate regions that may contain targets. It slides the window on the feature map and uses the anchor box mechanism to preset multiple anchor boxes for different scales and aspect ratios to preliminarily frame the potential target position, greatly reducing the search space for subsequent detection. Subsequently, Fast R-CNN performs fine classification and position regression on these candidate regions, and uses the features extracted by the deep convolutional neural network to determine the target category of each candidate region through the fully connected layer, such as pedestrians, vehicles, animals and other common target types, and accurately adjusts the bounding box coordinates to make the target positioning more accurate.

[0061] In this step, when the target detection network is applied to the video frames of the target video, for each input frame, the network will first preprocess the image, usually including image normalization operations to map pixel values ​​to a uniform range to improve the stability and generalization ability of the model. Then, through multi-layer convolution, pooling and other operations, the deep features of the image are gradually extracted. These feature maps contain rich semantic information and can capture the key visual features of the target, such as shape, texture, color, etc. Based on these features, RPN generates candidate regions, and then Fast R-CNN completes the final classification and regression, outputting the information of the target object to be tracked that is precisely defined in the form of bounding boxes. These bounding boxes carry the location coordinates of the target and the corresponding category labels, providing accurate and reliable basic data for subsequent multi-target tracking processes, such as region division, feature extraction, location prediction, etc., ensuring that the entire tracking system can operate efficiently based on accurately identified targets.

[0062] S2: Divide the video frame into multiple sub-areas, and count the number of target objects to be tracked in each sub-area to obtain a target density distribution map of the target objects to be tracked;

[0063] Preferably, in one embodiment of the present invention, the video frame is divided into a plurality of sub-regions, and the number of target objects to be tracked in each sub-region is counted to obtain a target density distribution map of the target objects to be tracked, including:

[0064] A fixed grid partitioning strategy is used to divide the video frame into multiple sub-regions;

[0065] Traversing the target detection network, outputting the bounding box information of all target objects to be tracked in each sub-region, and counting the number of target objects to be tracked in each sub-region by determining the positional relationship between the bounding box information and the sub-region;

[0066] Visualize the quantity information and obtain the target density distribution map.

[0067] Specifically, in actual operation, it is necessary to reasonably determine the grid parameters based on the size of the video frame and the approximate scale range of the target. In one embodiment of the present invention, for common high-definition video frames (resolution of 1920×1080), if the targets are mostly small objects or pedestrians, the screen can be evenly divided into 30×20 grid units; if the targets are mainly large vehicles, it may be more appropriate to divide them into 15×10 grids to ensure that each sub-area can effectively cover the target without causing a sharp increase in the amount of calculation due to over-fine division. This fixed grid division method is like building a regular "coordinate grid" for complex video images, providing a clear framework for subsequent data processing.

[0068] The target detection network is traversed to output the bounding box information of all target objects to be tracked in each sub-region. This step closely follows the target detection step. After the target detection network has accurately identified each target to be tracked and output its bounding box (usually represented by the upper left corner coordinates (x1, y1) and the lower right corner coordinates (x2, y2)), each sub-region is carefully checked. The quantity information is counted by judging the positional relationship between the bounding box information and the sub-region. The specific judgment logic is as follows: for a sub-region, its upper left corner coordinates are set to (i×w, j×h), and the lower right corner coordinates are ((i+1)×w, (j+1)×h), where i and j represent the horizontal and vertical indexes of the sub-region, respectively, and w and h are the width and height of the sub-region. If (i×w≤x1<(i+1)×w) and (j×h≤y1<(j+1)×h), or (i×w≤x2<(i+1)×w) and (j×h≤y2<(j+1)×h), the detection box falls within this sub-region, and the corresponding sub-region target number counter is incremented by 1. After traversing all sub-regions and bounding boxes, the number of targets in each sub-region can be accurately obtained.

[0069] Visualizing the quantity information and obtaining the target density distribution map aims to transform the abstract digital information into an intuitive image display. Various visualization tools can be used, such as the Matplotlib library in Python. Taking the form of a heat map as an example, the number of targets in the sub-region is mapped to different color intensities. The area with a large number of targets is displayed as a darker color (such as red), and the area with a small number is displayed as a lighter color (such as blue), thereby generating a colorful target density distribution map. This picture is like a "battlefield situation map", which clearly presents the density of targets in different areas of the video frame, allowing researchers or systems to grasp the target distribution dynamics at a glance, and then perform targeted secondary division, feature extraction and other operations on subsequent sub-regions based on this map, thereby improving the efficiency and accuracy of multi-target tracking.

[0070] S3: performing secondary division of the sub-region according to the target density distribution map, extracting features of the target object to be tracked according to the regional information of the secondary division, and obtaining the category features and position features of the target object to be tracked; wherein the secondary division is designed to further divide the sub-region according to the density of the target object to be tracked in the sub-region;

[0071] Preferably, in one embodiment of the present invention, the sub-areas are divided twice according to the target density distribution map, including:

[0072] When the target density distribution map shows that the density of the target object to be tracked in the first sub-region is higher than a preset high-density threshold, the first sub-region is further divided;

[0073] And when the target density distribution map shows that the density of the target object to be tracked in the second sub-region is lower than a preset low-density threshold, the adjacent sub-regions of the second sub-region are merged.

[0074] Specifically, in one embodiment of the present invention, taking urban traffic monitoring as an example, if the focus is on vehicle tracking, considering the driving characteristics of vehicles and the spatial distribution of roads, for high-definition video images, the high-density threshold can be set to an average of more than 5 vehicles per sub-area, and the low-density threshold can be set to an average of less than 1 vehicle per sub-area. When the target density distribution map shows that the density of the target object to be tracked in the first sub-area is higher than the preset high-density threshold, it means that the target in the area is too dense, which is likely to cause problems such as data association confusion and inaccurate feature extraction. At this time, it becomes necessary to further divide the first sub-area.

[0075] Among them, there are many methods for further division, the most common one is quadtree division. Divide the original first sub-area into four smaller sub-areas, then recount the number of targets in these new sub-areas and evaluate their density again. If there are still sub-areas with too high density, you can continue to recursively use quadtree division until the target density in the sub-area is within a reasonable range. This subdivision method is like fine-tuning congested traffic intersections, breaking down complex target clusters into smaller ones, reducing mutual interference between targets in each sub-area, and creating favorable conditions for subsequent accurate feature extraction.

[0076] In contrast, when the target density distribution map shows that the density of the target object to be tracked in the second sub-region is lower than the preset low-density threshold, merging the adjacent sub-regions of the second sub-region is an optimization strategy. The definition of adjacent sub-regions is usually based on spatial position relationships, such as adjacent sub-regions in the horizontal or vertical direction. The merging operation not only reduces unnecessary waste of computing resources, but also improves the representativeness and stability of features based on a larger range of regional information during subsequent feature extraction. For example, in sparse grassland monitoring scenarios, when tracking scattered wild animals, merging low-density areas can enable the feature extraction network to better capture the activity patterns of animals in a larger spatial range and avoid feature fragmentation due to too small factor areas.

[0077] After completing the secondary division, feature extraction is performed on the target object to be tracked based on the regional information of the secondary division to obtain category features and position features. For category features, if a deep learning model is used, the target image block is usually processed based on a convolutional neural network. In dense sub-regions, due to the high risk of occlusion, the network structure will focus on multi-scale feature fusion, using techniques such as dilated convolution to expand the receptive field, ensuring that even if part of the target appearance is occluded, discriminative category feature vectors can still be extracted. These vectors contain rich target appearance category information, such as vehicle type, animal species, etc. In sparse sub-regions, a relatively lightweight network structure is used to focus on key feature extraction, quickly and accurately determine the target category, and improve processing efficiency.

[0078] In terms of position feature extraction, in dense sub-areas, the bounding box coordinates of the target and the next moment position information predicted by the Kalman filter algorithm are comprehensively determined. Based on the historical motion trajectory of the target, the Kalman filter can still accurately estimate the short-term position direction of the target in scenes with dense targets and complex movements, provide forward-looking information for position features, and assist in subsequent tracking and matching. In sparse sub-areas, it mainly relies on the current bounding box coordinates of the target to simply and directly obtain position features, while reducing unnecessary computing overhead to achieve reasonable allocation and efficient use of resources, and comprehensively guarantee the accurate execution of multi-target tracking tasks.

[0079] S4: predicting the predicted position features of the target object to be tracked at the next moment based on the Kalman filter algorithm and the position features;

[0080] Preferably, in one embodiment of the present invention, the predicted position feature of the target object to be tracked at the next moment is predicted based on the Kalman filter algorithm and the position feature, which is expressed as:

[0081]

[0082] Where F is the state transfer matrix, Q is the process noise, and x tis the position feature of the target at time t, The predicted value of the target at time t+1 based on the position feature of the state at time t.

[0083] Among them, the Kalman filter algorithm is based on the linear system state space model. It cleverly integrates the historical position information of the target and the estimation of the dynamic characteristics of the system to achieve accurate calculation of the target's position at the next moment.

[0084] In the step, the Kalman filter must first be initialized for each target object to be tracked. The initialization process involves setting the initial state vector of the filter, which usually contains the current position of the target (expressed in the horizontal and vertical coordinates of the image coordinate system) and the speed information. The speed information can be estimated based on the position change of the target in the previous few frames through a simple differential operation, providing basic motion parameters for subsequent dynamic prediction.

[0085] However, in actual scenarios, the target motion is inevitably disturbed by various external factors, such as the impact of wind on flying objects and the interference of road bumps on vehicle driving. These interference factors are uniformly quantified by process noise. The covariance matrix of process noise needs to be carefully adjusted according to the specific application scenario and target characteristics. It reflects the system's estimation of the uncertainty of target motion. In complex dynamic scenarios, such as frequent starts, stops and turns of vehicles at traffic intersections, the value of process noise should be appropriately increased to adapt to the large movement changes of the target; in relatively stable scenarios, such as indoor object monitoring, the process noise can be set to a smaller value. By incorporating process noise into the prediction model, Kalman filtering can offset the impact of external interference on prediction accuracy to a certain extent, making the predicted position features more reliable.

[0086] S5: Obtain the real-time position feature and real-time category feature of the target object to be tracked at the next moment, calculate the category feature distance and position feature distance of the target object to be tracked based on the Hungarian algorithm to determine the matching relationship, and complete the tracking of the target object to be tracked.

[0087] Preferably, in one embodiment of the present invention, the matching relationship is determined by calculating the category feature distance and the position feature distance of the target object to be tracked based on the Hungarian algorithm, including:

[0088] Construct a bipartite graph according to the category feature, location feature, real-time location feature and real-time category feature of the target object to be tracked;

[0089] Calculating a category feature distance based on the category feature and the real-time category feature, and calculating a location feature distance based on the location feature and the real-time location feature;

[0090] The matching scheme with the minimum weighted sum of the category feature distance and the position feature distance in the bipartite graph is calculated based on the Hungarian algorithm, and the matching relationship between the current moment and the next moment of the target object to be tracked is determined according to the matching scheme.

[0091] Specifically, for example, in an intelligent traffic monitoring scenario, a new frame of video image is transmitted, and the deep learning-based target detection network again identifies vehicles, pedestrians and other targets in the picture, and outputs the updated bounding box coordinates of each target as the real-time position feature, as well as the vehicle type (sedan, SUV, truck, etc.) or pedestrian posture features obtained through model discrimination as the real-time category feature. These real-time information reflects the status of the target at the current latest moment.

[0092] Next, a bipartite graph is constructed based on the category features, position features, real-time position features, and real-time category features of the target object to be tracked. The construction of the bipartite graph is the basic architecture for the application of the Hungarian algorithm. The set of target objects to be tracked at the current moment is taken as the vertex set (V_1) on one side of the bipartite graph. Each vertex represents a target to be matched, which carries the category features extracted in the previous step (such as in the form of high-dimensional vectors, containing target appearance category discrimination information, learned by deep convolutional neural networks) and position features (including the next moment position predicted based on Kalman filtering and the current actual position information); while the vertex set (V_2) on the other side is composed of the real-time target objects obtained at the next moment, and its corresponding real-time position features and real-time category features have just been obtained through the latest round of target detection. The construction of the bipartite graph in this way builds a bridge for the subsequent precise matching of targets at different moments. Its purpose is to find the most reasonable correspondence between the two sets of vertices to ensure the consistency of the target tracking trajectory.

[0093] The category feature distance is calculated based on the category feature and the real-time category feature, and the location feature distance is calculated based on the location feature and the real-time location feature. For the category feature distance calculation, since category features are mostly presented in high-dimensional vectors, a variant method of cosine similarity is often used. In this way, the cosine value of the angle between two vectors in the feature space is converted into a distance metric. The distance value ranges from (0) to (1). The closer to (0), the more similar the two targets are in category features, that is, the more likely it is that the same target is presented at different times.

[0094] In terms of location feature distance calculation, it is mainly measured based on the target's coordinate information. Given the predicted location coordinates of the target (A) and the real-time location coordinates at the next moment, the Euclidean distance is used as the metric. This distance intuitively reflects the degree of deviation between the actual location of the target and the predicted location. The smaller the deviation, the more consistent the target's motion trajectory is with expectations, and the higher the possibility of matching.

[0095] Finally, the matching scheme with the minimum weighted sum of the category feature distance and the position feature distance in the bipartite graph is calculated based on the Hungarian algorithm, and the matching relationship between the current moment and the next moment of the target object to be tracked is determined according to the matching scheme.

[0096] An embodiment of the present invention provides a multi-target tracking device. For details, see Figure 2 , Figure 2 The structure diagram of a multi-target tracking device in one embodiment of the present invention is shown, which includes:

[0097] A detection module 11 is used to perform target detection on the acquired video frames of the target video based on a target detection network to obtain a target object to be tracked;

[0098] A first division module 12 is used to divide the video frame into a plurality of sub-regions, and count the number of target objects to be tracked in each sub-region to obtain a target density distribution map of the target objects to be tracked;

[0099] The second division module 13 is used to perform secondary division on the sub-region according to the target density distribution map, and extract features of the target object to be tracked according to the regional information of the secondary division to obtain the category features and position features of the target object to be tracked; wherein the secondary division is designed to further divide the sub-region according to the density of the target object to be tracked in the sub-region;

[0100] A prediction module 14, used for predicting the predicted position feature of the target object to be tracked at the next moment based on the Kalman filter algorithm and the position feature;

[0101] The matching module 15 is used to obtain the real-time position feature and real-time category feature of the target object to be tracked at the next moment, calculate the category feature distance and position feature distance of the target object to be tracked based on the Hungarian algorithm to determine the matching relationship, and complete the tracking of the target object to be tracked.

[0102] Preferably, in one embodiment of the present invention, the first partitioning module is specifically used to:

[0103] A fixed grid partitioning strategy is used to divide the video frame into multiple sub-regions;

[0104] Traversing the target detection network, outputting the bounding box information of all target objects to be tracked in each sub-region, and counting the number of target objects to be tracked in each sub-region by determining the positional relationship between the bounding box information and the sub-region;

[0105] Visualize the quantity information and obtain the target density distribution map.

[0106] Preferably, in one embodiment of the present invention, the second partitioning module is specifically used for:

[0107] When the target density distribution map shows that the density of the target object to be tracked in the first sub-region is higher than a preset high-density threshold, the first sub-region is further divided;

[0108] And when the target density distribution map shows that the density of the target object to be tracked in the second sub-region is lower than a preset low-density threshold, the adjacent sub-regions of the second sub-region are merged.

[0109] Preferably, in one embodiment of the present invention, the predicted position feature of the target object to be tracked at the next moment is predicted based on the Kalman filter algorithm and the position feature, which is expressed as:

[0110]

[0111] Where F is the state transfer matrix, Q is the process noise, and x t is the position feature of the target at time t, The predicted value of the target at time t+1 based on the position feature of the state at time t.

[0112] Preferably, in one embodiment of the present invention, the matching module is specifically used for:

[0113] Construct a bipartite graph according to the category feature, location feature, real-time location feature and real-time category feature of the target object to be tracked;

[0114] Calculating a category feature distance based on the category feature and the real-time category feature, and calculating a location feature distance based on the location feature and the real-time location feature;

[0115] The matching scheme with the minimum weighted sum of the category feature distance and the position feature distance in the bipartite graph is calculated based on the Hungarian algorithm, and the matching relationship between the current moment and the next moment of the target object to be tracked is determined according to the matching scheme.

[0116] Compared with the prior art, the beneficial effects of the present invention are at least one of the following:

[0117] 1) In the target detection and region division stage, the present invention processes the target video frames through an advanced target detection network, which can accurately identify various target objects to be tracked. Compared with the traditional method relying on simple threshold segmentation or template matching, it greatly improves the accuracy and robustness of target recognition.

[0118] 2) The present invention performs secondary division of sub-areas based on the target density distribution map, showing strong adaptive capabilities. For target-dense areas, such as crowded squares and traffic-congested intersections, further subdividing sub-areas can effectively reduce the complexity of targets in each sub-area, avoid confusion when associating data, and greatly improve processing efficiency.

[0119] 3) In the feature extraction link, the present invention adopts different strategies for different density areas according to the regional information after secondary division, and uses advanced methods such as multi-scale feature fusion in dense areas to ensure that even if the targets occlude each other, effective category features and position features can still be extracted; a lightweight feature extraction network is used in sparse areas to speed up the processing while ensuring accuracy, thereby achieving reasonable allocation and efficient utilization of resources.

[0120] 4) The present invention uses position features combined with the Kalman filter algorithm to predict the next moment position features of the target object to be tracked, and fully considers the motion continuity of the target. Moreover, after obtaining the real-time position features and real-time category features of the target at the next moment, the matching relationship is determined by calculating the category feature distance and the position feature distance. This dynamic matching mechanism based on feature distance can not only utilize the discriminability of the target appearance features, but also take into account the continuity of the motion trajectory.

[0121] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A multi-target tracking method, characterized in that: include: Perform target detection on the acquired video frames of the target video based on the target detection network to obtain the target object to be tracked; Dividing the video frame into a plurality of sub-regions, and counting the number of the target objects to be tracked in each sub-region to obtain a target density distribution map of the target objects to be tracked; Secondarily dividing the sub-region according to the target density distribution map, extracting features of the target object to be tracked according to the secondary divided region information, and obtaining category features and position features of the target object to be tracked; The secondary division is designed to further divide the target object to be tracked in the sub-area according to the density of the target object to be tracked; Predicting the predicted position feature of the target object to be tracked at the next moment based on the Kalman filter algorithm and the position feature; The real-time position feature and real-time category feature of the target object to be tracked at the next moment are obtained, and the category feature distance and position feature distance of the target object to be tracked are calculated based on the Hungarian algorithm to determine the matching relationship, so as to complete the tracking of the target object to be tracked.

2. The multi-target tracking method according to claim 1, characterized in that: The step of dividing the video frame into a plurality of sub-regions and counting the number of the target objects to be tracked in each sub-region to obtain a target density distribution map of the target objects to be tracked includes: Dividing the video frame into a plurality of sub-regions using a fixed grid partitioning strategy; Traversing the target detection network to output bounding box information of all the target objects to be tracked in each sub-region, and counting the number of target objects to be tracked in each sub-region by determining the positional relationship between the bounding box information and the sub-region; The quantity information is visualized to obtain the target density distribution map.

3. The multi-target tracking method according to claim 1, characterized in that: The secondary division of the sub-area according to the target density distribution map comprises: When the target density distribution map shows that the density of the target object to be tracked in the first sub-region is higher than a preset high-density threshold, further dividing the first sub-region; And when the target density distribution map shows that the density of the target object to be tracked in the second sub-region is lower than a preset low-density threshold, merging adjacent sub-regions of the second sub-region.

4. The multi-target tracking method according to claim 1, characterized in that: The predicted position feature of the target object to be tracked at the next moment is predicted based on the Kalman filter algorithm and the position feature, and is expressed as: Where F is the state transfer matrix, Q is the process noise, and x t is the position feature of the target at time t, The predicted value of the target at time t+1 based on the position feature of the state at time t.

5. The multi-target tracking method according to claim 1, characterized in that: The calculating the category feature distance and the position feature distance of the target object to be tracked based on the Hungarian algorithm to determine the matching relationship includes: Constructing a bipartite graph according to the category feature, location feature, real-time location feature and real-time category feature of the target object to be tracked; Calculating a category feature distance according to the category feature and the real-time category feature, and calculating a location feature distance according to the location feature and the real-time location feature; A matching scheme with the minimum weighted sum of the category feature distance and the position feature distance in the bipartite graph is calculated based on the Hungarian algorithm, and a matching relationship between the current moment and the next moment of the target object to be tracked is determined according to the matching scheme.

6. A multi-target tracking device, characterized in that: include: A detection module is used to perform target detection on the acquired video frames of the target video based on a target detection network to obtain a target object to be tracked; A first division module is used to divide the video frame into a plurality of sub-areas, and count the number of the target objects to be tracked in each sub-area to obtain a target density distribution map of the target objects to be tracked; A second division module is used to perform secondary division on the sub-region according to the target density distribution map, and extract features of the target object to be tracked according to the secondary divided region information to obtain category features and position features of the target object to be tracked; The secondary division is designed to further divide the target object to be tracked in the sub-area according to the density of the target object to be tracked; A prediction module, used for predicting the predicted position feature of the target object to be tracked at the next moment based on the Kalman filter algorithm and the position feature; The matching module is used to obtain the real-time position feature and real-time category feature of the target object to be tracked at the next moment, calculate the category feature distance and position feature distance of the target object to be tracked based on the Hungarian algorithm to determine the matching relationship, and complete the tracking of the target object to be tracked.

7. The multi-target tracking device according to claim 6, characterized in that: The first division module is specifically used for: Dividing the video frame into a plurality of sub-regions using a fixed grid partitioning strategy; Traversing the target detection network to output bounding box information of all the target objects to be tracked in each sub-region, and counting the number of target objects to be tracked in each sub-region by determining the positional relationship between the bounding box information and the sub-region; The quantity information is visualized to obtain the target density distribution map.

8. The multi-target tracking device according to claim 6, characterized in that: The second division module is specifically used for: When the target density distribution map shows that the density of the target object to be tracked in the first sub-region is higher than a preset high-density threshold, further dividing the first sub-region; And when the target density distribution map shows that the density of the target object to be tracked in the second sub-region is lower than a preset low-density threshold, merging adjacent sub-regions of the second sub-region.

9. The multi-target tracking device according to claim 6, characterized in that: The predicted position feature of the target object to be tracked at the next moment is predicted based on the Kalman filter algorithm and the position feature, and is expressed as: Where F is the state transfer matrix, Q is the process noise, and x t is the position feature of the target at time t, The predicted value of the target at time t+1 based on the position feature of the state at time t.

10. The multi-target tracking device according to claim 6, characterized in that: The matching module is specifically used for: Constructing a bipartite graph according to the category feature, location feature, real-time location feature and real-time category feature of the target object to be tracked; Calculating a category feature distance according to the category feature and the real-time category feature, and calculating a location feature distance according to the location feature and the real-time location feature; A matching scheme with the minimum weighted sum of the category feature distance and the position feature distance in the bipartite graph is calculated based on the Hungarian algorithm, and a matching relationship between the current moment and the next moment of the target object to be tracked is determined according to the matching scheme.

Images