A multi-target tracking method and device

Abstract

The application discloses a multi-target tracking method and device, wherein the method comprises the following steps: performing target detection on the video frames of the obtained target video to obtain a target object to be tracked; dividing the video frames into a plurality of sub-regions, and counting the number of the target object to be tracked in each sub-region to obtain a target density distribution map; performing secondary division on the sub-regions according to the target density distribution map, performing feature extraction on the target object to be tracked according to the region information of the secondary division to obtain the category feature and the position feature of the target object to be tracked; predicting the predicted position feature of the target object to be tracked at the next moment based on the position feature; obtaining the real-time position feature and the real-time category feature of the target object to be tracked at the next moment, and calculating the category feature distance and the position feature distance of the target object to be tracked to determine the matching relationship. The method provided by the application can more efficiently and accurately realize the multi-target tracking task in a complex and changeable actual scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition and target tracking, and particularly relates to a multi-target tracking method and device. BACKGROUND

[0002] Multiple Object Tracking (MOT) is a core task in the field of computer vision, aiming to simultaneously detect and track multiple targets from video or sensor data.

[0003] Existing multi-target tracking methods usually only rely on geometric distance or appearance similarity for matching in the matching stage, resulting in mis-matching phenomena in complex scenes, thereby increasing the computational overhead and decision-making time.

[0004] Therefore, how to optimize the multi-target tracking method has become a technical problem to be solved by those skilled in the art. SUMMARY

[0005] The present application provides a multi-target tracking method and device to solve the technical problem of optimizing the multi-target tracking method.

[0006] In order to solve the above technical problems, the present application provides a multi-target tracking method, comprising:

[0007] detecting a target video frame obtained by a target detection network to obtain a target object to be tracked;

[0008] dividing the video frame into a plurality of sub-regions, and counting the number of the target object to be tracked in each sub-region to obtain a target density distribution map of the target object to be tracked;

[0009] According to the target density distribution map, the sub-regions are further divided, and the target object to be tracked is extracted according to the region information of the further division to obtain the class feature and the position feature of the target object to be tracked; wherein the further division is designed according to the density of the target object to be tracked in the sub-region;

[0010] predicting the predicted position feature of the target object to be tracked at the next time based on the Kalman filtering algorithm and the position feature;

[0011] obtaining the real-time position feature and the real-time class feature of the target object to be tracked at the next time, calculating the class feature distance and the position feature distance of the target object to be tracked based on the Hungarian algorithm to determine the matching relationship, and completing the tracking of the target object to be tracked.

[0012] As one of the preferred solutions, the dividing the video frame into a plurality of sub-regions, and counting the number of the target objects to be tracked in each sub-region to obtain the target density distribution map, comprises:

[0013] The video frame is divided into a plurality of sub-regions by using a fixed grid division strategy;

[0014] The bounding box information of all the target objects to be tracked in each sub-region output by the target detection network is traversed, and the number information of the target objects to be tracked in each sub-region is counted by judging the positional relationship between the bounding box information and the sub-region;

[0015] The number information is visualized to obtain the target density distribution map.

[0016] As one of the preferred solutions, the sub-regions are further divided according to the target density distribution map, comprising:

[0017] When the target density distribution map shows that the density of the target objects to be tracked in a first sub-region is higher than a preset high-density threshold, the first sub-region is further divided;

[0018] And when the target density distribution map shows that the density of the target objects to be tracked in a second sub-region is lower than a preset low-density threshold, the adjacent sub-regions of the second sub-region are merged.

[0019] As one of the preferred solutions, the predicted position feature of the target object to be tracked at the next time is predicted based on the Kalman filtering algorithm and the position feature, and is represented as:

[0020]

[0021] Wherein, F is a state transition matrix, Q is a process noise, x t is the position feature of the target at time t, is the position feature prediction value of the target at time t+1 according to the state at time t.

[0022] As one of the preferred solutions, the class feature distance and the position feature distance of the target object to be tracked are calculated based on the Hungarian algorithm to determine the matching relationship, comprising:

[0023] A bipartite graph is constructed according to the class feature, the position feature, the real-time position feature and the real-time class feature of the target object to be tracked;

[0024] The class feature distance is calculated according to the class feature and the real-time class feature, and the position feature distance is calculated according to the position feature and the real-time position feature;

[0025] determine a matching relationship of the target object to be tracked according to the matching scheme.

[0026] Another embodiment of the present application provides a multi-target tracking device, comprising:

[0027] a detection module configured to perform target detection on video frames of a target video obtained by a target detection network to obtain a target object to be tracked;

[0028] a first division module configured to divide the video frames into a plurality of sub-regions and count a number of the target object to be tracked in each sub-region to obtain a target density distribution map of the target object to be tracked;

[0029] a second division module configured to perform secondary division on the sub-regions according to the target density distribution map and perform feature extraction on the target object to be tracked according to region information of the secondary division to obtain category features and position features of the target object to be tracked; wherein the secondary division is designed to further divide according to a density of the target object to be tracked in the sub-regions;

[0030] a prediction module configured to predict a predicted position feature of the target object to be tracked at a next time based on a Kalman filtering algorithm and the position features;

[0031] a matching module configured to obtain real-time position features and real-time category features of the target object to be tracked at the next time, calculate category feature distances and position feature distances of the target object to be tracked based on a Hungarian algorithm to determine a matching relationship, and complete tracking of the target object to be tracked.

[0032] As one preferred scheme, the first division module is specifically configured to:

[0033] divide the video frames into a plurality of sub-regions by using a fixed grid division strategy;

[0034] traverse boundary box information of all the target objects to be tracked in each sub-region output by the target detection network, and count number information of the target objects to be tracked in each sub-region by judging a positional relationship between the boundary box information and the sub-regions;

[0035] visualize the number information to obtain the target density distribution map.

[0036] As one preferred scheme, the second division module is specifically configured to:

[0037] when the target density distribution map shows that the density of the target object to be tracked in the first sub-region is higher than a preset high-density threshold, the first sub-region is further divided;

[0038] and when the target density distribution map shows that the density of the target object to be tracked in the second sub-region is lower than a preset low-density threshold, adjacent sub-regions of the second sub-region are merged.

[0039] As one of the preferred solutions, the Kalman filtering algorithm and the position feature predict the predicted position feature of the target object to be tracked at the next moment, which is represented as:

[0040]

[0041] wherein F is a state transition matrix, Q is process noise, x t is the position feature of the target at time t, is the position feature prediction value of the target at time t+1 according to the state at time t.

[0042] As one of the preferred solutions, the matching module is specifically configured to:

[0043] construct a bipartite graph according to the category feature, the position feature, the real-time position feature and the real-time category feature of the target object to be tracked;

[0044] calculate a category feature distance according to the category feature and the real-time category feature, and calculate a position feature distance according to the position feature and the real-time position feature;

[0045] calculate a matching scheme with the minimum weighted sum of the category feature distance and the position feature distance in the bipartite graph based on the Hungarian algorithm, and determine the matching relationship between the current moment and the next moment of the target object to be tracked according to the matching scheme.

[0046] Compared with the prior art, the present application has at least one of the following advantages:

[0047] 1) In the target detection and region division stage, the present application can accurately identify various target objects to be tracked by processing the target video frame through an advanced target detection network, which greatly improves the accuracy and robustness of target recognition compared with the traditional method relying on simple threshold segmentation or template matching.

[0048] 2) The present application performs secondary division on the sub-regions based on the target density distribution map, which shows strong adaptive ability. For target dense regions such as crowded squares and congested intersections, further subdividing the sub-regions can effectively reduce the complexity of the targets in each sub-region, avoid confusion during data association, and greatly improve the processing efficiency.

[0049] 3) The application adopts different strategies for different density regions according to the region information after secondary division in the feature extraction link, uses advanced methods such as multi-scale feature fusion in dense areas to ensure that effective class features and position features can be extracted even if the targets are mutually occluded, uses a lightweight feature extraction network in sparse areas to speed up the processing speed while ensuring accuracy, and realizes the rational allocation and efficient use of resources.

[0050] 4) The application uses position features combined with Kalman filtering algorithm to predict the next time position features of the target object to be tracked, fully considers the motion continuity of the target. Moreover, after obtaining the real-time position features and real-time class features of the target at the next time, the matching relationship is determined by calculating the class feature distance and the position feature distance. This dynamic matching mechanism based on feature distance can not only utilize the discriminativeness of the target appearance features, but also take into account the continuity of the motion trajectory. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a flow diagram of a multi-target tracking method in one of the embodiments of the application;

[0052] Figure 2 is a structural diagram of a multi-target tracking device in one of the embodiments of the application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, not all the embodiments. The purpose of providing these embodiments is to make the disclosure of the application more thorough and comprehensive. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the application.

[0054] In the description of the present application, the terms "first", "second", "third" and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second", "third" and the like can explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, the meaning of "multiple" is two or more.

[0055] In the description of the present application, it is necessary to point out that, unless otherwise explicitly defined and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, it can be the communication inside two elements. The terms "vertical", "horizontal", "left", "right", "up", "down" and similar expressions used herein are for the purpose of illustration only, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. The term "and / or" used herein includes any and all combinations of one or more related listed items. For those skilled in the art, the specific meaning of the above terms in the present application can be understood in specific cases.

[0056] In the description of the present application, it is necessary to point out that, unless otherwise defined, all technical and scientific terms used in the present application are the same as those commonly understood by those skilled in the art. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments, and are not intended to limit the present application. For those skilled in the art, the specific meaning of the above terms in the present application can be understood in specific cases.

[0057] An embodiment of the present application provides a multi-target tracking method, specifically, please refer to Figure 1 , Figure 1 The flowchart of the multi-target tracking method in one embodiment of the present application is shown, which includes steps S1-S5:

[0058] S1: based on the target detection network, the video frames of the target video obtained are detected to obtain the target object to be tracked;

[0059] Among them, the target video of the present application is widely derived from the actual scene of different industries needing to track and monitor specific targets, which can specifically include monitoring video of urban streets, video shot by monitoring cameras at traffic intersections, recording video of various sports competition sites, and monitoring video in automatic assembly line workshops.

[0060] Wherein, the target detection network of the application adopts Faster R-CNN, the core of which includes two parts of region proposal network (RPN) and Fast R-CNN. RPN is responsible for quickly generating a series of candidate regions that may contain targets. It slides a window on the feature map and uses anchor box mechanism to preset multiple anchor boxes for different scales and aspect ratios, preliminarily frames the potential target position, and greatly reduces the search space of subsequent detection. Then, Fast R-CNN classifies and regresses the candidate regions, uses the features extracted by the deep convolutional neural network to judge the target categories of each candidate region through the full connection layer, such as common target types such as pedestrians, vehicles and animals, and adjusts the boundary box coordinates to make the target positioning more accurate.

[0061] In this step, when the target detection network is applied to the video frames of the target video, the network first performs preprocessing on the image, usually including image normalization operation to map the pixel value to a unified range to improve the stability and generalization ability of the model. Then, the deep features of the image are gradually extracted through multiple convolution, pooling and other operations. These feature maps contain rich semantic information and can capture key visual features of the target such as shape, texture and color. Based on these features, RPN generates candidate regions, and Fast R-CNN completes the final classification and regression to output the target object information accurately defined by the boundary box. These boundary boxes carry the position coordinates of the target and the corresponding class label, providing accurate and reliable basic data for subsequent multi-target tracking processes such as region division, feature extraction and position prediction, ensuring that the entire tracking system can operate efficiently based on accurate target recognition.

[0062] S2: dividing the video frame into multiple sub-regions, and counting the number of target objects to be tracked in each sub-region to obtain a target density distribution map of the target objects to be tracked;

[0063] Preferably, in one embodiment of the application, the video frame is divided into multiple sub-regions, and the number of target objects to be tracked in each sub-region is counted to obtain a target density distribution map of the target objects to be tracked, comprising:

[0064] The video frame is divided into multiple sub-regions by using a fixed grid division strategy;

[0065] Traverse the boundary box information of all target objects to be tracked in each sub-region output by the target detection network, and count the number of target objects to be tracked in each sub-region by judging the positional relationship between the boundary box information and the sub-region;

[0066] Visualize the number information to obtain a target density distribution map.

[0067] Specifically, in actual operation, the grid parameters need to be reasonably determined according to the size of the video frame and the approximate scale range of the target. In an embodiment of the present application, for a common high-definition video frame (resolution 1920x1080), if the target is mostly small objects or pedestrians, the picture can be evenly divided into 30x20 grid units; if the target is mainly large vehicles, it is more appropriate to divide it into 15x10 grids, ensuring that each sub-region can effectively cover the target and not increase the calculation amount due to too fine division. This fixed grid division method, like a neat "coordinate grid" for complex video pictures, provides a clear framework for subsequent data processing.

[0068] Traverse the bounding box information of all target objects to be tracked in each sub-region output by the target detection network. This step closely follows the target detection step. After the target detection network has accurately identified each target to be tracked and output its bounding box (usually represented by the upper left corner coordinates (x1, y1) and the lower right corner coordinates (x2, y2)), start a detailed investigation of each sub-region. Count the number of information by judging the positional relationship between the bounding box information and the sub-region. The specific judgment logic is as follows: for a sub-region, its upper left corner coordinates are set as (ixw, jxh), and its lower right corner coordinates are set as ((i+1)xw, (j+1)xh), where i and j represent the indices of the sub-region in the horizontal and vertical directions, and w and h are the width and height of the sub-region. If (ixw≤x1<(i+1)xw) and (jxh≤y1<(j+1)xh), or (ixw≤x2<(i+1)xw) and (jxh≤y2<(j+1)xh), then the detection box falls within this sub-region, and the target number counter of the corresponding sub-region is incremented by 1. After traversing all sub-regions and bounding boxes, the number of targets in each sub-region can be accurately obtained.

[0069] Visualize the number information to obtain a target density distribution map, aiming to convert abstract numerical information into intuitive image display. Various visualization tools can be used, such as the Matplotlib library in Python. Taking a heat map as an example, the number of targets in a sub-region is mapped to different color intensities. Regions with a large number of targets display darker colors (such as red), while regions with fewer targets display lighter colors (such as blue), generating a colorful target density distribution map. This map is like a "battlefield situation map", clearly presenting the target density situation in different regions of the video frame, allowing researchers or systems to grasp the target distribution dynamics at a glance, and then performing targeted secondary division, feature extraction, etc. on subsequent sub-regions based on this map, improving the efficiency and accuracy of multi-target tracking.

[0070] S3: performing secondary division on the sub-region according to the target density distribution map, performing feature extraction on the target object to be tracked according to the secondary division region information, and obtaining the category feature and the position feature of the target object to be tracked; wherein the secondary division is designed to further divide according to the density of the target object to be tracked in the sub-region;

[0071] Preferably, in an embodiment of the present application, the secondary division of the sub-region according to the target density distribution map comprises:

[0072] When the target density distribution map shows that the density of the target object to be tracked in the first sub-region is higher than the preset high-density threshold, the first sub-region is further divided;

[0073] and when the target density distribution map shows that the density of the target object to be tracked in the second sub-region is lower than the preset low-density threshold, the adjacent sub-regions of the second sub-region are merged.

[0074] Specifically, in an embodiment of the present application, taking city traffic monitoring as an example, if the vehicle tracking is focused on, considering the driving characteristics and road space distribution of the vehicle, for a high-definition video picture, the high-density threshold can be set to more than 5 vehicles per sub-region on average, and the low-density threshold can be set to less than 1 vehicle per sub-region on average. When the target density distribution map shows that the density of the target object to be tracked in the first sub-region is higher than the preset high-density threshold, it means that the target in this region is too dense, which can easily cause problems such as data association confusion and inaccurate feature extraction, so it is necessary to further divide the first sub-region.

[0075] Among them, there are many methods for further division, and the common one is the quadtree division. The original first sub-region is equally divided into four smaller sub-regions, and then the number of targets in these new sub-regions is re-counted, and the density situation is re-evaluated. If the density of the sub-region is still too high, the quadtree division can be recursively used, until the target density in the sub-region is within a reasonable range. This subdivision method is like fine guiding the congested traffic intersection, which "divides the whole into parts" for the complex target gathering area, reduces the mutual interference between targets in each sub-region, and creates favorable conditions for subsequent accurate feature extraction.

[0076] In contrast, when the target density distribution map shows that the density of the target object to be tracked in the second sub-region is lower than the preset low-density threshold, merging the adjacent sub-regions of the second sub-region is an optimization strategy. The definition of the adjacent sub-regions is usually based on the spatial positional relationship, such as the horizontally or vertically adjacent sub-regions. The merging operation not only reduces the unnecessary waste of computing resources, but also improves the representativeness and stability of the features based on the larger range of regional information during subsequent feature extraction. For example, in the sparse grassland monitoring scene, when tracking the sporadically distributed wild animals, merging the low-density regions can enable the feature extraction network to better capture the activity patterns of the animals in a larger spatial range, avoiding feature fragmentation caused by small sub-regions.

[0077] After completing the secondary division, the features of the target object to be tracked are extracted based on the secondary division region information to obtain the category features and the position features. For the category features, if a deep learning model is used, the target image block is usually processed based on a convolutional neural network. In the dense sub-region, due to the high risk of occlusion, the network structure focuses on multi-scale feature fusion, and uses techniques such as atrous convolution to expand the receptive field, so that even if part of the target appearance is occluded, discriminative category feature vectors can still be extracted, which contain rich target appearance category information such as vehicle type and animal species. In the sparse sub-region, a relatively lightweight network structure is used to focus on key feature extraction, quickly and accurately judge the target category, and improve the processing efficiency.

[0078] In the dense sub-region, the position features are determined in combination with the bounding box coordinates of the target and the next time position information predicted based on the Kalman filtering algorithm. The Kalman filtering algorithm can still accurately predict the position trend of the target in the short term in the target dense and complex motion scene, providing forward-looking information for the position features and assisting subsequent tracking matching. In the sparse sub-region, the position features are mainly obtained based on the current bounding box coordinates of the target, which is simple and direct, and reduces unnecessary computational overhead to realize reasonable allocation and efficient use of resources and comprehensively ensure the accurate execution of the multi-target tracking task.

[0079] S4: predicting the predicted position features of the next time of the target object to be tracked based on the Kalman filtering algorithm and the position features;

[0080] Preferably, in an embodiment of the present application, the predicted position features of the next time of the target object to be tracked are predicted based on the Kalman filtering algorithm and the position features, which are represented as:

[0081]

[0082] wherein F is a state transition matrix, Q is a process noise, x ta position feature of the target at time t, a position feature prediction value of the target at time t+1 according to a state of the target at time t.

[0083] The Kalman filtering algorithm is based on a linear system state space model, and by skillfully fusing historical position information of the target and estimation of dynamic characteristics of the system, accurate calculation of the position of the target at the next time is realized.

[0084] In the step, the Kalman filter is first initialized for each target object to be tracked. The initialization process involves setting the initial state vector of the filter, which usually covers the current position of the target (represented by the horizontal coordinate and the vertical coordinate in the image coordinate system) and the velocity information. The velocity information can be estimated according to the position changes of the target in the previous frames, and the preliminary velocity information can be obtained by simple difference operation, which provides the basic motion parameters for subsequent dynamic prediction.

[0085] However, in actual scenarios, the motion of the target is inevitably disturbed by various external factors, such as the influence of wind on flying objects, the disturbance of road bumps on vehicle driving, etc. These disturbance factors are quantitatively represented by process noise. The covariance matrix of the process noise needs to be carefully tuned according to the specific application scenario and target characteristics, which reflects the estimation degree of the system's uncertainty of the target motion. In complex dynamic scenarios, such as frequent starting and stopping of vehicles at traffic intersections, the value of the process noise should be appropriately increased to adapt to the larger motion changes of the target; while in relatively stable scenarios, such as indoor object monitoring, the process noise can be set smaller. By incorporating the process noise into the prediction model, the Kalman filter can to some extent offset the influence of external disturbances on the prediction accuracy, making the predicted position feature more reliable.

[0086] S5: obtaining real-time position features and real-time category features of the target object to be tracked at the next time, calculating the category feature distance and the position feature distance of the target object to be tracked based on the Hungarian algorithm to determine the matching relationship, and completing the tracking of the target object to be tracked.

[0087] Preferably, in an embodiment of the present application, the category feature distance and the position feature distance of the target object to be tracked are calculated based on the Hungarian algorithm to determine the matching relationship, comprising:

[0088] constructing a bipartite graph according to the category features, the position features, the real-time position features and the real-time category features of the target object to be tracked;

[0089] calculating the category feature distance according to the category features and the real-time category features, and calculating the position feature distance according to the position features and the real-time position features;

[0090] The matching scheme with the minimum weighted sum of the category feature distance and the position feature distance in the bipartite graph is calculated based on the Hungarian algorithm, and the matching relationship between the target object to be tracked at the current time and the next time is determined according to the matching scheme.

[0091] Specifically, for example, in the intelligent traffic monitoring scene, a new frame of video image is transmitted, the target detection network based on deep learning identifies the vehicles, pedestrians and other targets in the picture again, outputs the updated boundary box coordinates of each target as the real-time position feature, and obtains the vehicle type (sedan, SUV, truck, etc.) or the pedestrian posture feature through model discrimination as the real-time category feature. These real-time information reflects the state of the target at the latest time.

[0092] Then, the bipartite graph is constructed according to the category feature, the position feature, the real-time position feature and the real-time category feature of the target object to be tracked. The construction of the bipartite graph is the basic framework of the application of the Hungarian algorithm. The set of target objects to be tracked at the current time is taken as one side of the vertex set (V_1) of the bipartite graph, each vertex represents a target to be matched, and carries the category feature (such as in the form of a high-dimensional vector, containing target appearance category discrimination information, learned by a deep convolutional neural network) and the position feature (containing the next time position predicted based on Kalman filtering and the current actual position information) extracted in the previous step. The other side of the vertex set (V_2) is composed of the real-time target objects obtained at the next time, and the corresponding real-time position feature and real-time category feature are obtained through the latest round of target detection. The bipartite graph is thus constructed, which builds a bridge for the precise matching of targets at different times, and the purpose is to find the most reasonable correspondence between the two sets of vertices, and to ensure the continuity of the target tracking trajectory.

[0093] The category feature distance is calculated according to the category feature and the real-time category feature, and the position feature distance is calculated according to the position feature and the real-time position feature. For the calculation of the category feature distance, since the category feature is usually presented in the form of a high-dimensional vector, a variant method of cosine similarity is often used. In this way, the cosine value of the angle between two vectors in the feature space is converted into a distance measure, and the distance value ranges between (0) and (1). The closer to (0), the more similar the two targets are in category feature, that is, the more likely they are the same target at different times.

[0094] In terms of position feature distance calculation, the coordinate information of the target is mainly used for measurement. The predicted position coordinates and the real-time position coordinates of the target (A) at the next time are known, and the Euclidean distance is used as the measurement standard. This distance directly reflects the deviation degree of the actual position of the target from the predicted position. The smaller the deviation, the more consistent the motion trajectory of the target with the expectation, and the higher the matching probability.

[0095] Finally, a matching scheme with the minimum weighted sum of the category feature distance and the position feature distance in the bipartite graph is calculated based on the Hungarian algorithm, and a matching relationship between the target object at the current time and the next time is determined according to the matching scheme.

[0096] An embodiment of the present application provides a multi-target tracking device, and specifically, refer to Figure 2 , Figure 2 Fig. 1 shows a structural schematic diagram of a multi-target tracking device in an embodiment of the present application, which comprises:

[0097] The detection module 11 is configured to perform target detection on the video frames of the acquired target video based on a target detection network to obtain the target objects to be tracked.

[0098] The first division module 12 is configured to divide the video frames into a plurality of sub-regions, and count the number of the target objects to be tracked in each sub-region to obtain a target density distribution map of the target objects to be tracked.

[0099] The second division module 13 is configured to perform secondary division on the sub-regions according to the target density distribution map, and perform feature extraction on the target objects to be tracked according to the region information of the secondary division to obtain category features and position features of the target objects to be tracked; wherein the secondary division is designed to further divide according to the density of the target objects to be tracked in the sub-regions.

[0100] The prediction module 14 is configured to predict the predicted position features of the target objects to be tracked at the next time based on a Kalman filtering algorithm and the position features.

[0101] The matching module 15 is configured to acquire real-time position features and real-time category features of the target objects to be tracked at the next time, calculate the category feature distance and the position feature distance of the target objects to be tracked based on the Hungarian algorithm to determine the matching relationship, and complete the tracking of the target objects to be tracked.

[0102] Preferably, in an embodiment of the present application, the first division module is specifically configured to:

[0103] divide the video frames into a plurality of sub-regions by using a fixed grid division strategy;

[0104] traverse the bounding box information of all the target objects to be tracked in each sub-region output by the target detection network, and count the number information of the target objects to be tracked in each sub-region by judging the positional relationship between the bounding box information and the sub-region;

[0105] visualize the number information to obtain the target density distribution map.

[0106] Preferably, in an embodiment of the present application, the second division module is specifically configured to:

[0107] when the target density distribution map shows that the density of the target object to be tracked in the first sub-region is higher than the preset high-density threshold, the first sub-region is further divided;

[0108] and when the target density distribution map shows that the density of the target object to be tracked in the second sub-region is lower than the preset low-density threshold, the adjacent sub-regions of the second sub-region are merged.

[0109] Preferably, in an embodiment of the present application, the Kalman filtering algorithm and the position feature are used to predict the predicted position feature of the target object to be tracked at the next time, which is represented as:

[0110]

[0111] wherein F is a state transition matrix, Q is process noise, x t is the position feature of the target at time t, is the predicted value of the position feature of the target at time t+1 according to the state at time t.

[0112] Preferably, in an embodiment of the present application, the matching module is specifically used for:

[0113] constructing a bipartite graph according to the category feature, the position feature, the real-time position feature and the real-time category feature of the target object to be tracked;

[0114] calculating the category feature distance according to the category feature and the real-time category feature, and calculating the position feature distance according to the position feature and the real-time position feature;

[0115] calculating the matching scheme with the minimum weighted sum of the category feature distance and the position feature distance in the bipartite graph based on the Hungarian algorithm, and determining the matching relationship between the current time and the next time of the target object to be tracked according to the matching scheme.

[0116] Compared with the prior art, the present application has at least one of the following advantages:

[0117] 1) In the target detection and region division stage of the present application, the target video frame is processed by an advanced target detection network, which can accurately identify various target objects to be tracked. Compared with the traditional method relying on simple threshold segmentation or template matching, the accuracy and robustness of target recognition are greatly improved.

[0118] 2) The present application performs secondary division on the sub-regions based on the target density distribution map, which shows strong adaptive ability. For target dense areas such as crowded squares and congested intersections, further subdividing the sub-regions can effectively reduce the complexity of the targets in each sub-region, avoid confusion during data association, and greatly improve the processing efficiency.

[0119] 3) In the feature extraction link, according to the region information after secondary division, different strategies are adopted for different density regions, in the dense area, advanced methods such as multi-scale feature fusion are used to ensure that even if the targets are occluded, effective class features and position features can still be extracted; in the sparse area, a lightweight feature extraction network is used to speed up the processing speed while ensuring the accuracy, realizing the reasonable allocation and efficient use of resources.

[0120] 4) The present application uses position features combined with Kalman filtering algorithm to predict the next time position features of the target object to be tracked, fully considering the motion continuity of the target. Moreover, after obtaining the real-time position features and real-time class features of the target at the next time, the matching relationship is determined by calculating the class feature distance and the position feature distance. This dynamic matching mechanism based on feature distance can not only utilize the discriminability of target appearance features, but also take into account the continuity of motion trajectory.

[0121] The above-mentioned embodiments only express several embodiments of the present application, and the description is more specific and detailed, but it cannot be understood as a limitation on the scope of the patent of the present application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A multi-target tracking method, characterized in that, include: The target object to be tracked is obtained by performing target detection on the video frames of the acquired target video based on the target detection network. The video frame is divided into multiple sub-regions, and the number of target objects to be tracked in each sub-region is counted to obtain a target density distribution map of the target objects to be tracked. The sub-region is further divided based on the target density distribution map, and the features of the target object to be tracked are extracted based on the information of the second division of the region to obtain the category features and location features of the target object to be tracked. The secondary partitioning is designed to further divide the target object to be tracked into sub-regions based on the density of the target object in the sub-region; Based on the Kalman filter algorithm and the aforementioned position features, the predicted position features of the target object to be tracked at the next moment are predicted. The real-time location features and real-time category features of the target object to be tracked are obtained at the next moment. The category feature distance and location feature distance of the target object to be tracked are calculated based on the Hungarian algorithm to determine the matching relationship and complete the tracking of the target object to be tracked.

2. The multi-target tracking method as described in claim 1, characterized in that, The step of dividing the video frame into multiple sub-regions and counting the number of target objects to be tracked in each sub-region to obtain a target density distribution map of the target objects to be tracked includes: The video frame is divided into multiple sub-regions using a fixed grid partitioning strategy; The target detection network is traversed to output the bounding box information of all the target objects to be tracked in each sub-region. By judging the positional relationship between the bounding box information and the sub-region, the number of target objects to be tracked in each sub-region is counted. The quantity information is visualized to obtain the target density distribution map.

3. The multi-target tracking method as described in claim 1, characterized in that, The step of further dividing the sub-region based on the target density distribution map includes: When the target density distribution map shows that the density of the target object to be tracked in the first sub-region is higher than the preset high density threshold, the first sub-region is further divided. And when the target density distribution map shows that the density of the target object to be tracked in the second sub-region is lower than a preset low density threshold, the adjacent sub-regions of the second sub-region are merged.

4. The multi-target tracking method as described in claim 1, characterized in that, The predicted position features of the target object to be tracked at the next moment, based on the Kalman filter algorithm and the position features, are expressed as follows: Where F is the state transition matrix, Q is the process noise, and x t Let be the positional characteristics of the target at time t. The target is predicted based on the positional characteristics of the state at time t+1.

5. The multi-target tracking method as described in claim 1, characterized in that, The calculation of the category feature distance and location feature distance of the target object to be tracked based on the Hungarian algorithm to determine the matching relationship includes: A bipartite graph is constructed based on the category features, location features, real-time location features, and real-time category features of the target object to be tracked. The category feature distance is calculated based on the category feature and the real-time category feature, and the location feature distance is calculated based on the location feature and the real-time location feature; The matching scheme with the minimum weighted sum of the category feature distance and the position feature distance in the bipartite graph is calculated based on the Hungarian algorithm, and the matching relationship between the target object to be tracked at the current time and the next time is determined according to the matching scheme.

6. A multi-target tracking device, characterized in that, include: The detection module is used to perform target detection on the video frames of the acquired target video based on the target detection network to obtain the target object to be tracked. The first segmentation module is used to divide the video frame into multiple sub-regions and count the number of target objects to be tracked in each sub-region to obtain a target density distribution map of the target objects to be tracked. The second segmentation module is used to perform secondary segmentation of the sub-region based on the target density distribution map, and to extract features of the target object to be tracked based on the secondary segmentation region information to obtain the category features and location features of the target object to be tracked. The secondary partitioning is designed to further divide the target object to be tracked into sub-regions based on the density of the target object in the sub-region; The prediction module is used to predict the predicted position features of the target object to be tracked at the next moment based on the Kalman filter algorithm and the position features. The matching module is used to obtain the real-time location features and real-time category features of the target object to be tracked at the next moment, calculate the category feature distance and location feature distance of the target object to be tracked based on the Hungarian algorithm to determine the matching relationship, and complete the tracking of the target object to be tracked.

7. The multi-target tracking device as described in claim 6, characterized in that, The first partitioning module is specifically used for: The video frame is divided into multiple sub-regions using a fixed grid partitioning strategy; The target detection network is traversed to output the bounding box information of all the target objects to be tracked in each sub-region. By judging the positional relationship between the bounding box information and the sub-region, the number of target objects to be tracked in each sub-region is counted. The quantity information is visualized to obtain the target density distribution map.

8. The multi-target tracking device as described in claim 6, characterized in that, The second partitioning module is specifically used for: When the target density distribution map shows that the density of the target object to be tracked in the first sub-region is higher than the preset high density threshold, the first sub-region is further divided. And when the target density distribution map shows that the density of the target object to be tracked in the second sub-region is lower than a preset low density threshold, the adjacent sub-regions of the second sub-region are merged.

9. The multi-target tracking device as described in claim 6, characterized in that, The predicted position features of the target object to be tracked at the next moment, based on the Kalman filter algorithm and the position features, are expressed as follows: Where F is the state transition matrix, Q is the process noise, and x t Let be the positional characteristics of the target at time t. The target is predicted based on the positional characteristics of the state at time t+1.

10. The multi-target tracking device as described in claim 6, characterized in that, The matching module is specifically used for: A bipartite graph is constructed based on the category features, location features, real-time location features, and real-time category features of the target object to be tracked. The category feature distance is calculated based on the category feature and the real-time category feature, and the location feature distance is calculated based on the location feature and the real-time location feature; The matching scheme with the minimum weighted sum of the category feature distance and the position feature distance in the bipartite graph is calculated based on the Hungarian algorithm, and the matching relationship between the target object to be tracked at the current time and the next time is determined according to the matching scheme.

Images