Multi-target tracking system and application based on distance intersection-union ranking and matching

Abstract

This invention discloses a multi-target tracking system and its application based on distance intersection ratio (DIoU) ranking matching, belonging to the field of multi-target tracking technology. The multi-target tracking system includes a target detection module, a matching module, and a trajectory prediction module. The target detection module outputs the bounding box information of observed targets and removes redundant targets. The matching module calculates the distance intersection ratio (DIoU) between all observed targets and all predicted targets, generates a DIoU matrix, and performs DIoU ranking matching to filter out successfully matched and unmatched target information. For successfully matched predicted targets, the prediction state is updated; for unmatched predicted targets, the confidence level is decayed. All unmatched observed target information is sent to the trajectory prediction module for motion state prediction to obtain predicted target trajectory information. Finally, the predicted target trajectory information is sent to the matching module. This invention can effectively reduce the frame rate requirement for system trajectory prediction and improve the overall tracking quality.

Description

Technical Field

[0001] This invention relates to the field of multi-target tracking technology, and in particular to a multi-target tracking system and its application based on distance intersection ratio (DIRR) sorting matching. Background Technology

[0002] Multiple Object Tracking (MOT) aims to estimate the trajectory of objects based on their identities within image sequences or videos. Due to its academic and commercial potential, MOT has gained increasing attention in computer vision and has become a significant research area in the field in recent years, achieving considerable progress. With the application of MOT technology in intelligent security, higher demands are being placed on its real-time performance and accuracy.

[0003] Common multi-target tracking methods include optical flow, kernel correlation filter (KCF) based tracking algorithms, and deep learning-based tracking algorithms. Among these, to ensure more stable detection distance and accuracy, current mainstream multi-target tracking applications increasingly use deep learning-based algorithms for basic target detection, followed by tracking algorithms for auxiliary trajectory prediction. For details, see [link to relevant documentation]. Figure 1 As shown, the main process of current deep learning-based multi-object tracking algorithms is illustrated. It typically includes two parts: detection and tracking. After acquiring the input image data, the image data is input into the object detection module. The object detection model detects multiple target objects appearing in each input image frame (obtaining the bounding boxes of the target objects). Then, the multiple target objects appearing in each image frame are matched. During the matching process, feature extraction (usually visual and motion features) and motion prediction are performed on each detected target object. Then, target matching is performed by calculating the similarity of features to obtain the tracking trajectory information of each target object.

[0004] In the matching process, a key technique is estimating the model's motion state based on historical data. This typically employs a Kalman filter-based system state estimation method, which usually includes two processes: prediction and update. First, let the motion state variables of each object be:

[0005] X = [xysrv] x v y v s ] T

[0006] Where x, y, s, and r represent the current x-axis coordinate, y-axis coordinate, bounding box size (area), and aspect ratio (i.e., the ratio of the length to the width of the bounding box), respectively, and v x v y v sThese represent the rate of change of the object's current x-axis coordinate (i.e., velocity along the x-axis), y-axis coordinate (i.e., velocity along the y-axis), and bounding box size (area), respectively. The corresponding linear motion equations are as follows:

[0007] X k+1 =FX k +W k

[0008] Z k =HX k +V k

[0009]

[0010]

[0011] Where k is the state frame; X k X represents the system's predicted motion state at frame k. k+1 Let F represent the system's predicted motion state at frame k+1, and let W be the state transition matrix. k The noise is approximately Gaussian (system process noise); Z k V represents the actual observed state (measurement state) at frame k, H is the state transition matrix (measurement matrix), and V k The noise is approximately Gaussian (measurement noise); Q and P are the covariance matrices. For X k The estimated value, For X k-1 The estimated value, For P k The estimated value, P k-1 This is the covariance matrix at frame k-1 (the object's state uncertainty is propagated at each time step). The algorithm calculates the Kalman gain K (also known as the Kalman coefficient) based on the difference between the predicted and actual observations, thereby generating the predicted target position and calculating its matching degree with the actual observed target (measured target), thus matching and correcting the existing motion trajectory. When matching any two rectangular targets to calculate the matching degree, the intersection-over-union (IoU) ratio is typically used as a metric. IoU represents the degree of overlap between two targets; it is the ratio of the intersection to the union of the two rectangular target boxes. Figure 2 As shown, the formula for calculating IoU is as follows:

[0012]

[0013] Among them, S A S represents the area covered by the target bounding box A.B S0 represents the coverage area of ​​target box B, and S0 represents the overlapping area of ​​target boxes A and B. The IoU value is between [0, 1]. When IoU = 0, the two rectangular targets do not overlap; when IoU = 1, the two rectangular targets overlap. Based on these metrics, it can be determined whether the predicted target and the actual observed target are the same target. Whether the predicted target and the actual observed target are sufficiently close, and whether they can be correctly matched, directly determines the stability and recall rate of the algorithm.

[0014] Traditional multi-target tracking algorithms have the following drawbacks:

[0015] 1) Linear models are typically used to predict the motion state of a target, which can lead to difficulties in matching the predicted target with the actual observed target when the input frame rate decreases or the object is moving at varying speeds or at high speeds. To address this issue, existing technologies have introduced prediction models that incorporate acceleration motion. However, due to the IoU calculation mechanism, situations where the IoU between the predicted target and the actual observed target is zero can still occur, potentially leading to problems such as lost predicted trajectories. Furthermore, when using IoU-based threshold-based bounding box matching, if multiple predicted targets simultaneously match the actual observed target, it can result in duplicate matching of target identities (i.e., target ID or target id) and target identity swapping (i.e., tracking target identity swapping, also known as IDSwitches).

[0016] 2) In scenarios using deep learning for object detection, a common phenomenon is the trade-off between single-frame detection quality and trajectory prediction performance. For example, when performing object detection on a single frame, to improve detection quality, a neural network model with a larger number of parameters is often used (which increases the computational cost of single-frame detection). To ensure real-time tracking, operations such as interval sampling are typically performed on the video to balance the additional computational cost. These operations reduce the input frame rate for prediction, increasing the difficulty of the tracking algorithm and affecting trajectory prediction quality. Considering the overall performance of the tracking system, existing multi-object tracking methods struggle to simultaneously guarantee both single-frame detection quality and trajectory prediction quality while maintaining the output frame rate. Summary of the Invention

[0017] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a multi-target tracking system and application based on distance intersection-union (DIoU) ranking matching. The multi-target tracking scheme provided by this invention uses DIoU ranking in the matching module to match observed targets with predicted targets, and utilizes DIoU ranking matching information to assist the trajectory prediction module in trajectory prediction. This allows the trajectory prediction part to match a wider range of observed targets and adjust the optimal target based on the matching information, thereby reducing the frame rate requirement of the system's trajectory prediction part. This achieves stable tracking in low input frame rate or high-speed motion scenarios and improves the overall tracking quality.

[0018] To achieve the above objectives, the present invention provides the following technical solution:

[0019] A multi-target tracking system based on distance intersection ratio (DRM) ranking matching includes a target detection module, a matching module, and a trajectory prediction module, wherein the matching module is connected to the target detection module and the trajectory prediction module.

[0020] The target detection module is used to perform target detection on the input single-frame image to obtain the bounding box information of the observed targets in the single-frame image and remove redundant targets;

[0021] The matching module includes a DIoU matrix generation submodule, an optimal match filtering submodule, and a tracking trajectory output submodule. The DIoU matrix generation submodule is connected to the target detection module and is used to acquire all observed target information after removing redundant targets, calculate the distance intersection-union ratio (DIoU) of all observed targets with all current predicted targets, and generate a DIoU matrix. The optimal match filtering submodule is used to sort and match according to the aforementioned DIoU matrix to filter out successfully matched and unmatched target information, update the prediction state for successfully matched predicted targets, decay the confidence of unmatched predicted targets, and send all unmatched observed target information to the trajectory prediction module. The tracking trajectory output submodule is used to output the updated tracking trajectory and retain all predicted targets until the next frame image input.

[0022] The trajectory prediction module is used to predict the motion state of each unmatched observation target to obtain the predicted target trajectory information, and send the predicted target trajectory information to the aforementioned tracking trajectory output submodule.

[0023] Furthermore, the trajectory prediction module includes a target initialization submodule, a tracking trajectory pool submodule, and a target trajectory prediction module;

[0024] The target initialization submodule is connected to the aforementioned optimal matching and filtering submodule. It is used to obtain information on all unmatched observation targets, establish an initial motion model including acceleration for each unmatched observation target, and initialize the parameters of the motion model.

[0025] The tracking trajectory pool submodule is used to update the tracking trajectory pool according to the motion model constructed by the target initialization submodule, and to update the tracking trajectory pool after obtaining the tracking trajectory update information fed back by the aforementioned tracking trajectory output submodule.

[0026] The target trajectory prediction module is used to establish motion equations for the observed target and predict the target state at the next moment to obtain the predicted target trajectory information, and to send the predicted target trajectory information to the aforementioned tracking trajectory output submodule to update the tracking trajectory of the corresponding target.

[0027] Furthermore, the target detection module includes a convolutional neural network target detection model with fixed parameters;

[0028] The target detection model is configured to: receive single-frame image data input, identify the targets in the input image data, output the bounding box coordinates and size information (x, y, w, h) of the observed targets, and then perform non-maximum suppression processing to remove redundant targets; where x, y, w, and h represent the x-axis coordinate value, y-axis coordinate value, bounding box width, and bounding box height of the bounding box center, respectively.

[0029] Furthermore, the DIoU matrix generation submodule is configured to perform the following processing to generate the DIoU matrix:

[0030] Obtain all observed targets A after removing redundant targets. M×4 Information, and all current forecast targets B N×4 Information, where M is the total number of observed targets, N is the total number of predicted targets, and M and N are integers greater than 1;

[0031] For A M×4 Each observation target A in i Calculate the observed target A i With B N×4 Each predicted target B in j The DIoU value between D ij Thus, all observed targets A are obtained. M×4 With all predicted targets B N×4 DIoU matrix D M×N ,in,

[0032]

[0033] In the formula, i represents AM×4 The i-th observed target in the dataset, i = 1, 2, ..., M;

[0034] j represents B N×4 The j-th prediction target in the dataset, where j = 1, 2, ..., N;

[0035] d represents the observed target A i With predicted target B j The length of the line connecting the centroids of the two points;

[0036] c represents the observed target A i With predicted target B j The length of the diagonal of the circumscribed rectangle between them;

[0037] D M×N The matrix contains M×N elements.

[0038] Furthermore, the optimal matching filtering submodule is configured to perform the following processing for DIoU sorting matching:

[0039] Step 1: Obtain the DIoU matrix D M×N All elements D ij Press D ij The values ​​are sorted from largest to smallest to obtain set H;

[0040] Step 2: Take the first element D from set H. m，n Delete the DIoU matrix D M×N The corresponding element D m，n Rows and columns D m D n At the same time, delete the corresponding element D from set H. m，n This yields the updated set H;

[0041] Step 3: Repeat step 2 until the DIoU matrix D is reached. M×N There are no elements in the set H; at this time, the index rows and columns i and j of all elements taken out of the set H are the indices of the successfully matched observation target and prediction target pairs, and the index rows and columns i and j of the remaining elements in the set H are the indices of the unmatched observation targets and prediction targets.

[0042] Furthermore, for all successfully matched predicted targets, the Kalman gain K of the predicted target is updated using the corresponding observed target value, and the predicted state is updated as follows:

[0043]

[0044]

[0045]

[0046]

[0047] In the formula, k is the state frame, F and H are the state transition matrices, and F T H T These are the transposes of matrices F and H, respectively; P, Q, and R are the covariance matrices; and K is the Kalman gain. k Let X be the observed target value at frame k. k Let K be the predicted value at frame k. k P represents the Kalman gain at frame k. k Let P be the covariance matrix at frame k. k-1 Let be the covariance matrix at frame k-1. For X k The estimated value, For P k The estimated value;

[0048] For all unsuccessfully matched predicted targets, the confidence level of the predicted target is reduced based on a preset confidence decay algorithm, which is set by the system or the user.

[0049] Furthermore, in the initial motion model including acceleration, the motion state of each unmatched observed target is defined as:

[0050] X = [xysrv] x v y v s a x a y ] T

[0051] In the formula, x, y, s, and r represent the current x-axis coordinates of the observed target, respectively. y Axis coordinate values, bounding box area, and bounding box aspect ratio, v x v y v s These represent the current x-axis coordinates, y-axis coordinates of the observed target, and the rate of change of the bounding box area, respectively. x a y These represent the accelerations of the observed target along the x-axis and y-axis, respectively.

[0052] The target initialization submodule is configured to initialize the covariance matrices P, Q, and R in the motion state equations of the motion model, as well as the state transition matrices F and H. All of the above parameters are static parameters and are set by the system or the user.

[0053] Furthermore, the target trajectory prediction module is configured as follows:

[0054] For each unmatched observed target, the target state at the next moment is predicted according to the equation of motion, as follows:

[0055] X k+1 =FX k +W k

[0056] Z k =HX k +V k

[0057]

[0058] Where k is the state frame, X k X represents the system's predicted motion state at frame k. k+1 Z represents the system's predicted motion state at frame k+1. k This represents the actual observed state at frame k, where F and H are the state transition matrices, and W is the actual state observed at frame k. k V k The approximate Gaussian noise at frame k is... For X k The estimated value, For X k-1 The estimated value.

[0059] Furthermore, the matching module also includes a matching deduplication module. In this case, the optimal matching filtering submodule can send the matching results of the observed target and the predicted target to the matching deduplication module.

[0060] The matching and deduplication module is used to receive the matching results sent by the optimal matching and filtering submodule, perform target deduplication processing according to the matching results, and feed back the deduplicated observation target and prediction target information to the DIoU matrix generation submodule.

[0061] The DIoU matrix generation submodule can update the DIoU matrix based on the deduplication information provided in the feedback.

[0062] This invention also provides a multi-target tracking method based on distance intersection-union ranking matching, comprising the following steps:

[0063] Receive input image;

[0064] For each image frame of the input image, the following steps are performed: The target detection module performs target detection on the single frame image to obtain the bounding box information of the observed targets in that single frame image, and removes redundant targets before sending it to the matching module; the matching module obtains all observed target information after removing redundant targets, calculates the Distance Intersection over Union (DIoU) of all observed targets with all currently predicted targets, generates a DIoU matrix, sorts and matches according to the aforementioned DIoU matrix to filter out successfully matched and unmatched target information, updates the prediction state for successfully matched predicted targets, decays the confidence of unmatched predicted targets, sends all unmatched observed target information to the trajectory prediction module, outputs the updated tracking trajectory, and retains all predicted targets until the next frame image input; wherein, the trajectory prediction module can predict the motion state of each unmatched observed target to obtain the predicted target trajectory information, and sends the predicted target trajectory information to the aforementioned matching module;

[0065] Output the target's identity information from the input image.

[0066] By adopting the above technical solution, this invention, as an example, has the following advantages and positive effects compared with the prior art:

[0067] Compared to traditional multi-target tracking algorithms that use IoU thresholding for matching, this invention provides a matching optimization scheme employing sorted non-maximum suppression (DIoU) filtering. This scheme leverages the characteristic of DIoU to compute non-intersecting targets, breaking the rigid requirement of overlapping targets in adjacent frames in traditional algorithms, thereby reducing the frame rate requirement for the system's trajectory prediction component. Furthermore, using DIoU sorted matching to assist prediction allows the trajectory prediction component to match a wider range of observed targets and adjust the optimal target based on the matching information. This improves upon a series of problems caused by traditional threshold filtering, further enhancing overall tracking quality and achieving stable tracking in low input frame rates or high-speed motion scenarios.

[0068] On the other hand, an acceleration-based target velocity model is proposed, which, compared with the traditional linear velocity model, can support more complex motion scenarios and supports higher target initial velocities or lower input frame rates.

[0069] This invention, based on optimized target matching prediction, further provides an optimized acceleration motion model, which can effectively reduce the frame rate requirement of the system trajectory prediction part and improve the overall tracking quality. This invention does not conflict with existing multi-target tracking schemes and can be applied to any system using IoU threshold matching to optimize both the target matching and trajectory prediction parts. Attached Figure Description

[0070] Figure 1This is a schematic diagram of a deep learning-based multi-target tracking process provided in the prior art.

[0071] Figure 2 This is a schematic diagram of the Intersection over Union (IoU) calculation provided in the prior art.

[0072] Figure 3 This is a schematic diagram of the logical structure of the multi-target tracking system based on DIoU sorting and matching provided by the present invention. Detailed Implementation

[0073] The following detailed description, in conjunction with the accompanying drawings and specific embodiments, provides a further detailed account of the multi-target tracking system and its applications based on distance intersection-union ranking matching disclosed in this invention. It should be noted that the technical features or combinations of technical features described in the following embodiments should not be considered isolated; they can be combined to achieve better technical effects. In the accompanying drawings of the following embodiments, the same reference numerals in each drawing represent the same features or components, which can be applied to different embodiments. Therefore, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0074] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings are merely for illustrative purposes and to aid those skilled in the art in understanding and reading the invention. They are not intended to limit the conditions under which the invention can be implemented. Any modifications to the structure, changes in proportions, or adjustments to size, provided they do not affect the effectiveness or purpose of the invention, should fall within the scope of the technical content disclosed in the invention. The scope of the preferred embodiments of the present invention includes other implementations, wherein functions may be performed not in the order stated or discussed, including substantially simultaneously or in reverse order, depending on the functions involved. This should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0075] Techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0076] Example

[0077] See Figure 3The image shown illustrates a multi-target tracking system provided in this embodiment, specifically a real-time multi-target tracking system based on distance intersection-comparison ratio (DCR) ranking matching. This multi-target tracking system can locate moving objects within an input image sequence and identify moving objects in different image frames to obtain object identity information (i.e., object IDs). These objects can be any type of object, such as people, vehicles, various animals, etc., without any limitations.

[0078] The multi-target tracking system adopts a target detection-based multi-target tracking scheme, mainly including a target detection module, a matching module, and a trajectory prediction module. The matching module connects the target detection module and the trajectory prediction module. The target detection module is used to perform target detection on the input single-frame image using a target detection algorithm, obtaining the bounding box and features of the detected targets (usually using a deep neural network to extract features of each detected target). The matching module and the trajectory prediction module are mainly used to perform auxiliary trajectory prediction for the targets using a tracking algorithm. A tracker is set for each target, and each tracker stores the historical features of its respective target. When predicting the target position in the current image frame, the tracker information from the previous image frame can be used. The main process of the tracking algorithm is as follows: using a Kalman filter (Kalman filtering can predict the current position based on the target's position at previous times) to predict the target position in the current image frame to obtain the predicted target box; calculating the matching degree between the observed target and the predicted target in the current image frame to obtain the matching result; performing data association (common algorithms such as the Hungarian algorithm) to assign a target ID to each target; and outputting the target ID. This invention mainly improves the matching and prediction parts of the tracking algorithm process.

[0079] In this embodiment, the target detection module is used to perform target detection on the input single-frame image to obtain the bounding box information of the observed targets in the single-frame image and remove redundant targets. Specifically, the target detection module preferably adopts a convolutional neural network target detection model with fixed parameters. The model size and parameter size can be set by the system or user according to hardware efficiency requirements and specific device performance, which is existing technology and will not be elaborated here. After receiving the single-frame image data input, the target detection model identifies the targets in the input image data, and then outputs the bounding box coordinates and size information (x, y, w, h) of the observed targets. Then, non-maximum suppression (NMS) processing is performed to remove redundant targets. The x, y, w, and h represent the x-axis coordinate value, y-axis coordinate value, bounding box width, and bounding box height, respectively. Non-maximum suppression processing can be implemented using various existing NMS algorithms, which will not be elaborated here.

[0080] Preferably, the object detection model can be any object detection model such as YOLO object detection model, Mask-RCNN object detection model, Fast-RCNN object detection model, or Faster-RCNN object detection model.

[0081] The matching module may include a DIoU matrix generation submodule, an optimal matching filtering submodule, and a tracking trajectory output submodule.

[0082] The DIoU matrix generation submodule is connected to the target detection module. It is used to obtain information on all observed targets after removing redundant targets, calculate the distance intersection-union ratio (DIoU) between all observed targets and all current predicted targets, and generate a DIoU matrix.

[0083] The optimal matching filtering submodule is used to sort and match according to the aforementioned DIoU matrix to filter out the target information that is successfully matched and unmatched—that is, to filter out the predicted targets and observed targets that are successfully matched, the predicted target information that is unmatched, and the observed target information that is unmatched; to update the prediction status for the predicted targets that are successfully matched, to decay the confidence of the predicted targets that are unmatched, and to send all the observed target information that is unmatched to the trajectory prediction module.

[0084] The tracking trajectory output submodule is used to output the updated tracking trajectory and retain all predicted targets until the next frame image input.

[0085] In this embodiment, the DIoU matrix generation submodule is configured to perform the following processing to generate the DIoU matrix.

[0086] First, obtain all observed targets A after removing redundant targets. M×4 Information, and all current forecast targets B N ×4 Information, where M is the total number of observed targets, N is the total number of predicted targets, and M and N are integers greater than 1. That is, A M×4 Includes M observation targets, B N×4 It includes N prediction targets, and the input information for each target is 4 items - namely the bounding box coordinates and size information (x, y, w, h) of the target. X, y, w, h represent the x-axis coordinate value of the center of the bounding box, the y-axis coordinate value, the width of the bounding box, and the height of the bounding box, respectively.

[0087] Then, for A M×4 Each observation target A in i Calculate the observed target A i With B N×4 Each predicted target B in j The DIoU value between D ijThus, all observed targets A are obtained. M×4 With all predicted targets B N×4 DIoU matrix D M×N ,in,

[0088]

[0089] In the formula, i represents A M×4 The i-th observed target in the dataset, i = 1, 2, ..., M;

[0090] j represents B N×4 The j-th prediction target in the dataset, where j = 1, 2, ..., N;

[0091] d represents the observed target A i With predicted target B j The length of the line connecting the centroids of the two points;

[0092] c represents the observed target A i With predicted target B j The length of the diagonal of the circumscribed rectangle between them;

[0093] D M×N The matrix contains M×N elements.

[0094] In this embodiment, the optimal matching filtering submodule is configured to perform the following processing to perform DIoU sorting matching.

[0095] Step 1: Obtain the DIoU matrix D M×N All elements D ij Press D ij The values ​​are sorted from largest to smallest to obtain set H. As an example, and not a limitation, this could be the distance intersection-union (DIoU) value D calculated between observed target A2 and predicted target B3. 23 For all D ij The maximum value in D is... 23 It is the first element in set H, and is represented as D in set H. 2，3 D i|i＝2，j|j＝3 .

[0096] Step 2: Take the first element D from set H. m，n D i|i＝m，j|j＝n Delete the DIoU matrix D M×N The corresponding element D m，n Rows and columns D m D n At the same time, delete the corresponding element D from set H. m，n This yields the updated set H. As an example, and not a limitation, consider the first element D in set H. 2，3 —That is, m=2, n=3; in matrix DM×N The corresponding element D 2，3 The rows and columns are D2 and D3, i.e., D M×N Delete row 2 and column 3; delete D M×N D corresponds to 2，3 Rows and columns D2 and D3 — that is, delete D M×N Remove all elements in the second row and third column of set H; simultaneously, delete the corresponding element D from set H. 2，3 The first element in set H is adjusted to be in the order of D. 2，3 The next element after that.

[0097] Step 3: Repeat step 2 until the DIoU matrix D is reached. M×N There are no elements in the set. At this point, the index rows and columns i and j of all elements removed from the set H are the indices of the successfully matched observation and prediction target pairs, and the index rows and columns i and j of the remaining elements in the set H are the indices of the unmatched observation and prediction targets.

[0098] For all successfully matched predicted targets, update the Kalman gain K of the predicted target using the corresponding observed target value and update the predicted state. The specific update process is as follows:

[0099]

[0100]

[0101]

[0102]

[0103] In the formula, k is the state frame, F and H are the state transition matrices, and F T H T These are the transposes of matrices F and H, respectively; P, Q, and R are the covariance matrices; and K is the Kalman gain. k Let X be the observed target value at frame k. k Let K be the predicted value at frame k. k P represents the Kalman gain at frame k. k Let P be the covariance matrix at frame k. k-1 Let be the covariance matrix at frame k-1. For X k The estimated value, For P k The estimated value;

[0104] For all unmatched predicted targets, the confidence level of the predicted target is reduced based on a preset confidence reduction algorithm, which is set by the system or the user. Specifically, the user can select the corresponding reduction algorithm according to the actual application environment, referring to existing technologies, which will not be elaborated upon here.

[0105] In this embodiment, the trajectory prediction module is used to predict the motion state of each unmatched observation target to obtain the predicted target trajectory information, and send the predicted target trajectory information to the aforementioned tracking trajectory output submodule.

[0106] Specifically, the trajectory prediction module may include a target initialization submodule, a tracking trajectory pool submodule, and a target trajectory prediction module.

[0107] The target initialization submodule is connected to the aforementioned optimal matching and filtering submodule. It is used to obtain information on all unmatched observation targets, establish an initial motion model including acceleration for each unmatched observation target, and initialize the parameters of the motion model.

[0108] Specifically, in the motion model, the motion state of each unmatched observed target is defined as:

[0109] X = [xysrv] x v y v s a x a y ] T

[0110] In the formula, x, y, s, and r represent the current x-axis coordinates, y-axis coordinates, bounding box area, and bounding box aspect ratio of the observed target, respectively. x v y v s These represent the current x-axis coordinates, y-axis coordinates of the observed target, and the rate of change of the bounding box area, respectively. x a y These represent the accelerations of the observed target along the x-axis and y-axis, respectively.

[0111] The target initialization submodule initializes the covariance matrices P, Q, and R, as well as the state transition matrices F and H, in the motion state equations of the motion model. These parameters are all static and can be set by the system or user according to actual needs. The setting and selection of these parameters can be referenced from existing technologies and will not be elaborated further here.

[0112] The tracking trajectory pool submodule is used to update the tracking trajectory pool according to the motion model constructed by the target initialization submodule, and to update the tracking trajectory pool after obtaining the tracking trajectory update information fed back by the aforementioned tracking trajectory output submodule.

[0113] The target trajectory prediction module is used to establish motion equations for the observed target and predict the target state at the next moment to obtain the predicted target trajectory information, and to send the predicted target trajectory information to the aforementioned tracking trajectory output submodule to update the tracking trajectory of the corresponding target.

[0114] Specifically, the target trajectory prediction module is configured to: for each unmatched observed target, predict the target state at the next moment based on the motion state equation, as follows:

[0115] X k+1 =FX k +W k

[0116] Z k =HX k +V k

[0117]

[0118] Where k is the state frame, X k X represents the system's predicted motion state at frame k. k+1 Z represents the system's predicted motion state at frame k+1. k This represents the actual observed state at frame k, where F and H are the state transition matrices, and W is the actual state observed at frame k. k V k The approximate Gaussian noise at frame k is... For X k The estimated value, For X k-1 The estimated value.

[0119] See Figure 3 As shown in this embodiment, the matching module may further include a matching deduplication module.

[0120] At this point, the optimal matching filtering submodule sends the matching results of the observed target and the predicted target to the matching deduplication module. The matching deduplication module receives the matching results from the optimal matching filtering submodule, performs target deduplication based on the matching results, and feeds back the deduplicated observed target and predicted target information to the DIoU matrix generation submodule. The DIoU matrix generation submodule updates the DIoU matrix based on the aforementioned deduplicated information.

[0121] This invention employs a matching optimization scheme based on sorted non-maximum suppression (MOPS) filtering. This expands the matching range of predicted targets and can filter out the optimal matching mode. It significantly improves upon traditional multi-target tracking algorithms that rely solely on IoU threshold filtering and suffer from issues such as duplicate matching, swapping, and loss of identities due to excessive prediction bias in high-speed targets. Furthermore, it reduces the frame rate requirement for the system's trajectory prediction component, enhancing overall tracking quality. Moreover, the trajectory prediction uses an acceleration-based target velocity model. Compared to traditional linear velocity models that do not consider acceleration, this model supports more complex motion scenarios, higher initial target velocities, or lower input frame rates, improving system adaptability.

[0122] Another embodiment of the present invention provides a multi-target tracking method based on distance intersection ratio (DIR) sorting and matching, the multi-target tracking method comprising the following steps.

[0123] S100 receives the input image.

[0124] S200, perform the following steps for each image frame of the input image:

[0125] The target detection module performs target detection on a single frame image to obtain the bounding box information of the observed targets in that single frame image, and sends the information to the matching module after removing redundant targets. The matching module obtains the information of all observed targets after removing redundant targets, calculates the distance intersection-union ratio (DIoU) between all observed targets and all current predicted targets, and generates a DIoU matrix. Based on the aforementioned DIoU matrix, it sorts and matches to filter out the target information that is successfully matched and unmatched. The prediction state of the predicted targets that are successfully matched is updated, and the confidence of the predicted targets that are unmatched is decayed. The information of all unmatched observed targets is sent to the trajectory prediction module, and the updated tracking trajectory is output. All predicted targets are retained until the next frame image is input. The trajectory prediction module can predict the motion state of each unmatched observed target to obtain the predicted target trajectory information, and send the predicted target trajectory information to the aforementioned matching module.

[0126] S300 outputs the target identity information in the input image.

[0127] In specific implementation, the matching module may include a DIoU matrix generation submodule, an optimal matching filtering submodule, and a tracking trajectory output submodule.

[0128] The DIoU matrix generation submodule is connected to the target detection module. It is used to obtain information on all observed targets after removing redundant targets, calculate the distance intersection-union ratio (DIoU) between all observed targets and all current predicted targets, and generate a DIoU matrix.

[0129] The optimal matching filtering submodule is used to sort and match according to the aforementioned DIoU matrix to filter out the target information that is successfully matched and unmatched, update the prediction status for the predicted targets that are successfully matched, decay the confidence of the predicted targets that are unmatched, and send all the observation target information that is not matched to the trajectory prediction module.

[0130] The tracking trajectory output submodule is used to output the updated tracking trajectory and retain all predicted targets until the next frame image input.

[0131] The trajectory prediction module includes a target initialization submodule, a tracking trajectory pool submodule, and a target trajectory prediction module.

[0132] The target initialization submodule is connected to the aforementioned optimal matching and filtering submodule. It is used to obtain information on all unmatched observation targets, establish an initial motion model including acceleration for each unmatched observation target, and initialize the parameters of the motion model.

[0133] The tracking trajectory pool submodule is used to update the tracking trajectory pool according to the motion model constructed by the target initialization submodule, and to update the tracking trajectory pool after obtaining the tracking trajectory update information fed back by the aforementioned tracking trajectory output submodule.

[0134] The target trajectory prediction module is used to establish motion equations for the observed target and predict the target state at the next moment to obtain the predicted target trajectory information, and to send the predicted target trajectory information to the aforementioned tracking trajectory output submodule to update the tracking trajectory of the corresponding target.

[0135] Other technical features are described in the preceding embodiments and will not be repeated here.

[0136] In the above description, the disclosure of this invention is not intended to limit itself to these aspects. Rather, within the scope of the objectives of this disclosure, components can be selectively and operationally combined in any number. Furthermore, terms such as “comprising,” “encompassing,” and “having” should be interpreted by default as inclusive or open-ended, rather than exclusive or closed, unless explicitly defined as such. All technical, scientific, or other terms are to be understood by those skilled in the art, unless defined as such. Public terms found in dictionaries should not be interpreted in the context of the relevant technical documents in an overly idealistic or impractical manner, unless explicitly defined as such in this disclosure. Any modifications or alterations made by those skilled in the art based on the foregoing disclosure are within the scope of the claims.

Claims

1. A multi-target tracking system based on distance intersection-union-ratio ranking matching, characterized in that: It includes a target detection module, a matching module, and a trajectory prediction module, wherein the matching module is connected to the target detection module and the trajectory prediction module; The target detection module is used to perform target detection on the input single-frame image to obtain the bounding box information of the observed targets in the single-frame image and remove redundant targets; The matching module includes a DIoU matrix generation submodule, an optimal match filtering submodule, and a tracking trajectory output submodule. The DIoU matrix generation submodule is connected to the target detection module and is used to acquire all observed target information after removing redundant targets, calculate the distance intersection-union ratio (DIoU) of all observed targets with all current predicted targets, and generate a DIoU matrix. The optimal match filtering submodule is used to sort and match according to the aforementioned DIoU matrix to filter out successfully matched and unmatched target information, update the prediction state for successfully matched predicted targets, decay the confidence of unmatched predicted targets, and send all unmatched observed target information to the trajectory prediction module. The tracking trajectory output submodule is used to output the updated tracking trajectory and retain all predicted targets until the next frame image input. The trajectory prediction module is used to predict the motion state of each unmatched observation target to obtain the predicted target trajectory information, and send the predicted target trajectory information to the aforementioned tracking trajectory output submodule. The trajectory prediction module includes a target initialization submodule, a tracking trajectory pool submodule, and a target trajectory prediction module. The target initialization submodule is connected to the aforementioned optimal matching and filtering submodule. It is used to obtain information on all unmatched observation targets, establish an initial motion model for each unmatched observation target, and initialize the parameters of the motion model. The tracking trajectory pool submodule is used to update the tracking trajectory pool based on the constructed motion model, and to update the tracking trajectory pool after obtaining the tracking trajectory update information fed back by the tracking trajectory output submodule. The target trajectory prediction module is used to establish the motion equation for the observed target and predict the target state at the next moment, obtain the predicted target trajectory information, and send the predicted target trajectory information to the tracking trajectory output submodule to update the tracking trajectory of the corresponding target.

2. The multi-target tracking system according to claim 1, characterized in that: The target detection module includes a convolutional neural network target detection model with fixed parameters; The target detection model is configured to: receive single-frame image data input, identify the targets in the input image data, output the bounding box coordinates and size information (x, y, w, h) of the observed targets, and then perform non-maximum suppression processing to remove redundant targets; where x, y, w, and h represent the x-axis coordinate value, y-axis coordinate value, bounding box width, and bounding box height of the bounding box center, respectively.

3. The multi-target tracking system according to claim 1, characterized in that, The DIoU matrix generation submodule is configured to perform the following processing to generate the DIoU matrix: Obtain all observed targets A after removing redundant targets. M×4 Information, and all current forecast targets B N×4 Information, where M is the total number of observed targets, N is the total number of predicted targets, and M and N are integers greater than 1; For A M×4 Each observation target A in i Calculate the observed target A i With B N×4 Each predicted target B in j The DIoU value between D ij Thus, all observed targets A are obtained. M×4 With all predicted targets B N×4 DIoU matrix D M×N ,in, In the formula, i represents A M×4 The A-th observation target in the dataset, i = 1, 2, ..., M; j represents B N×4 The j-th prediction target in the dataset, where j = 1, 2, ..., N; d represents the observed target A i With predicted target B j The length of the line connecting the centroids of the two points; c represents the observed target A i With predicted target B j The length of the diagonal of the circumscribed rectangle between them; D M×N The matrix contains M×N elements.

4. The multi-target tracking system according to claim 3, characterized in that, The optimal matching filtering submodule is configured to perform the following processing for DIoU sorting matching: Step 1: Obtain the DIoU matrix D M×N All elements D ij Press D ij The values ​​are sorted from largest to smallest to obtain set H; Step 2: Take the first element D from set H. m，n Delete the DIoU matrix D M×N The corresponding element D m，n Rows and columns D m D n At the same time, delete the corresponding element D from set H. m，n This yields the updated set H; Step 3: Repeat step 2 until the DIoU matrix D is reached. M×N There are no elements in the set H; at this time, the index rows and columns i,j of all elements taken out of the set H are the indices of the successfully matched observation target and prediction target pairs, and the index rows and columns i,j of the remaining elements in the set H are the indices of the unmatched observation targets and prediction targets.

5. The multi-target tracking system according to claim 1, characterized in that: For all successfully matched predicted targets, update the Kalman gain K of the predicted target using the corresponding observed target value and update the predicted state, as follows: In the formula, k is the state frame, F and H are the state transition matrices, and F T H T These are the transposes of matrices F and H, respectively; P, Q, and R are the covariance matrices; and K is the Kalman gain. k Let X be the observed target value at frame k. k Let K be the predicted value at frame k. k P represents the Kalman gain at frame k. k Let P be the covariance matrix at frame k. k-1 Let be the covariance matrix at frame k-1. For X k The estimated value, For P k The estimated value; For all unsuccessfully matched predicted targets, the confidence level of the predicted target is reduced based on a preset confidence decay algorithm, which is set by the system or the user.

6. The multi-target tracking system according to claim 5, characterized in that: The motion model is a motion model that includes acceleration, and the motion state of each unmatched observed target is defined as follows: X=[x y s r v x v y v s a x a y ] T In the formula, x, y, s, and r represent the current x-axis coordinates, y-axis coordinates, bounding box area, and bounding box aspect ratio of the observed target, respectively. x v y v s These represent the current x-axis coordinates, y-axis coordinates, and the rate of change of the bounding box area of ​​the observed target, respectively. x a y These represent the accelerations of the observed target along the x-axis and y-axis, respectively. The target initialization submodule is configured to initialize the covariance matrices P, Q, and R in the motion state equations of the motion model, as well as the state transition matrices F and H. All of the above parameters are static parameters and are set by the system or the user.

7. The multi-target tracking system according to claim 6, characterized in that, The target trajectory prediction module is configured as follows: For each unmatched observed target, the target state at the next moment is predicted according to the equation of motion, as follows: X k+1 =FX k +W k Z k =HX k +V k Where k is the state frame, X k X represents the system's predicted motion state at frame k. k+1 Z represents the system's predicted motion state at frame k+1. k This represents the actual observed state at frame k, where F and H are the state transition matrices, and W is the actual state observed at frame k. k V k The approximate Gaussian noise at frame k is... For X k The estimated value, For X k-1 The estimated value.

8. The multi-target tracking system according to claim 1, characterized in that: The matching module also includes a matching deduplication module. In this case, the optimal matching filtering submodule can send the matching results of the observed target and the predicted target to the matching deduplication module. The matching and deduplication module is used to receive the matching results sent by the optimal matching and filtering submodule, perform target deduplication processing according to the matching results, and feed back the deduplicated observation target and prediction target information to the DIoU matrix generation submodule. The DIoU matrix generation submodule can update the DIoU matrix based on the deduplication information provided in the feedback.

9. A multi-target tracking method based on distance intersection-union-ratio ranking matching, characterized in that... The following steps are required: Receive input image; For each image frame of the input image, the following steps are performed: The target detection module performs target detection on the single frame image to obtain the bounding box information of the observed targets in that single frame image, and removes redundant targets before sending it to the matching module; the matching module obtains all observed target information after removing redundant targets, calculates the distance intersection-union ratio (DIoU) between all observed targets and all currently predicted targets, generates a DIoU matrix, sorts and matches according to the aforementioned DIoU matrix to filter out successfully matched and unmatched target information, updates the prediction state for successfully matched predicted targets, decays the confidence level for unmatched predicted targets, sends all unmatched observed target information to the trajectory prediction module, outputs the updated tracking trajectory, and retains all predicted targets until the next frame image input; the trajectory prediction module can predict the motion state of each unmatched observed target to obtain the predicted target trajectory information, and sends the predicted target trajectory information to the aforementioned matching module; Output the target's identity information from the input image; The trajectory prediction module includes a target initialization submodule, a tracking trajectory pool submodule, and a target trajectory prediction module. The target initialization submodule is connected to the aforementioned optimal matching and filtering submodule. It is used to obtain information on all unmatched observation targets, establish an initial motion model for each unmatched observation target, and initialize the parameters of the motion model. The tracking trajectory pool submodule is used to update the tracking trajectory pool based on the constructed motion model, and to update the tracking trajectory pool after obtaining the tracking trajectory update information fed back by the tracking trajectory output submodule. The target trajectory prediction module is used to establish the motion equation for the observed target and predict the target state at the next moment, obtain the predicted target trajectory information, and send the predicted target trajectory information to the tracking trajectory output submodule to update the tracking trajectory of the corresponding target.

Images