Object target tracking method, device and electronic equipment

Abstract

This application provides a method, apparatus, and electronic device for tracking objects. The method includes: receiving target object intelligence information sent by a tracker in real time; determining whether to continue tracking the target object based on the target object intelligence information, and determining to stop tracking the target object at least if the number of loss waiting times is greater than or equal to a predetermined number. By incorporating multiple location information of the target, the area and aspect ratio of the target's bounding box into the location prediction, the accuracy of prediction is improved during target tracking. Determining to stop tracking the target object when the number of loss waiting times is greater than or equal to a predetermined number improves tracking efficiency, thereby solving the problem of low target tracking efficiency in existing solutions.

Description

Technical Field

[0001] This application relates to the field of object tracking technology, and more specifically, to a method, apparatus, computer-readable storage medium, and electronic device for tracking object targets. Background Technology

[0002] Existing technical solutions are mostly designed for urban highway scenarios, and the objects they consider are various types of object tracking. In particular, for dynamic object tracking, the algorithms and models used are very complex, with a large amount of computation and low real-time performance. However, the actual scenarios in mining areas do not have as many complex targets as highway scenarios, and the corresponding radar perception results will have a lot of redundant data.

[0003] Target tracking includes single-target tracking and multi-target tracking. Single-target tracking can handle problems such as deformation and occlusion by modeling the appearance or motion of the target; while multi-target tracking is more complex. In addition to the problems encountered in single-target tracking, it is also necessary to consider the correlation and matching between targets. Moreover, multi-target tracking tasks often encounter problems such as frequent target occlusion, unknown trajectory start and end times, targets that are too small, targets that are similar in appearance, and low frame rate of interaction between targets.

[0004] Currently, due to the harsh environmental conditions in mining areas, autonomous driving in mining scenarios has become an urgent problem to solve. The mainstream approach in perception is the use of LiDAR and millimeter-wave radar. Radar perception is mainly used for multi-target tracking, which is divided into three steps: target tracking, motion prediction, and data association. To address the requirements of autonomous driving for transport vehicles in typical mining environments, the mining perception system should meet the following requirements: 1) It should be able to perform multi-target tracking over long distances, while also being able to handle frequent occlusion, unstable target clustering detection, and unstable target association; 2) It should be able to perform stable multi-target tracking under adverse weather conditions; 3) Since the number and types of targets in mining scenarios are usually small and mostly stationary obstacles, it is necessary to ensure recognition accuracy while using high-efficiency algorithms and saving computational resources. Summary of the Invention

[0005] The main objective of this application is to provide a method, apparatus, computer-readable storage medium, and electronic device for tracking objects, so as to at least solve the problem of low target tracking efficiency in existing solutions.

[0006] To achieve the above objectives, according to one aspect of this application, a method for tracking an object target is provided. The method includes: receiving target object intelligence information sent by a tracker in real time, the target object intelligence information including a target object identifier, a target object motion model, and a number of loss-wait attempts, wherein the number of loss-wait attempts characterizes the number of times the target object has not been detected, the target object motion model is obtained using multiple sets of historical target object motion data, each set of historical target object motion data including: multiple location information of the target object, the area of ​​the bounding box of the target object, and its aspect ratio, acquired within a historical time period; and determining whether to continue tracking the target object based on the target object intelligence information, wherein if the number of loss-wait attempts is greater than or equal to a predetermined number, it is determined that tracking of the target object should cease.

[0007] Optionally, in determining whether to continue tracking the target object based on the target object intelligence information, the method further includes: using constraint equations to constrain the target object motion model to obtain a constrained target object motion model, wherein the constraint equations characterize the relationship between the prior and posterior states of the target object, and the target object motion model characterizes the relationship between the position information of the target object and the area and aspect ratio of the bounding box of the target object; using the constrained target object motion model, predicting the target object's position information matrix based on the target object's position information at the previous moment, the area and aspect ratio of the bounding box of the target object, and obtaining the predicted target object position information matrix. The system comprises multiple predicted location information of the target object; based on the location information of the target object at the previous time step and the location information of all predicted target objects in the predicted target object location information matrix, multiple relationship coefficients are determined, which are used to characterize the overlap between the bounding box of the target object at the previous time step and the bounding box of the target object in the predicted time step; based on all the relationship coefficients, the location information of the target object at the previous time step is matched with the location information of all predicted target objects in the predicted target object location information matrix to determine the predicted location information of the target object, which is one of the predicted location information of the target object.

[0008] Optionally, the constraint equations include:

[0009] in, Let be the prior state estimate of the target object at time t. The prior state is used to characterize the state predicted based on all the location information of the target object. The posterior state estimate of the target object at time t-1 is given. The posterior state characterizes the state predicted based on all the location information of the target object, the area of ​​the bounding box of the target object, and the aspect ratio. t B represents all the location information of the target object, the area of ​​the bounding box of the target object, and the aspect ratio. t The matrix transformed into states, where F is the state transition matrix;

[0010] P t - =FP t-1 F T +Q; where P t - Let P be the covariance of the estimated prior state of the target object at time t. t-1 F is the covariance of the estimated prior state of the target object at time t-1. T Let F be the transpose of F, and Q be the process error value;

[0011] Among them, K t Here, H represents the Kalman coefficients, and H is the observation transformation matrix from the initial state to the observed state of the target object. This observation transformation matrix characterizes the relationship between the initial state and the observations of the target object. T Let H be the transpose of H, and R be the observation error value;

[0012] in, Let z be the posterior state estimate of the target object at time t. t These are the observed values;

[0013] P t = (1-K) t H)P t -; where P t Let be the covariance of the estimated posterior state of the target object at time t.

[0014] Optionally, based on all the relationship coefficients, matching the position information of the target object at the previous time step with the position information of all predicted target objects in the predicted target object position information matrix to determine the predicted target object position information includes: constructing an overlap degree mapping relationship, which is used to characterize the mapping relationship between the relationship coefficients and the predicted target object position information; determining a target relationship coefficient based on all the relationship coefficients, where the target relationship coefficient is the maximum value among all the relationship coefficients; and determining the predicted target object position information based on the target relationship coefficient, the overlap degree mapping relationship, and the position information of all predicted target objects in the predicted target object position information matrix.

[0015] Optionally, the motion model of the target object is:

[0016]

[0017] Where x represents the state of the target object, u and v are the horizontal and vertical positions of the center point of the target object, respectively, and s and r are the area and aspect ratio of the bounding box of the target object, respectively. and These are the horizontal and vertical velocities of the center point of the target object, respectively. The area of ​​the bounding box changes at a certain rate.

[0018] Optionally, after determining that the target object will no longer be tracked if the number of lost waits is greater than or equal to a predetermined number, the method further includes deleting the target object's identity identifier from the tracking identity repository.

[0019] Optionally, the position information of the target object includes the horizontal and vertical positions of the center point of the target object.

[0020] According to another aspect of this application, a target tracking device is provided, comprising a receiving unit and a determining unit; the receiving unit is configured to receive target object intelligence information sent by a tracker in real time, the target object intelligence information including target identification, target motion model, and loss waiting count, wherein the loss waiting count is used to characterize the number of times the target has not been detected, the target motion model is obtained using multiple sets of historical target motion data, each set of historical target motion data including: multiple location information of the target, the area of ​​the target's bounding box, and the aspect ratio of the target acquired within a historical time period; the determining unit is configured to determine whether to continue tracking the target object based on the target object intelligence information, and at least if the loss waiting count is greater than or equal to a predetermined number, determine whether to stop tracking the target object.

[0021] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform any of the object tracking methods described above.

[0022] According to another aspect of this application, an electronic device is provided, the electronic device including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including a method for performing any of the object target tracking methods described above.

[0023] By applying the technical solution of this application, multiple location information of the target, the area and aspect ratio of the target's bounding box are incorporated into the location prediction considerations, which improves the accuracy of prediction during target tracking. When the number of lost waiting times is greater than or equal to a predetermined number, it is determined that the target object will no longer be tracked, thus improving tracking efficiency and solving the problem of low target tracking efficiency in existing solutions. Attached Figure Description

[0024] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0025] Figure 1 A hardware structure block diagram of a mobile terminal for performing an object tracking method according to an embodiment of this application is shown;

[0026] Figure 2 A flowchart illustrating an object tracking method according to an embodiment of this application is shown.

[0027] Figure 3 A structural block diagram of an object tracking device according to an embodiment of this application is shown. Detailed Implementation

[0028] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0032] Radar sensing: mainly refers to lidar sensing and millimeter-wave radar sensing, specifically outputting information such as the target's position, speed, and size relative to the radar.

[0033] Deep learning refers to methods that learn the inherent patterns and hierarchical representations of sample data, enabling machines to learn and analyze like humans, and to identify and classify data such as text, images, and sound.

[0034] Interactive multi-model: This method uses multiple motion models to predict observed targets.

[0035] As described in the background section, existing technical solutions are mostly designed for urban highway scenarios, considering the tracking of various types of objects, especially dynamic object tracking. The algorithms and models used are very complex, computationally intensive, and have low real-time performance. However, actual mining areas do not have as many complex targets as highway scenarios, resulting in a lot of redundant data in the radar sensing results. To address the low target tracking efficiency of existing solutions, embodiments of this application provide a method, apparatus, computer-readable storage medium, and electronic device for tracking objects.

[0036] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0037] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for an object tracking method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0038] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the device information display method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0039] This embodiment provides a method for tracking object targets that runs on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0040] Figure 2 This is a flowchart illustrating an object tracking method according to an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0041] Step S201: Receive target object intelligence information sent by the tracker in real time. The target object intelligence information includes the target object identification, the target object motion model, and the number of loss waits. The number of loss waits is used to characterize the number of times the target object has not been detected. The target object motion model is obtained using multiple sets of historical target object motion data. Each set of historical target object motion data includes: multiple location information of the target object, the area and aspect ratio of the bounding box of the target object, acquired within the historical time period.

[0042] The location information of the target object includes the horizontal and vertical positions of the center point of the target object;

[0043] The motion model of the target object mentioned above is as follows:

[0044]

[0045] Where x represents the state of the target object, u and v represent the horizontal and vertical positions of the center point of the target object, respectively, and s and r represent the area and aspect ratio of the bounding box of the target object, respectively. and These are the horizontal and vertical velocities of the center point of the aforementioned target object, respectively. The rate of change of the area of ​​the bounding box mentioned above;

[0046] Before step S201, the visual information location corresponding to the target sensed by the radar is first used as input. A deep learning model based on an hourglass model is used to extract deep and shallow information from the image. To prevent feature loss, the original input results are retained in each downsampling step and passed to the corresponding sampling stage. The final output is:

[0047]

[0048] The parameter catgory represents the specific type of the extracted image. In actual mining operations, it can be different types such as mining trucks, loaders, excavators, power towers, and retaining walls. width and length represent the width and length of the corresponding target's precise bounding box.

[0049] In multi-target tracking, the first step is to identify the target and determine its precise location using relevant algorithms. However, in target recognition and identification tasks for urban highways, the sheer number and variety of targets to be detected, along with the simultaneous generation of bounding boxes for targets of different sizes and subsequent category classification training, pose a significant challenge to deep learning-based target identification methods. For example, the YOLO series of target detection networks requires generating feature maps of different sizes to correspond to the identification of targets of varying sizes, which increases the complexity of the network model and the computational load. A solution to this problem is to first determine the approximate location and size of the target using a low-cost radar sensing algorithm, and then use deep learning for target recognition and identification. Since the task of deep learning is simplified from determining and recognizing target bounding boxes to target identification, the corresponding model size is significantly reduced.

[0050] Step S202: Based on the above target object intelligence information, determine whether to continue tracking the target object. At least if the number of lost waiting times is greater than or equal to a predetermined number, determine whether to stop tracking the target object.

[0051] If the number of lost waiting times is less than the predetermined number, it is determined to continue tracking the target object.

[0052] In step S202, the method further includes the following steps (i.e., in determining whether to continue tracking the target object based on the aforementioned target object intelligence information, the method further includes the following steps):

[0053] The above target object motion model is constrained by the constraint equation to obtain the constrained target object motion model. The constraint equation is used to characterize the relationship between the prior state and the posterior state of the target object, and the target object motion model is used to characterize the relationship between the position information of the target object and the area and aspect ratio of the bounding box of the target object.

[0054] In this process, when a target is detected for the first time in the video, a motion model is built for each target. Then, Kalman filtering is used to predict the state of each target in the next frame. When a detected target in the next frame is associated with the predicted target, the bounding box of the detected target is used to update the model state. If no target is detected in the next frame, only state prediction is performed, without state updating. Since the state prediction and state updating of the target motion model involve Kalman filtering, Kalman filtering can be divided into a state prediction equation and a state update equation. The state prediction equation predicts the prior estimates of the state variables and the prior estimates of the error covariance based on the state estimates from the previous time step. The state update equation is responsible for combining the prior estimates with the new detection (observation) results to construct an improved posterior estimate.

[0055] The above constraint equations include:

[0056] First constraint formula: in, Let be the prior state estimate of the target object at time t, which is the result of state prediction. The prior state is used to characterize the state predicted based on all the aforementioned location information of the target object. Let u be the posterior state estimate of the target object at time t-1, i.e., the optimal estimate. The posterior state is used to characterize the state predicted based on all the aforementioned positional information of the target object, the area of ​​the bounding box of the target object, and the aspect ratio. t Given all the aforementioned location information of the target object, the area of ​​the bounding box of the target object, and the aspect ratio, B represents u.t The matrix transformed into states, F is the state transition matrix, which is a conjectured model of the target state transition;

[0057] Second constraint formula: P t -=FP t-1 F T +Q; where P t - Let P be the covariance of the estimated prior state of the target object at time t, representing the uncertainty of the prior state. t-1 F is the covariance of the estimated prior state of the target object at time t-1. T Let F be the transpose of F, and Q be the process error value, representing the error between the process state transition matrix and the actual process.

[0058] The formula for the first pair of predicted correction models: Among them, K t K represents the Kalman coefficient. t The larger the value, the more the posterior state estimate is biased towards the observation. H is the observation transformation matrix from the initial state to the observed state of the target object. This observation transformation matrix is ​​used to characterize the relationship between the initial state and the observation of the target object, establishing the relationship between the state and the observation. T Let H be the transpose of H, and R be the observation error value, representing the error between the true value and the observed value;

[0059] The second set of formulas for the corrected prediction model: in, Let z be the posterior state estimate of the target object at time t. t For the observed values, It is the residual between the observed value and the predicted value;

[0060] The third pair of prediction correction model formulas: P t = (1-K) t H)P t -; where P t Let be the covariance of the estimated posterior state of the aforementioned target object at time t.

[0061] in, P t = (1-K) t H)P t - is the state update equation for the Kalman filter. and P t -=FP t-1 F T +Q is the state prediction equation for the Kalman filter.

[0062] The first pair of prediction correction model formulas, the first pair of prediction correction model formulas, and the first pair of prediction correction model formulas together constitute the prediction correction model. The prediction correction model contains... The input to the prediction part becomes The prediction part consists of the first constraint formula and the second constraint formula. The prediction part uses the actual observation values ​​to correct the predicted values.

[0063] For dynamic target motion models, there are no additional input variables, u t =0, the observations are only u, v, s and r, and H is a 4×7 matrix:

[0064]

[0065]

[0066]

[0067]

[0068]

[0069] F is the state transition matrix, and H is the matrix for inputs with only four variables, where the observed values ​​correspond to Z. t =[uvsr] is the state transition matrix used in the 1×4 target motion model matrix, where x is the state of the target object, u t and v t These represent the horizontal and vertical positions of the center point of the target object at time t, respectively, and s t Let u be the area of ​​the bounding box of the target object at time t. t-1 and v t-1 These represent the horizontal and vertical positions of the center point of the target object at time t-1, respectively. t-1 Let be the area of ​​the bounding box of the target object at time t-1. and These represent the horizontal and vertical velocities of the center point of the target object at time t-1, respectively. Let Δt be the rate of change of the area of ​​the bounding box at time t-1, and let Δt be the time of change. Since the target motion between frames in the video is regarded as an identity linear model, the changes of u, v, and s between two frames can be regarded as uniform motion.

[0070] For unobservable pixel orientation and area change rates, the covariance P is set in the initial configuration. t High uncertainty is introduced into the velocity components of the model. At the same time, a small process error Q is set, and the observation error R is ignored.

[0071] Using the target object motion model with the above constraints, prediction is made based on the target object's position information at the previous moment, the area of ​​the target object's bounding box, and the aspect ratio, resulting in a predicted target object position information matrix. The predicted target object position information matrix is ​​composed of multiple predicted target object position information.

[0072] Based on the position information of the target object at the previous time step and all the predicted position information of the target object in the predicted position information matrix, multiple relationship coefficients are determined. These relationship coefficients are used to characterize the degree of overlap between the bounding box of the target object at the previous time step and the bounding box of the target object in the predicted time step.

[0073] This approach allows for better data correlation between predicted and observed values ​​of the target object. Assuming there are four targets to track, and the previously predicted target positions were a, b, c, and d, while four targets have been detected at positions A, B, C, and D, the relationship coefficient between the predicted position of the i-th target and the detected position of the j-th target is C. ij The overlap of target boxes can usually be used as the relation coefficient C. ij The next task is to consider how to make the assignment correspondences so that C ij The maximum value is shown in the following equation:

[0074]

[0075] Because of C ij Representing the degree of overlap, then C ij The larger the value, the greater the relevance. Therefore, we take its reciprocal and subtract 0 to ensure that each row and column contains 0. When the number of independent 0 elements is the same as the matrix order, we can output the following results: the next frame of a corresponds to D, the next frame of b corresponds to B, the next frame of c corresponds to C, and the next frame of d corresponds to D.

[0076] When it is found that only 3 independent 0 elements can be found, and the order of the coefficient matrix is ​​4, all 0 elements need to be removed using the minimum of 3 straight lines in the trial allocation test, and the rows and columns are marked according to the rules; the coefficient matrix is ​​updated, and it is found that the smallest element not covered by the straight lines is 1. Therefore, according to the rules, the elements in the original coefficient matrix are added by 1 or subtracted by 1, and 0 elements are added to obtain a new coefficient matrix; the trial allocation operation is performed again on the new coefficient matrix, and it is found that 4 independent 0 elements with the same order as the coefficient matrix can be found, and the algorithm ends; finally, the optimal solution is given.

[0077] Based on all the above relationship coefficients, the position information of the target object at the previous time step is matched with the position information of all the predicted target objects in the predicted target object position information matrix to determine the predicted target object position information. The predicted target object position information is one of all the predicted target object position information.

[0078] Specifically, based on all the aforementioned relationship coefficients, the position information of the target object at the previous time step is matched with the position information of all predicted target objects in the predicted target object position information matrix to determine the predicted target object position information, including:

[0079] Construct an overlap mapping relationship, which is used to characterize the mapping relationship between the above-mentioned relationship coefficients and the above-mentioned predicted location information of the above-mentioned target object;

[0080] Specifically, the overlap mapping relationship facilitates the subsequent lookup of the location information of the predicted target objects by using the target relationship coefficient;

[0081] Based on all the above relationship coefficients, a target relationship coefficient is determined, which is the maximum value among all the above relationship coefficients;

[0082] Specifically, for example, if the relationship coefficients are 1, 2, and 3, then the predicted location information of the target object corresponding to 3 is the predicted location information of the target object.

[0083] Based on the target relationship coefficients, the overlap mapping relationship, and all the predicted target object location information in the predicted target object location information matrix, the predicted target object location information is determined.

[0084] For example, if the relationship coefficients are 1, 2, and 3, since 3 is the largest among the three relationship coefficients, then the predicted location information of the target object corresponding to 3 is the predicted location information of the target object.

[0085] In one embodiment of this application, after determining that the target object will no longer be tracked when the number of lost waiting times is greater than or equal to a predetermined number, the method further includes: deleting the target object's identity identifier from the tracking identity repository.

[0086] For example, if the relationship coefficients are 1, 2, and 3, then the predicted location information of the target object corresponding to 3 is the predicted location information of the target object.

[0087] When trajectory tracking begins, if a target's pose is detected but no tracker can associate it, a corresponding tracker needs to be initialized for each detected target. During tracker initialization, an identity flag is assigned in a random order. Once established, the identity flag remains unchanged unless the tracker is destroyed. The target motion model is set based on the initial detected bounding box position and geometry, but the velocity is set to zero. Since velocity is unobservable, a large value is used for initialization of the velocity component covariance to reflect this uncertainty. Furthermore, a 5-frame trial period is set for each tracker. If the tracker can associate with new detection results within these 5 frames, it is considered a true tracker; otherwise, it is immediately destroyed. This accumulates sufficient evidence to prevent false alarms. However, if a target is not detected in a new frame due to occlusion or other reasons, the tracker corresponding to that target will not associate with new detections. In this case, the tracker's loss waiting time is incremented, indicating a tracking loss. Therefore, through rule setting, when a tracker's loss waiting time (i.e., the number of loss waiting times) is greater than zero, the tracker will not make new predictions in the new frame and will remain at the position where the tracking was lost. Once the occlusion disappears and the lost target is detected again, a new connection is established with the tracker, the loss waiting time is reset to zero, and tracking continues. This demonstrates that this tracking method and rules possess a certain degree of error correction capability, which can improve the stability of trajectory tracking.

[0088] When the loss waiting time exceeds a certain duration, tracking is considered a failure, and the tracker is destroyed. However, during dynamic target observation, certain prior information, such as the number of targets, can be obtained. Therefore, the trajectory tracking rules stipulate that when a tracker is destroyed, the released identity token can be used to establish a new tracker for re-tracking. For example, in a three-target trajectory tracking task, problems such as missed detection due to prolonged occlusion or erroneous detection due to poor detection model performance can lead to excessively long loss waiting times, causing tracking failure for a certain target. In this case, the entire tracking system can calculate the tracking failure of that target using the other two trackers. When a new target is detected and cannot be associated with an existing tracker, a new tracker with the identity token of that target will be generated, instead of creating a new tracker. Under this rule, even if one tracker fails, the failed target can be re-tracked using prior conditions and other well-tracking trackers.

[0089] Multi-target tracking is the process of processing received measurements to maintain an estimate of the current state of multiple targets. Target states can include characteristic parameters such as motion states. In autonomous driving scenarios, targets are categorized as dynamic and static, with dynamic targets exhibiting different motion characteristics depending on their type. After determining the target type and characteristics, different levels of computational resources are allocated based on the type. Static obstacles are handled using static obstacle avoidance methods without establishing kinematic models. Dynamic obstacles are kinematically modeled, and Kalman filters and the Hungarian algorithm are used for trajectory management. The core of the interactive multi-model-based multi-target tracking algorithm mainly includes target identification based on deep learning and radar algorithms, a Kalman filter tracker, and a Hungarian algorithm trajectory management algorithm.

[0090] First, radar is used to extract the approximate location information of the target. Then, an optimized deep learning algorithm is used to calculate the target type and its precise location, thereby determining the type of obstacle. When the target obstacle is a dynamic obstacle, a corresponding kinematic model is established. Finally, Kalman filtering and the Hungarian algorithm are used to manage the target trajectory. If the target is a static obstacle, computation can be saved by not establishing a kinematic model, and a static obstacle avoidance method can be used to solve the problem. The key aspect of this application is tracking dynamic obstacles.

[0091] By incorporating multiple location information of the target, the area and aspect ratio of the target's bounding box into the location prediction, the accuracy of prediction is improved during target tracking. If the number of lost waiting times is greater than or equal to a predetermined number, it is determined that the target object will no longer be tracked, thus improving tracking efficiency and solving the problem of low target tracking efficiency in existing solutions.

[0092] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0093] This application also provides an object tracking device. It should be noted that the object tracking device of this application can be used to execute the object tracking method provided in this application. This device is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0094] The following describes the object tracking device provided in the embodiments of this application.

[0095] Figure 3 This is a structural block diagram of an object tracking device according to an embodiment of this application. Figure 3 As shown, the device includes a receiving unit 31 and a determining unit 32. The receiving unit 31 is used to receive target object intelligence information sent by the tracker in real time. The target object intelligence information includes target identification, target motion model, and number of loss waiting times. The number of loss waiting times is used to characterize the number of times the target has not been detected. The target motion model is obtained using multiple sets of historical target motion data. Each set of historical target motion data includes multiple location information of the target, the area of ​​the target's bounding box, and the aspect ratio of the target, acquired within a historical time period. The determining unit 32 is used to determine whether to continue tracking the target object based on the target object intelligence information. At least if the number of loss waiting times is greater than or equal to a predetermined number, it determines that it will no longer continue tracking the target object.

[0096] In the aforementioned device, multiple location information of the target, the area of ​​the target's bounding box, and the aspect ratio are incorporated into the location prediction considerations, thereby improving the accuracy of prediction during target tracking. If the number of lost waiting times is greater than or equal to a predetermined number, it is determined that the target object will no longer be tracked, thus improving tracking efficiency and solving the problem of low target tracking efficiency in existing solutions.

[0097] In one embodiment of this application, the determining unit includes a first processing module, a second processing module, a third processing module, and a fourth processing module. In the process of determining whether to continue tracking the target object based on the target object intelligence information, the first processing module is used to constrain the target object motion model using constraint equations to obtain a constrained target object motion model. The constraint equations characterize the relationship between the prior and posterior states of the target object, and the target object motion model characterizes the relationship between the target object's position information and the area and aspect ratio of its bounding box. The second processing module uses the constrained target object motion model to predict the target object's position based on its position information at the previous time step, the area of ​​its bounding box, and its aspect ratio, obtaining a predicted target object position information matrix. The measured target object position information matrix consists of multiple predicted target object position information. The third processing module is used to determine multiple relationship coefficients based on the target object's position information at the previous time step and all predicted target object position information in the predicted target object position information matrix. These relationship coefficients characterize the overlap between the target object's bounding box at the previous time step and the target object's bounding box in the predicted time step. The fourth processing module is used to match the target object's position information at the previous time step with all predicted target object position information in the predicted target object position information matrix based on all the relationship coefficients, and determine the predicted target object position information. The predicted target object position information is one of all predicted target object position information.

[0098] In one embodiment of this application, the above-mentioned constraint equations include:

[0099] in, Let be the prior state estimate of the target object at time t. The prior state is used to characterize the state predicted based on all the aforementioned location information of the target object. The posterior state estimate of the target object at time t-1 is given. The posterior state characterizes the state predicted based on all the aforementioned location information of the target object, the area of ​​the bounding box of the target object, and the aspect ratio. t Given all the aforementioned location information of the target object, the area of ​​the bounding box of the target object, and the aspect ratio, B represents u. t The matrix transformed into states, where F is the state transition matrix;

[0100] P t -=FPt-1 F T +Q; where P t - Let P be the covariance of the estimated prior state of the target object at time t. t-1 F is the covariance of the estimated prior state of the target object at time t-1. T Let F be the transpose of F, and Q be the process error value;

[0101] Among them, K t Here, H represents the Kalman coefficients, and H is the observation transformation matrix from the initial state to the observed state of the target object. This observation transformation matrix characterizes the relationship between the initial state and the observations of the target object. T Let H be the transpose of H, and R be the observation error value;

[0102] in, Let z be the posterior state estimate of the target object at time t. t These are the observed values;

[0103] P t = (1-K) t H)P t - Among them, P t Let be the covariance of the estimated posterior state of the aforementioned target object at time t.

[0104] In one embodiment of this application, the fourth processing module includes a construction module, a first determining module, and a second determining module. The construction module is used to construct an overlap mapping relationship, which is used to characterize the mapping relationship between the relationship coefficients and the predicted location information of the target object. The first determining module is used to determine a target relationship coefficient based on all the relationship coefficients, where the target relationship coefficient is the maximum value among all the relationship coefficients. The second determining module is used to determine the predicted location information of the target object based on the target relationship coefficient, the overlap mapping relationship, and all the predicted location information of the target object in the predicted location information matrix.

[0105] In one embodiment of this application, the motion model of the target object is as follows:

[0106]

[0107] Where x represents the state of the target object, u and v represent the horizontal and vertical positions of the center point of the target object, respectively, and s and r represent the area and aspect ratio of the bounding box of the target object, respectively. and These are the horizontal and vertical velocities of the center point of the aforementioned target object, respectively. The rate of change of the area of ​​the bounding box mentioned above.

[0108] In one embodiment of this application, the device further includes a processing unit. After determining that the target object will no longer be tracked when the number of lost waiting times is greater than or equal to a predetermined number, the processing unit is used to delete the target object's identity identifier from the tracking identity repository.

[0109] In one embodiment of this application, the location information of the target object includes the horizontal and vertical positions of the center point of the target object.

[0110] The aforementioned object tracking device includes a processor and a memory. The receiving unit and determining unit, among others, are stored as program units in the memory. The processor executes these program units stored in the memory to achieve the corresponding functions. All of the aforementioned modules reside in the same processor; alternatively, the modules may be located in different processors in any combination.

[0111] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured; adjusting kernel parameters can address the low target tracking efficiency of existing solutions.

[0112] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0113] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device containing the computer-readable storage medium to perform a method for tracking the object target.

[0114] This invention provides a processor for running a program, wherein the program executes a method for tracking the object target.

[0115] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps: receiving target object intelligence information sent by a tracker in real time. The target object intelligence information includes a target object identifier, a target object motion model, and a number of loss-wait attempts. The number of loss-wait attempts characterizes the number of times the target object has not been detected. The target object motion model is obtained using multiple sets of historical target object motion data. Each set of historical target object motion data includes, within a historical time period, multiple location information of the target object, the area of ​​the target object's bounding box, and its aspect ratio. Based on the target object intelligence information, it determines whether to continue tracking the target object, and at least if the number of loss-wait attempts is greater than or equal to a predetermined number, it determines that tracking of the target object should cease. The device described herein can be a server, PC, PAD, mobile phone, etc.

[0116] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps: receiving target object intelligence information sent by a tracker in real time, the target object intelligence information including target object identification, target object motion model, and loss waiting count, wherein the loss waiting count is used to characterize the number of times the target object has not been detected, the target object motion model is obtained using multiple sets of historical target object motion data, each of the multiple sets of historical target object motion data including: multiple location information of the target object, the area and aspect ratio of the bounding box of the target object acquired within the historical time period; determining whether to continue tracking the target object based on the target object intelligence information, and determining not to continue tracking the target object at least if the loss waiting count is greater than or equal to a predetermined number.

[0117] This application also provides an electronic device including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for performing any of the above-described object tracking methods.

[0118] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0119] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0121] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0122] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0123] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0124] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0125] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0126] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0127] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0128] 1) The object tracking method of this application incorporates multiple location information of the target, the area and aspect ratio of the bounding box of the target into the location prediction consideration, which improves the accuracy of prediction during the target tracking process. When the number of lost waiting times is greater than or equal to the predetermined number, it is determined that the target object will no longer be tracked, thereby improving the tracking efficiency and solving the problem of low target tracking efficiency in the existing solution.

[0129] 2) The object tracking device of this application incorporates multiple location information of the target, the area and aspect ratio of the bounding box of the target into the location prediction consideration, which improves the accuracy of prediction during the target tracking process. When the number of lost waiting times is greater than or equal to the predetermined number, it is determined that the target object will no longer be tracked, thereby improving the tracking efficiency and solving the problem of low target tracking efficiency in the existing solution.

[0130] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A method for tracking an object target, characterized in that, include: The system receives target object intelligence information sent by the tracker in real time. The target object intelligence information includes the target object's identity, target object motion model, and number of loss waits. The number of loss waits represents the number of times the target object has not been detected. The target object motion model is obtained using multiple sets of historical target object motion data. Each set of historical target object motion data includes: multiple location information of the target object, the area of ​​the target object's bounding box, and the aspect ratio of the bounding box acquired within the historical time period. Based on the target object intelligence information, determine whether to continue tracking the target object; at least if the number of lost waiting times is greater than or equal to a predetermined number, determine whether to stop tracking the target object. In the process of determining whether to continue tracking the target object based on the target object intelligence information, the method further includes: The motion model of the target object is constrained by a constraint equation to obtain a constrained motion model of the target object. The constraint equation is used to characterize the relationship between the prior state and the posterior state of the target object, and the motion model of the target object is used to characterize the relationship between the position information of the target object and the area and aspect ratio of the bounding box of the target object. Using the constrained target object motion model, prediction is made based on the target object's position information at the previous moment, the area of ​​the target object's bounding box, and the aspect ratio, to obtain the predicted target object's position information matrix. The predicted target object's position information matrix is ​​composed of multiple predicted target object position information. Based on the position information of the target object at the previous time step and all the predicted position information of the target object in the predicted position information matrix, multiple relationship coefficients are determined. The relationship coefficients are used to characterize the degree of overlap between the bounding box of the target object at the previous time step and the bounding box of the target object in the predicted time step. Based on all the relationship coefficients, the position information of the target object at the previous time step is matched with the position information of all predicted target objects in the predicted position information matrix of the target object to determine the predicted position information of the target object. The predicted position information of the target object is one of all the predicted position information of the target object. The step of matching the position information of the target object at the previous time step with all the predicted position information of the target object in the predicted position information matrix based on all the relationship coefficients to determine the predicted position information of the target object includes: Construct an overlap mapping relationship, which is used to characterize the mapping relationship between the relationship coefficient and the predicted location information of the target object; Based on all the said relation coefficients, a target relation coefficient is determined, which is the maximum value among all the said relation coefficients; Based on the target relationship coefficient, the overlap mapping relationship, and all the predicted target object position information in the predicted target object position information matrix, the predicted target object position information is determined.

2. The method according to claim 1, characterized in that, The constraint equations include: in, Let be the prior state estimate of the target object at time t. The prior state is used to characterize the state predicted based on all the location information of the target object. The posterior state estimate of the target object at time t-1 is given. The posterior state characterizes the state predicted based on all the location information of the target object, the area of ​​the bounding box of the target object, and the aspect ratio. t B represents all the location information of the target object, the area of ​​the bounding box of the target object, and the aspect ratio. t The matrix transformed into states, where F is the state transition matrix; P t - =FP t-1 F T +Q; where P t - Let P be the covariance of the estimated prior state of the target object at time t. t-1 F is the covariance of the estimated prior state of the target object at time t-1. T Let F be the transpose of F, and Q be the process error value; Among them, K t Here, H represents the Kalman coefficients, and H is the observation transformation matrix from the initial state to the observed state of the target object. This observation transformation matrix characterizes the relationship between the initial state and the observations of the target object. T Let H be the transpose of H, and R be the observation error value; in, Let z be the posterior state estimate of the target object at time t. t These are the observed values; P t = (1-K) t H)P t - Among them, P t Let be the covariance of the estimated posterior state of the target object at time t.

3. The method according to claim 1, characterized in that, The motion model of the target object is as follows: Where x represents the state of the target object, u and v are the horizontal and vertical positions of the center point of the target object, respectively, and s and r are the area and aspect ratio of the bounding box of the target object, respectively. and These are the horizontal and vertical velocities of the center point of the target object, respectively. The rate of change of the area of ​​the bounding box.

4. The method according to claim 1, characterized in that, After determining that tracking the target object will no longer continue if the number of lost waiting times is greater than or equal to a predetermined number, the method further includes: Remove the target object's identity from the tracking identity repository.

5. The method according to any one of claims 1 to 4, characterized in that, The location information of the target object includes the horizontal and vertical positions of the center point of the target object.

6. A tracking device for an object target, characterized in that, include: The receiving unit is used to receive target object intelligence information sent by the tracker in real time. The target object intelligence information includes target identification, target motion model and loss waiting count. The loss waiting count is used to characterize the number of times the target has not been detected. The target motion model is obtained using multiple sets of historical target motion data. Each set of historical target motion data includes: multiple location information of the target, the area of ​​the target's bounding box and the aspect ratio of the target, acquired within the historical time period. The determining unit is configured to determine whether to continue tracking the target object based on the target object intelligence information, and at least if the number of lost waiting times is greater than or equal to a predetermined number, determine that the tracking of the target object should no longer be continued; The determining unit includes: The first processing module is used to constrain the motion model of the target object using constraint equations to obtain a constrained motion model of the target object. The constraint equations are used to characterize the relationship between the prior state and the posterior state of the target object, and the motion model of the target object is used to characterize the relationship between the position information of the target object and the area and aspect ratio of the bounding box of the target object. The second processing module is used to use the constrained target object motion model to predict the target object's position information matrix based on the target object's position information at the previous moment, the area of ​​the target object's bounding box, and the aspect ratio. The predicted target object's position information matrix is ​​composed of multiple predicted target object position information matrices. The third processing module is used to determine multiple relationship coefficients based on the position information of the target object at the previous time step and all the predicted position information of the target object in the predicted position information matrix of the target object. The relationship coefficients are used to characterize the degree of overlap between the bounding box of the target object at the previous time step and the bounding box of the target object in the predicted time step. The fourth processing module is used to match the position information of the target object at the previous time step with all the predicted position information of the target object in the predicted position information matrix of the target object according to all the relationship coefficients, and determine the predicted position information of the target object, wherein the predicted position information of the target object is one of all the predicted position information of the target object; The fourth processing module includes: The construction module is used to construct an overlap mapping relationship, which is used to characterize the mapping relationship between the relationship coefficient and the predicted location information of the target object; The first determining module is used to determine a target relation coefficient based on all the relation coefficients, wherein the target relation coefficient is the maximum value among all the relation coefficients; The second determining module is used to determine the predicted position information of the target object based on the target relationship coefficient, the overlap mapping relationship, and all the predicted position information of the target object in the predicted position information matrix of the target object.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the object tracking method according to any one of claims 1 to 5.

8. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including a method for performing an object target tracking method according to any one of claims 1 to 5.

Images