Target tracking method and device, electronic equipment and storage medium

Abstract

The invention relates to a target tracking method and device, electronic equipment and a storage medium, and is applied to the technical field of automatic driving, and the method comprises the steps: determining the motion state data of a first target at a previous moment and the motion state data of a third target at a current moment; and predicting the motion state data of the first target to obtain the motion state data of the first target at the current moment. And determining an association cost value between the first target and the third target, and constructing an association cost matrix. Determining global optimal matching from the association cost matrix; adding the motion state data of each third target at the current moment into the corresponding track data, determining the first target which is not successfully matched as a lost target, and deleting the lost target which is continuously matched for N times; the current moment is used as the previous moment, the next moment is used as the current moment, the state of the Kalman filter is updated, and the first step is executed until the images around the vehicle are stopped being collected. The accuracy of target tracking can be improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a target tracking method, device, electronic device and storage medium. Background Art

[0002] Autonomous driving technology is a hot topic in the current automotive industry and technological development. This technology plays a significant role in improving traffic safety, alleviating urban congestion, and promoting economic development. However, autonomous driving vehicles need to continuously perceive the motion state of themselves and surrounding vehicles in various environments and road conditions to ensure that the vehicle can drive safely in complex and changing road environments. In related technologies, targets can be tracked through generative models (such as optical flow method, particle filtering, Meanshift algorithm, etc.). However, a single mathematical model has great limitations in describing the target to be tracked, so this method has low accuracy. Summary of the invention

[0003] In order to solve the above technical problems, the present application provides a target tracking method, device, electronic device and storage medium.

[0004] According to a first aspect of the present application, a target tracking method is provided, comprising:

[0005] Acquire the motion state data of the first target at the previous moment, the motion state data of the first target includes: the motion state data of the second target and the motion state data of the lost target; the motion state data of the second target is obtained by inputting the image of the surroundings of the vehicle collected at the previous moment into a pre-trained 3D target detection model; the lost target is empty at the initial moment;

[0006] Collecting images around the vehicle at the current moment, and inputting the images into the 3D object detection model to obtain motion state data of the third object at the current moment;

[0007] The motion state data of the first target at the previous moment is predicted by the Kalman filter to obtain the motion state data of the first target at the current moment;

[0008] According to the motion state data of the first target and the third target at the current moment, determine the association cost value between each first target and each third target, and construct the association cost matrix between the first target and the third target according to the association cost value; wherein the association cost value is related to the position, overlap and direction of the first target and the third target;

[0009] Determining the global optimal match of the first objective and the third objective from the association cost matrix using the Hungarian algorithm;

[0010] Adding the motion state data of each third target at the current moment to the corresponding trajectory data;

[0011] The first target that is not successfully matched is determined as a lost target, and the lost targets that have been matched N times consecutively among the lost targets are deleted; N is an integer greater than 1;

[0012] The current moment is taken as the previous moment, the next moment is taken as the current moment, and the state of the Kalman filter is updated, and the step of obtaining the motion state data of the first target at the previous moment is returned until the image acquisition around the vehicle is stopped.

[0013] Optionally, the method further comprises:

[0014] Before predicting the motion state data of the first target at the previous moment through the Kalman filter, constructing the bird's-eye view motion state data of the first target at the previous moment according to the motion state data of the first target at the previous moment;

[0015] The method of predicting the motion state data of the first target at the previous moment by using the Kalman filter to obtain the motion state data of the first target at the current moment specifically includes:

[0016] The bird's-eye view motion state data of the first target at the previous moment is predicted by a Kalman filter to obtain the bird's-eye view motion state filtering data of the first target at the current moment;

[0017] The bird's-eye view motion state filtering data of the first target at the current moment and the motion state data in the Z direction of the motion state data of the first target at the previous moment are used as the motion state data of the first target at the current moment.

[0018] Optionally, the motion state data of the third target includes: the position coordinates, size and direction of the third target; the motion state data of the first target includes: the position coordinates, size and direction of the first target;

[0019] The step of determining the association cost between each third target and each first target according to the motion state data of the third target and the first target at the current moment includes:

[0020] For each third target and each first target, calculating a position cost value between the third target and the first target according to the position coordinates of the third target and the position coordinates of the first target;

[0021] Calculate a bounding box of the third object according to the position coordinates and size of the third object, and calculate a bounding box of the first object according to the position coordinates and size of the first object;

[0022] Calculating an overlap cost value between the bounding box of the third object and the bounding box of the first object;

[0023] Calculating a direction cost between the third target and the first target according to the direction of the third target and the direction of the first target;

[0024] The weighted average of the position cost value, the overlap cost value and the direction cost value between the third target and the first target is used as the association cost value between the third target and the first target.

[0025] Optionally, the motion state data of the first target further includes: a perception confidence of the first target;

[0026] The constructing of the association cost matrix between the third target and the first target includes:

[0027] For each first target, the product of the association cost value between the third target and the first target and the perceived confidence of the first target at the previous moment is taken as an element in the association cost matrix to construct the association cost matrix.

[0028] Optionally, calculating the position cost between the third target and the first target according to the position coordinates of the third target and the position coordinates of the first target includes:

[0029] Assume that p i represents the position coordinates of the third target i at the current moment from a bird's eye view, p j represents the position coordinates of the first target j at the current moment from a bird's eye view;

[0030] According to the formula:

[0031] Calculate the position cost P between the first target j and the third target i (i，j) , w is a preset value, w is greater than 0 and less than or equal to 2.

[0032] Optionally, calculating the overlap cost between the bounding box of the third object and the bounding box of the first object includes:

[0033] Assumption A i represents the object bounding box of the third target i in the three-dimensional space at the current moment, B j represents the object bounding box of the first target j in the three-dimensional space at the current moment;

[0034] According to the formula:

[0035] Calculate the overlap cost LIoU between the first target j and the third target i (i，j) .

[0036] Optionally, calculating the direction cost between the third target and the first target according to the direction of the third target and the direction of the first target includes:

[0037] Assumption F i represents the direction vector of the third target i at the current moment, F j represents the direction vector of the first target j at the current moment;

[0038] According to the formula: S (i-j) =1-cos(F i , F j );

[0039] Calculate the direction cost S between the first target j and the third target i (i，j) .

[0040] According to a second aspect of the present application, a target tracking device is provided, comprising:

[0041] The first target motion state data acquisition module is used to acquire the motion state data of the first target at the previous moment, wherein the motion state data of the first target includes: the motion state data of the second target and the motion state data of the lost target; the motion state data of the second target is obtained by inputting the image of the surroundings of the vehicle collected at the previous moment into a pre-trained 3D target detection model; the lost target at the initial moment is empty;

[0042] A third target motion state data determination module is used to collect images around the vehicle at the current moment, and input the images into the 3D target detection model to obtain the motion state data of the third target at the current moment;

[0043] A Kalman filter module is used to predict the motion state data of the first target at a previous moment through a Kalman filter to obtain the motion state data of the first target at a current moment;

[0044] An association cost value determination module, used to determine the association cost value between each first target and each third target according to the motion state data of the first target and the third target at the current moment; wherein the association cost value is related to the position, overlap and direction of the first target and the third target;

[0045] An association cost matrix construction module, used to construct an association cost matrix between the first target and the third target according to the association cost value;

[0046] A global optimal matching judgment module, used to determine the global optimal matching of the first target and the third target from the association cost matrix using the Hungarian algorithm;

[0047] A trajectory adding module, used to add the motion state data of each third target at the current moment to the corresponding trajectory data;

[0048] A lost target updating module, used to determine the first target that has not been successfully matched as a lost target, and delete the lost targets that have been matched N times in succession among the lost targets; N is an integer greater than 1;

[0049] The updating module is used to take the current moment as the previous moment and the next moment as the current moment, and to update the state of the Kalman filter, and return to the first target motion state data acquisition module until the image acquisition around the vehicle is stopped.

[0050] Optionally, the target tracking device further includes:

[0051] A bird's-eye view processing module is used to construct bird's-eye view motion state data of the first target at the previous moment according to the motion state data of the first target at the previous moment before predicting the motion state data of the first target at the previous moment through a Kalman filter;

[0052] The Kalman filter module is specifically used to predict the bird's-eye view motion state data of the first target at the previous moment through a Kalman filter to obtain the bird's-eye view motion state filtering data of the first target at the current moment; and use the bird's-eye view motion state filtering data of the first target at the current moment and the motion state data in the Z direction of the motion state data of the first target at the previous moment as the motion state data of the first target at the current moment.

[0053] Optionally, the motion state data of the third target includes: the position coordinates, size and direction of the third target; the motion state data of the first target includes: the position coordinates, size and direction of the first target;

[0054] The association cost value determination module is specifically used to calculate, for each third target and each first target, a position cost value between the third target and the first target according to the position coordinates of the third target and the position coordinates of the first target; calculate a bounding box of the third target according to the position coordinates and size of the third target, and calculate a bounding box of the first target according to the position coordinates and size of the first target; calculate an overlap cost value between the bounding box of the third target and the bounding box of the first target; calculate a direction cost value between the third target and the first target according to the direction of the third target and the direction of the first target; and use a weighted average of the position cost value, the overlap cost value and the direction cost value between the third target and the first target as the association cost value between the third target and the first target.

[0055] Optionally, the motion state data of the first target further includes: a perception confidence of the first target;

[0056] The constructing of the association cost matrix between the third target and the first target includes:

[0057] For each first target, the product of the association cost value between the third target and the first target and the perceived confidence of the first target at the previous moment is taken as an element in the association cost matrix to construct the association cost matrix.

[0058] Alternatively, assume that p i represents the position coordinates of the third target i at the current moment from a bird's eye view, p j represents the position coordinates of the first target j at the current moment from a bird's eye view;

[0059] The association cost value determination module is specifically used to calculate the position cost value between the third target and the first target through the following steps:

[0060] According to the formula:

[0061]

[0062] Calculate the position cost P between the first target j and the third target i (i，j) , w is a preset value, w is greater than 0 and less than or equal to 2.

[0063] Optionally, the association cost value determining module is specifically configured to calculate the overlap cost value between the bounding box of the third object and the bounding box of the first object through the following steps:

[0064] Assumption A i represents the object bounding box of the third target i in the three-dimensional space at the current moment, B j represents the object bounding box of the first target j in the three-dimensional space at the current moment;

[0065] According to the formula:

[0066] Calculate the overlap cost LIoU between the first target j and the third target i (i，j) .

[0067] Optionally, the association cost value determination module is specifically configured to calculate the direction cost value between the third target and the first target through the following steps:

[0068] Assumption F i represents the direction vector of the third target i at the current moment, F j represents the direction vector of the first target j at the current moment;

[0069] According to the formula: S (i，j) =1-cos(F i , F j );

[0070] Calculate the direction cost S between the first target j and the third target i(i，j) .

[0071] According to a third aspect of the present application, an electronic device is provided, comprising: a processor, wherein the processor is configured to execute a computer program stored in a memory, wherein the computer program implements the method described in the first aspect when executed by the processor.

[0072] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in the first aspect is implemented.

[0073] According to a fifth aspect of the present application, a computer program product is provided. When the computer program product is run on a computer, the computer is enabled to execute the method described in the first aspect.

[0074] Compared with the prior art, the technical solution provided by the embodiments of the present application has the following advantages:

[0075] The 3D target detection model is used to perform target detection on the image at the previous moment to obtain the motion state data of the second target at the previous moment, and the second target and the target that was not successfully matched before (i.e., the lost target) are merged into the first target. Target detection is performed on the image at the current moment to obtain the motion state data of the third target at the current moment. The Kalman filter is used to predict the motion state data of the first target at the previous moment, which can effectively deal with the noise in the perception process, thereby maintaining good anti-interference ability in complex traffic environments. For each first target and each third target, the matching relationship between the first target and the third target is judged from multiple dimensions such as the position, overlap, and direction correlation of the first target and the third target, so that the global optimal match can be determined more accurately, improving the accuracy and robustness of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0077] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0078] Figure 1 A flow chart of a target tracking method in an embodiment of the present application;

[0079] Figure 2 A schematic diagram of the structure of a target tracking device in an embodiment of the present application;

[0080] Figure 3 A schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0081] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0082] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0083] See also Figure 1 , Figure 1 A flow chart of a target tracking method in an embodiment of the present application may include the following steps:

[0084] Step S102, obtaining the motion state data of the first target at the previous moment, the motion state data of the first target includes: the motion state data of the second target and the motion state data of the lost target, the motion state data of the second target is obtained by inputting the image around the vehicle collected at the previous moment into a pre-trained 3D target detection model.

[0085] When the vehicle is driving, it can periodically collect images of the surroundings. The 3D target detection model can process the images and identify the motion state data of the second target in the image. The second target is the target identified from the image, which can be one or more. The motion state data of the second target can include: the position coordinates, size and direction of the first target, etc.

[0086] The lost target refers to the target that is lost during the target tracking process. The lost target is empty at the initial moment. As time changes, the lost target is constantly updated. The second target and the lost target are merged into the first target to continue tracking the lost target.

[0087] Since target tracking is performed in real time, the 3D target detection model can process the collected images in real time, that is, the images collected at the previous moment have been processed by the 3D target detection model at the previous moment. Therefore, in this cycle, the motion state data of the first target at the previous moment can be directly obtained.

[0088] Step S104, collecting images of the surroundings of the vehicle at the current moment, and inputting the images into a 3D object detection model to obtain motion state data of the third object at the current moment.

[0089] During this cycle, the 3D target detection model processes the image around the vehicle at the current moment to obtain the motion state data of the third target. Similarly, the third target is also a target identified from the image, which can be one or more. The motion state data of the third target can include: the position coordinates, size and direction of the first target, etc.

[0090] Step S106, predicting the motion state data of the first target at the previous moment through a Kalman filter to obtain the motion state data of the first target at the current moment.

[0091] The Kalman filter algorithm is a filtering algorithm based on the state space model, which is mainly used to estimate and predict the system state. The Kalman filter algorithm estimates and predicts the system state through the system model and the measurement model, and improves the estimation accuracy through continuous updating. The core of the Kalman filter algorithm is the state space model, which consists of two equations: the state transfer equation and the observation equation. The state transfer equation is used to describe the dynamic changes of the system state, and the observation equation is used to describe the relationship between the observed quantity and the system state.

[0092] Optionally, before predicting the motion state data of the first target at the previous moment through the Kalman filter, the bird's-eye view motion state data of the first target at the previous moment is constructed according to the motion state data of the first target at the previous moment. The bird's-eye view motion state data of the first target at the previous moment can be expressed as: (p x , v x , p y , v y ), (p x , p y ) represents the position coordinates of the first target from a bird's eye view, (v x , V y ) represents the speed of the first target in the X and Y directions from a bird's eye view.

[0093] Afterwards, the Kalman filter is used to predict the bird's-eye view motion state data of the first target at the previous moment, and the bird's-eye view motion state filter data of the first target at the current moment is obtained; the bird's-eye view motion state filter data of the first target at the current moment and the motion state data in the Z direction of the motion state data of the first target at the previous moment are used as the motion state data of the first target at the current moment.

[0094] The motion state of the vehicle at the current moment is predicted through the state transfer equation, and the covariance matrix is ​​updated.

[0095] According to the following prediction model formula: Get a priori estimate

[0096] Among them, F krepresents the state transfer equation, represents the posterior estimate of the motion state data of the first target at time k-1 (i.e., the previous time), represents the prior estimation of the motion state data of the first target at time k (i.e., the current time) through the state transfer equation, P k-1|k-1 represents the state covariance matrix at time k-1, P k|k-1 represents the predicted state covariance matrix generated at time k, W k The noise matrix representing the state variables, Q k Noise matrix representing the covariance.

[0097] According to the formula: The predicted motion state data of the first target at time k can be obtained

[0098] Among them, H represents the observation matrix representation.

[0099] In this way, simplifying the motion state data of the three-dimensional space to the BEV (Bird's Eye View) perspective can reduce the amount of calculation, so that it can run efficiently on devices with limited resources.

[0100] Step S108, determining the association cost between each first target and each third target according to the motion state data of the first target and the third target at the current moment, wherein the association cost is related to the position, overlap and direction of the first target and the third target.

[0101] The motion state data of the first target includes: the position coordinates, size and direction of the first target; the motion state data of the third target includes: the position coordinates, size and direction of the third target.

[0102] For each third target and each first target, the position cost value between the third target and the first target can be calculated according to the position coordinates of the third target and the position coordinates of the first target. The closer the distance between the first target and the third target is, the smaller the corresponding position cost value is.

[0103] According to the position coordinates and size of the third target, the bounding box of the third target is calculated, and according to the position coordinates and size of the first target, the bounding box of the first target is calculated, and then the overlap cost between the bounding box of the third target and the bounding box of the first target is calculated. The more the bounding box of the first target overlaps with the bounding box of the third target, the smaller the overlap cost. According to the direction of the third target and the direction of the first target, the direction cost between the third target and the first target is calculated. The more consistent the directions of the third target and the first target are, the smaller the direction cost is.

[0104] Alternatively, assume that A irepresents the object bounding box of the third target i in the three-dimensional space at the current moment, B j represents the object bounding box of the first target j in the three-dimensional space at the current moment;

[0105] According to the formula:

[0106] Calculate the overlap cost LIoU between the first target j and the third target i (i，j) .

[0107] Assumption F i represents the direction vector of the third target i at the current moment, F j represents the direction vector of the first target j at the current moment;

[0108] According to the formula: S (i，j) =1-cos(F i , F j );

[0109] Calculate the direction cost S between the first target j and the third target i (i，j) .

[0110] Alternatively, assume that p i represents the position coordinates of the third target i at the current moment from a bird's eye view, p j represents the position coordinates of the first target j at the current moment from a bird's eye view;

[0111] According to the formula:

[0112] Calculate the position cost P between the first target j and the third target i (i，j) .

[0113] Since the overlap cost value and the direction cost value are both less than or equal to 1, the position cost value can be set to be no greater than w, where w is a preset value, greater than 0 and less than or equal to 2, for example, w can be 2, etc. In this way, the accuracy of the associated cost value can be avoided from being reduced due to the position cost value being much greater than the overlap cost value and the direction cost value.

[0114] Finally, the weighted average of the position cost, overlap cost, and direction cost between the third target and the first target is taken as the association cost between the third target and the first target, which can be expressed as the following formula:

[0115] c ij =w p P (i，j )+w LIoU LqCy (i，j) +w s S (i，j) ;

[0116] Among them, c ij represents the association cost between the first target j and the third target i, W p , W LIoU and Ws represent the corresponding weight factors, representing the confidence of each feature. The credibility of LIoU and direction matching is high, so the weight ratio can be relatively high. LIoU and Ws are set to 0.5 and 0.3 respectively. Location perception is greatly affected by environmental factors, so its weight can be relatively small. p Set to 0.2.

[0117] Step S110: constructing an association cost matrix between the first target and the third target according to the association cost value.

[0118] By calculating the association cost value between each first target and each third target, the association cost matrix C can be obtained, which can be expressed as follows:

[0119]

[0120] Considering that the 3D object detection model has different perception difficulties for objects at different distances, the perception confidence can also be introduced into the cost matrix to improve the accuracy of object perception. For example, the farther the perceived object is, the lower the perception confidence of the object is, and the closer the perceived object is, the higher the perception confidence of the object is.

[0121] Optionally, the motion state data of the first target also includes: the perception confidence of the first target. For each first target, the product of the association cost value between the third target and the first target and the perception confidence of the first target at the previous moment is used as an element in the association cost matrix to construct the association cost matrix.

[0122] After introducing the perception confidence, the obtained association cost value is expressed as follows:

[0123]

[0124] Among them, f j Represents the perceptual confidence corresponding to the first target j.

[0125] Step S112, using the Hungarian algorithm, determining the global optimal match between the first target and the third target from the association cost matrix.

[0126] The Hungarian algorithm is used to find the global optimal match for the association cost matrix, and the best matching relationship between the first target at the previous moment and the third target at the current moment is determined, that is, the same target is determined from all the first targets and all the third targets. The matching results include: the first target and the third target that are successfully matched, the first target that is not successfully matched, and the third target that is not successfully matched.

[0127] Step S114, adding the motion state data of each third target at the current moment to the corresponding trajectory data.

[0128] For the first target and the third target that are successfully matched, it means that the first target can be tracked successfully. For the third target that is not successfully matched, it means that the third target is a new target, and the motion state data of the third target is the starting trajectory. Regardless of whether the third target is successfully matched, the motion state data of the third target needs to be added to the corresponding trajectory data.

[0129] Step S116, determining the first target that has not been successfully matched as a lost target, and deleting the lost targets that have been matched N times in succession among the lost targets; N is an integer greater than 1.

[0130] For the first target that is not matched successfully, it means that the first target existed at the previous moment, but does not exist at the current moment, that is, the first target is a lost target and can be tracked subsequently.

[0131] If the lost target has been matched N times without success, it means that the lost target has not appeared again in N consecutive frames of images after the last appearance. The lost target can be deleted and no longer tracked in the future to reduce the amount of calculation and save storage space.

[0132] Step S118, determining whether to stop collecting images around the vehicle.

[0133] If the acquisition of images around the vehicle is not stopped, step S120 is executed; if the acquisition of images around the vehicle is stopped, the process ends.

[0134] Step S120, taking the current moment as the previous moment, taking the next moment as the current moment, and updating the state of the Kalman filter.

[0135] The vehicle continuously collects new images while driving, and targets are tracked by updating the images, time, and state of the Kalman filter and executing the above cycle.

[0136] The method to update the state of the Kalman filter is as follows:

[0137] According to the Kalman gain calculation formula:

[0138]

[0139] Calculate the Kalman gain K at time k k , where R k is the observation noise covariance matrix at time k.

[0140] The a priori estimate is obtained through the above prediction model After that, the posterior estimate can be obtained after processing by the observation model, which is specifically based on the following observation model:

[0141]

[0142] Get the posterior estimate of the first target motion state data at time k And the state covariance matrix P at time k k|k , Z k represents the actual motion state data of the first target at time k, that is, the motion state data obtained by the 3D target detection model.

[0143] The target tracking method of the embodiment of the present application uses a 3D target detection model to perform target detection on the image at the previous moment, obtain the motion state data of the second target at the previous moment, and merge the second target and the target that was not successfully matched before (i.e., the lost target) into the first target. Target detection is performed on the image at the current moment respectively to obtain the motion state data of the third target at the current moment. Simplifying the motion state data of the three-dimensional space to the BEV perspective can reduce the amount of calculation, so that it can run efficiently on devices with limited resources. Through the Kalman filter, the motion state data of the first target at the previous moment is predicted, and the noise existing in the perception process can be effectively processed, so as to maintain good anti-interference ability in a complex traffic environment. For each first target and each third target, the matching relationship between the first target and the third target is judged from multiple dimensions such as the position, overlap and direction correlation of the first target and the third target, so that the global optimal match can be determined more accurately, and the accuracy and robustness of target tracking can be improved.

[0144] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0145] Corresponding to the above method embodiment, the present application embodiment also provides a target tracking device, see Figure 2 , the target tracking device 200 comprises:

[0146] The first target motion state data acquisition module 202 is used to acquire the motion state data of the first target at the previous moment, the motion state data of the first target includes: the motion state data of the second target and the motion state data of the lost target; the motion state data of the second target is obtained by inputting the image of the surroundings of the vehicle collected at the previous moment into the pre-trained 3D target detection model; the lost target at the initial moment is empty;

[0147] The third target motion state data determination module 204 is used to collect images around the vehicle at the current moment and input the images into the 3D target detection model to obtain the motion state data of the third target at the current moment;

[0148] A Kalman filter module 206 is used to predict the motion state data of the first target at a previous moment through a Kalman filter to obtain the motion state data of the first target at a current moment;

[0149] The association cost value determination module 208 is used to determine the association cost value between each first target and each third target according to the motion state data of the first target and the third target at the current moment; wherein the association cost value is related to the position, overlap and direction of the first target and the third target;

[0150] An association cost matrix construction module 210 is used to construct an association cost matrix between the first target and the third target according to the association cost value;

[0151] A global optimal matching judgment module 212 is used to determine the global optimal matching of the first target and the third target from the association cost matrix using the Hungarian algorithm;

[0152] A trajectory adding module 214 adds the motion state data of each third target at the current moment to the corresponding trajectory data;

[0153] A lost target updating module 216 is used to determine the first target that has not been successfully matched as a lost target, and delete the lost targets that have been matched N times in succession among the lost targets; N is an integer greater than 1;

[0154] The updating module 218 is used to take the current moment as the previous moment and the next moment as the current moment, and update the state of the Kalman filter, and return to the first target motion state data acquisition module 202 until the image acquisition around the vehicle stops.

[0155] Optionally, the target tracking device 200 further includes:

[0156] A bird's-eye view processing module is used to construct bird's-eye view motion state data of the first target at the previous moment according to the motion state data of the first target at the previous moment before predicting the motion state data of the first target at the previous moment through a Kalman filter;

[0157] The Kalman filter module 206 is specifically used to predict the bird's-eye view motion state data of the first target at the previous moment through the Kalman filter to obtain the bird's-eye view motion state filter data of the first target at the current moment; and use the bird's-eye view motion state filter data of the first target at the current moment and the motion state data in the Z direction of the motion state data of the first target at the previous moment as the motion state data of the first target at the current moment.

[0158] Optionally, the motion state data of the third target includes: the position coordinates, size and direction of the third target; the motion state data of the first target includes: the position coordinates, size and direction of the first target;

[0159] The association cost value determination module 208 is specifically used to calculate, for each third target and each first target, the position cost value between the third target and the first target according to the position coordinates of the third target and the position coordinates of the first target; calculate the bounding box of the third target according to the position coordinates and size of the third target, and calculate the bounding box of the first target according to the position coordinates and size of the first target; calculate the overlap cost value between the bounding box of the third target and the bounding box of the first target; calculate the direction cost value between the third target and the first target according to the direction of the third target and the direction of the first target; and use the weighted average of the position cost value, overlap cost value and direction cost value between the third target and the first target as the association cost value between the third target and the first target.

[0160] Optionally, the motion state data of the first target further includes: a perception confidence of the first target;

[0161] The association cost matrix construction module 210 is specifically used to construct the association cost matrix for each first target by taking the product of the association cost value between the third target and the first target and the perception confidence of the first target at the previous moment as an element in the association cost matrix.

[0162] Alternatively, assume that p i represents the position coordinates of the third target i at the current moment from a bird's eye view, p j represents the position coordinates of the first target j at the current moment from a bird's eye view;

[0163] The association cost determination module 208 is specifically configured to calculate the location cost between the third target and the first target through the following steps:

[0164] According to the formula:

[0165] Calculate the position cost P between the first target j and the third target i (i，j) , w is a preset value, w is greater than 0 and less than or equal to 2.

[0166] Optionally, the association cost value determining module 208 is specifically configured to calculate the overlap cost value between the bounding box of the third object and the bounding box of the first object through the following steps:

[0167] Assumption A i represents the object bounding box of the third target i in the three-dimensional space at the current moment, B j represents the object bounding box of the first target j in the three-dimensional space at the current moment;

[0168] According to the formula:

[0169] Calculate the overlap cost LIoU between the first target j and the third target i (i，j) .

[0170] Optionally, the association cost value determining module 208 is specifically configured to calculate the direction cost value between the third target and the first target through the following steps:

[0171] Assumption F i represents the direction vector of the third target i at the current moment, F j represents the direction vector of the first target j at the current moment;

[0172] According to the formula: S (i，j) =1-cos(F i , F j );

[0173] Calculate the direction cost S between the first target j and the third target i (i，j) .

[0174] The specific details of each module or unit in the above device have been described in detail in the corresponding method, so they will not be repeated here.

[0175] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0176] The present application also provides an electronic device, see Figure 3 , Figure 3 The electronic device of the embodiment of the present application includes: a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304;

[0177] Memory 303, used for storing computer programs;

[0178] The processor 301 is used to implement the above-mentioned target tracking method when executing the program stored in the memory 303.

[0179] It should be noted that the communication bus 304 mentioned in the above electronic device can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 304 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0180] The communication interface 302 is used for communication between the electronic device and other devices.

[0181] The memory 303 may include a RAM (Random Access Memory), or may include a non-volatile memory, such as at least one disk memory. Optionally, the memory 303 may also be at least one storage device located away from the aforementioned processor.

[0182] The above-mentioned processor 301 can be a general-purpose processor, including: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0183] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned target tracking method is implemented.

[0184] It should be noted that the computer-readable storage medium shown in the present application may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, radio frequency, etc., or any suitable combination of the above.

[0185] In an embodiment of the present application, a computer program product is also provided. When the computer program product is run on a computer, the computer executes the above-mentioned target tracking method.

[0186] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0187] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target tracking method, characterized in that: include: Acquire the motion state data of the first target at the previous moment, the motion state data of the first target includes: the motion state data of the second target and the motion state data of the lost target; the motion state data of the second target is obtained by inputting the image of the surroundings of the vehicle collected at the previous moment into a pre-trained 3D target detection model; the lost target is empty at the initial moment; Collecting images around the vehicle at the current moment, and inputting the images into the 3D object detection model to obtain motion state data of the third object at the current moment; The motion state data of the first target at the previous moment is predicted by the Kalman filter to obtain the motion state data of the first target at the current moment; According to the motion state data of the first target and the third target at the current moment, determine the association cost value between each first target and each third target, and construct the association cost matrix between the first target and the third target according to the association cost value; wherein the association cost value is related to the position, overlap and direction of the first target and the third target; Determining the global optimal match of the first objective and the third objective from the association cost matrix using the Hungarian algorithm; Adding the motion state data of each third target at the current moment to the corresponding trajectory data; The first target that is not successfully matched is determined as a lost target, and the lost targets that have been matched N times consecutively among the lost targets are deleted; N is an integer greater than 1; The current moment is taken as the previous moment, the next moment is taken as the current moment, and the state of the Kalman filter is updated, and the step of obtaining the motion state data of the first target at the previous moment is returned until the image acquisition around the vehicle is stopped.

2. The method according to claim 1, characterized in that The method further comprises: Before predicting the motion state data of the first target at the previous moment through the Kalman filter, constructing the bird's-eye view motion state data of the first target at the previous moment according to the motion state data of the first target at the previous moment; The method of predicting the motion state data of the first target at the previous moment by using the Kalman filter to obtain the motion state data of the first target at the current moment specifically includes: The bird's-eye view motion state data of the first target at the previous moment is predicted by a Kalman filter to obtain the bird's-eye view motion state filtering data of the first target at the current moment; The bird's-eye view motion state filtering data of the first target at the current moment and the motion state data in the Z direction of the motion state data of the first target at the previous moment are used as the motion state data of the first target at the current moment.

3. The method according to claim 1, characterized in that The motion state data of the third target includes: the position coordinates, size and direction of the third target; the motion state data of the first target includes: the position coordinates, size and direction of the first target; The step of determining the association cost between each third target and each first target according to the motion state data of the third target and the first target at the current moment includes: For each third target and each first target, calculating a position cost value between the third target and the first target according to the position coordinates of the third target and the position coordinates of the first target; Calculate a bounding box of the third object according to the position coordinates and size of the third object, and calculate a bounding box of the first object according to the position coordinates and size of the first object; Calculating an overlap cost value between the bounding box of the third object and the bounding box of the first object; Calculating a direction cost between the third target and the first target according to the direction of the third target and the direction of the first target; The weighted average of the position cost value, the overlap cost value and the direction cost value between the third target and the first target is used as the association cost value between the third target and the first target.

4. The method according to claim 3, characterized in that The motion state data of the first target also includes: the perception confidence of the first target; The constructing of the association cost matrix between the third target and the first target includes: For each first target, the product of the association cost value between the third target and the first target and the perceived confidence of the first target at the previous moment is taken as an element in the association cost matrix to construct the association cost matrix.

5. The method according to claim 3, characterized in that: The calculating the position cost value between the third target and the first target according to the position coordinates of the third target and the position coordinates of the first target includes: Assume that p i represents the position coordinates of the third target i at the current moment from a bird's eye view, p j represents the position coordinates of the first target j at the current moment from a bird's eye view; According to the formula: Calculate the position cost P between the first target j and the third target i (i，j) , w is a preset value, w is greater than 0 and less than or equal to 2.

6. The method according to claim 3, characterized in that The calculating the overlap cost value between the bounding box of the third object and the bounding box of the first object includes: Assumption A i represents the object bounding box of the third target i in the three-dimensional space at the current moment, B j represents the object bounding box of the first target j in the three-dimensional space at the current moment; According to the formula: Calculate the overlap cost LIoU between the first target j and the third target i (i，j) .

7. The method according to claim 3, characterized in that The calculating the direction cost between the third target and the first target according to the direction of the third target and the direction of the first target includes: Assumption F i represents the direction vector of the third target i at the current moment, F j represents the direction vector of the first target j at the current moment; According to the formula: S (i，j) =1-cos(F i , F j ); Calculate the direction cost S between the first target j and the third target i (i，j) .

8. A target tracking device, characterized in that: The device comprises: The first target motion state data acquisition module is used to acquire the motion state data of the first target at the previous moment, wherein the motion state data of the first target includes: the motion state data of the second target and the motion state data of the lost target; the motion state data of the second target is obtained by inputting the image of the surroundings of the vehicle collected at the previous moment into a pre-trained 3D target detection model; the lost target at the initial moment is empty; A third target motion state data determination module, used to collect images around the vehicle at the current moment, and input the images into the 3D target detection model to obtain motion state data of the third target at the current moment; A Kalman filter module is used to predict the motion state data of the first target at a previous moment through a Kalman filter to obtain the motion state data of the first target at a current moment; An association cost value determination module, used to determine the association cost value between each first target and each third target according to the motion state data of the first target and the third target at the current moment; wherein the association cost value is related to the position, overlap and direction of the first target and the third target; An association cost matrix construction module, used to construct an association cost matrix between the first target and the third target according to the association cost value; A global optimal matching judgment module, used to determine the global optimal matching of the first target and the third target from the association cost matrix using the Hungarian algorithm; A trajectory adding module, used to add the motion state data of each third target at the current moment to the corresponding trajectory data; A lost target updating module, used to determine the first target that has not been successfully matched as a lost target, and delete the lost targets that have been matched N times in succession among the lost targets; N is an integer greater than 1; The updating module is used to take the current moment as the previous moment and the next moment as the current moment, and to update the state of the Kalman filter, and return to the first target motion state data acquisition module until the image acquisition around the vehicle is stopped.

9. An electronic device, characterized in that: include: A processor, wherein the processor is used to execute a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images