Target tracking prediction method and device, electronic equipment and storage medium

Abstract

The application discloses a target tracking prediction method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining an optimal estimation value of a state quantity of a target at a current moment in a non-common-view area of a camera through Kalman filtering, and establishing a target tracking trajectory; judging whether the target enters a common-view area of the camera according to the target tracking trajectory; if yes, taking a predicted value of the optimal estimation value of the state quantity of the target at a last moment in the non-common-view area of the camera as an estimation value of the state quantity of the target at the current moment in the common-view area of the camera; obtaining an optimal estimation value of the state quantity of the target at the current moment in the common-view area of the camera according to the estimation value of the state quantity of the target at the current moment in the common-view area of the camera and a measured value of the state quantity of the target at the current moment in the common-view area of the camera, and tracking the target. The application reduces the situation that the target jumps in the roadside multi-camera perception.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a target tracking and prediction method, device, electronic device, and storage medium. Background Technology

[0002] In multi-camera target fusion, there exists a shared field of view within the detection field of view of each monocular camera. Since monocular cameras cannot directly obtain the depth information of the target, the depth information of the target detected by each monocular camera depends on the latitude and longitude calibration of the corresponding pixels on the ground in the pixel plane, thereby obtaining the target's depth information.

[0003] However, the common field of view between cameras is usually small, and at the edge of the common field of view (non-common field of view), it is easy for the position and velocity of the detected target to change. Summary of the Invention

[0004] This application provides a target tracking and prediction method, apparatus, electronic device, and storage medium to reduce the occurrence of position and velocity jumps in the detected target.

[0005] The embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, embodiments of this application provide a target tracking and prediction method, wherein the method includes:

[0007] Obtain the optimal estimate of the target's state at the current moment in the non-common field of view of the camera, and establish the target tracking trajectory;

[0008] Based on the target tracking trajectory, determine whether the target has entered the common field of view of the camera;

[0009] If so, the predicted value of the optimal estimate of the target's state quantity in the non-common field of view of the camera at the previous moment is used as the estimate of the target's state quantity in the common field of view of the camera at the current moment;

[0010] Based on the estimated value of the target's state quantity at the current moment in the camera's common field of view and the measured value of the target's state quantity at the current moment in the camera's common field of view, the optimal estimated value of the target's state quantity at the current moment in the camera's common field of view is obtained, and the target is tracked.

[0011] In some embodiments, the method further includes:

[0012] When the target enters the common field of view of the camera from the non-common field of view, the observation error, prediction error and motion model in the Kalman filter model are switched, while maintaining the same target tracking trajectory as the target in the non-common field of view of the camera;

[0013] And / or,

[0014] When the target is in the common field of view of the camera, the state variables of the target at the current moment and the next moment are predicted and the target is tracked. The observation error, prediction error and motion model in the Kalman filter model are maintained, and the same target tracking trajectory is maintained with the target in the non-common field of view of the camera.

[0015] In some embodiments, switching the observation error, prediction error, and motion model in the Kalman filter model includes:

[0016] Increase the observation error of the Kalman filter model, reduce the prediction error, and change the motion model from a uniform motion model to a uniformly accelerated motion model.

[0017] In some embodiments, the method further includes:

[0018] Based on the target tracking trajectory, determine whether the target has entered the non-common field of view of the new camera;

[0019] If so, the optimal estimate of the target's state at the current moment in the non-common field of view of the camera is then predicted using the Kalman filter model, while simultaneously establishing the target tracking trajectory.

[0020] In some embodiments, obtaining the optimal estimate of the target's state at the current moment in the non-common field of view of the camera and establishing the target tracking trajectory includes:

[0021] Hungarian matching is performed using the measured value of the target's state quantity at the current moment in the camera's non-common field of view and the predicted value of the target's state quantity at the current moment in the camera's non-common field of view.

[0022] Based on the matching results, the tracking target association is completed and the optimal estimate of the target's state variables at the current moment is obtained, and a tracking trajectory is established;

[0023] The above steps are repeated to predict the state of the target at the next time step from the current time step.

[0024] In some embodiments, obtaining a predicted value of the target's state quantity at the current moment in the non-common field of view of the camera includes:

[0025] Obtain the state variables of multiple targets in the non-common field of view of the camera at the previous time step;

[0026] Based on the state quantities of the multiple targets in the non-common field of view of the camera at the previous moment, the current state quantities of the multiple targets are estimated and predicted using a Kalman filter model, thereby obtaining the predicted values ​​of the current state quantities of the multiple targets in the non-common field of view of the camera.

[0027] In some embodiments, the step of predicting the current state value based on the optimal estimate of the target's state value at the previous moment in the camera's common field of view to obtain a predicted value of the current state value, comparing the predicted value of the current state value with the measured value of the target's state value at the current moment in the camera's common field of view, obtaining the optimal estimate of the target's state value at the current moment in the camera's common field of view, and tracking the target includes:

[0028] Obtain the measurement value of the target's current state in the camera's common field of view;

[0029] The optimal estimate of the target's state in the camera's non-common field of view at the previous moment is used to predict the target's state at the current moment using a Kalman filter model. The predicted value is then used as the estimate of the target's state in the camera's common field of view at the current moment.

[0030] Based on the estimated value of the target's state quantity in the camera's common field of view at the current moment and the measured value of the target's state quantity in the camera's common field of view at the current moment, a Hungarian matching is performed;

[0031] The target association is completed based on the matching results, and the optimal estimate of the target's state at the current moment in the camera's common field of view is obtained, while the target tracking is completed.

[0032] Secondly, embodiments of this application also provide a target tracking and prediction device, wherein the device includes:

[0033] Observation module: used to obtain measurement values;

[0034] Kalman prediction module: used to obtain predicted values ​​based on the motion model and observation model, and combine them with measured values ​​to obtain the optimal state variables;

[0035] Tracking module: used for target association and obtaining tracking trajectory;

[0036] The device is also used for,

[0037] The optimal estimate of the target's state at the current moment is predicted by the Kalman wave module in the non-common field of view of the camera, and the target tracking trajectory is established by the tracking module.

[0038] Based on the target tracking trajectory, determine whether the target has entered the common field of view of the camera;

[0039] If so, the predicted value of the optimal estimate of the target's state quantity in the non-common field of view of the camera at the previous moment is used as the estimated value of the target's state quantity in the common field of view of the camera at the current moment;

[0040] Based on the target's state quantity in the camera's common field of view at the previous moment and the measured value of the target's state quantity in the camera's common field of view at the current moment in the observation module, the optimal estimate of the target's state quantity in the camera's common field of view at the current moment is obtained through the Kalman prediction module, and the target is tracked.

[0041] Thirdly, embodiments of this application also provide an electronic device, including: a processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the above-described method.

[0042] Fourthly, embodiments of this application also provide a computer-readable storage medium that stores one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform the above-described method.

[0043] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:

[0044] By obtaining the optimal estimate of the target's state variables at the current moment in the camera's non-common field of view area and establishing a target tracking trajectory, it is then determined whether the target has entered the camera's common field of view area based on the target tracking trajectory. If so, the optimal estimate of the target's state variables at the current moment in the camera's common field of view area is obtained based on the estimated value and the measured value of the target's state variables at the current moment in the camera's common field of view area, and the target continues to be tracked.

[0045] After predicting and tracking the target and establishing the trajectory in the non-common field of view of the camera, the position can be corrected based on the previous tracking and prediction in the common field of view of the camera. The position and velocity of the detected target are added as state variables to the Kalman filter to reduce the occurrence of abrupt changes in position and velocity. Attached Figure Description

[0046] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0047] Figure 1 This is a schematic diagram of the target tracking and prediction method in the embodiments of this application;

[0048] Figure 2 This is a schematic diagram of the target tracking and prediction device in the embodiments of this application;

[0049] Figure 3 This is a schematic diagram of the common-view area and non-common-view area of ​​the roadside camera in the target tracking and prediction method in the embodiments of this application;

[0050] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] During their research, the inventors discovered that roadside units typically use three types of cameras: close-up cameras, long-range cameras, and fisheye cameras, each responsible for target detection in different fields of view. The problem of abrupt changes in the position and speed of targets occurs during multi-camera target fusion.

[0053] Thus, a shared viewing area exists within the detection field of view of multiple cameras. However, since a monocular camera cannot directly obtain the depth information of the target, the depth information of the target detected by each camera depends on the latitude and longitude calibration of the corresponding pixels on the ground in the pixel plane, thereby obtaining the depth information of the target.

[0054] Within a shared field of view, when a target moves from one camera's field of view to another, the disappearance of the target in one camera's field of view and its reappearance in another occur gradually due to the division of their respective fields of view. As the target gradually disappears from one camera's field of view, the ground latitude and longitude corresponding to the edge of the detected pixel frame change only slightly, resulting in a small change in the target's position, even though the actual target position has changed significantly. Similarly, when the target reappears in another camera's field of view, the ground latitude and longitude corresponding to the edge of the detected pixel frame change only slightly, resulting in a small change in the target's position. At the boundary between the two camera fields of view, a large jump in the target's position occurs, and this large jump in position causes a large jump in velocity.

[0055] To address the aforementioned shortcomings, the embodiments of this application can employ a target tracking prediction method to solve the problem of detecting position and velocity jumps of the target. Furthermore, the tracking prediction can be implemented using a prediction algorithm based on Kalman filtering and a Hungarian matching algorithm.

[0056] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0057] This application provides a target tracking and prediction method, such as... Figure 1 The diagram shows a flowchart of a target tracking and prediction method in an embodiment of this application. The method includes at least the following steps S110 to S140:

[0058] Step S110: Obtain the optimal estimate of the target's state at the current moment in the non-common field of view of the camera, and establish the target tracking trajectory.

[0059] like Figure 2 As shown, the target being tracked is illustrated using a "vehicle" as an example. The "non-common field of view" of a camera refers to the individual detection range of each camera. As a target passes through a roadside unit, it sequentially enters the different cameras on the roadside unit, such as a fisheye camera, a close-up camera, and a distant camera in that order. Therefore, the non-common field of view of each camera is the individual detection range of each of the fisheye, close-up, and distant cameras.

[0060] If we want to obtain the optimal estimate of the target's state at the current moment in the non-common field of view of the camera, we need to determine an optimal estimate from multiple estimates of the state at the current moment.

[0061] When calculating the estimated state variables of multiple targets at the current moment, the state variables of multiple targets in the non-common field of view of the camera at the current moment can be predicted using a Kalman filter model, thus obtaining the estimated state variables of multiple targets. Then, the estimated values ​​need to be matched with the measured values ​​(observations) at the current moment to obtain an optimal estimate.

[0062] The Kalman filter model uses the optimal estimate from the previous time step to predict the current value, and simultaneously uses the current observation to correct the current estimate, thus obtaining the optimal result.

[0063] Hungarian matching is performed on the measured and predicted state values ​​of the multiple targets in the non-common field of view region of the camera. Based on the matching results, target association is completed, and then the optimal estimate of the state value at the current moment is obtained. Finally, the tracking trajectory of the target is established. In other words, within the non-field-of-view edge region (common field of view region) of the camera, the target is predicted and tracked using a Kalman filter algorithm and a Hungarian matching algorithm, and a trajectory is established.

[0064] Hungarian matching can be used for target tracking. Typically, the Intersection over Union (IoU) between the predicted bounding box of a target and the bounding box of the target in the previous frame can be used to determine if it belongs to the same target. The specific principles are not elaborated here. The Hungarian matching algorithm is used to determine whether the predicted value of the target at the current moment and the measured value of the target at the current moment belong to the same target.

[0065] It should be noted that if no field of view transition occurs, the state variables of the next time step will repeat the above steps, and stable Kalman prediction can be maintained based on a stable target tracking trajectory.

[0066] Here, "field transition" refers to the same target moving out of the field of view of one camera while simultaneously moving into the field of view of another camera.

[0067] Step S120: Based on the target tracking trajectory, determine whether the target has entered the common field of view of the camera.

[0068] The system determines whether the target has entered the camera's shared field of view (COP) by tracking its trajectory, i.e., whether a field-of-view transition has occurred. If not, target tracking and target position prediction continue. If so, the target is considered to have entered the camera's COP, and the predicted target state variables need to be optimized.

[0069] Step S130: If yes, then the predicted value of the optimal estimate of the target's state quantity in the non-common field of view of the camera at the previous moment is used as the estimate of the target's state quantity in the common field of view of the camera at the current moment.

[0070] The optimal estimate of the target's state in the camera's non-common field of view, obtained in the previous step, is used as the target's state in the camera's common field of view at the previous moment. The purpose of this is:

[0071] When the target is in the camera's common field of view, the Kalman filter algorithm and Hungarian matching algorithm are used to predict and track the target and establish a trajectory. When the target is in the camera's non-common field of view, the position can be corrected based on the tracking and prediction. The position and velocity of the detected target are added as state variables to the Kalman filter to reduce position and velocity jumps.

[0072] Step S140: Based on the estimated value of the target's state quantity at the current moment in the camera's common field of view and the measured value of the target's state quantity at the current moment in the camera's common field of view, obtain the optimal estimated value of the target's state quantity at the current moment in the camera's common field of view, and track the target.

[0073] Based on the target's state variables in the camera's shared field of view at the previous moment and the measured state variables at the current moment, the estimated state variables of multiple targets in the camera's shared field of view at the current moment can be predicted. Hungarian matching is then performed using the measured and predicted state variables of these multiple targets in the camera's shared field of view. Based on the matching results, target association is completed, and the optimal estimate of the current state variables is obtained. Finally, the target's tracking trajectory is established. That is, even when the target is outside the camera's shared field of view, its position can be corrected based on previous tracking and prediction. The detected target's position and velocity are added as state variables to the Kalman filter to reduce position and velocity jumps.

[0074] The process of completing the target association based on the matching results and then obtaining the optimal estimate of the state variables at the current time is as follows:

[0075] The target's current state value in the camera's common field of view is measured. Based on the previous state value, the predicted state value at the current time is predicted using a Kalman filter. Hungarian matching is then performed to complete target association and obtain the optimal estimate of the current state value. Tracking is completed simultaneously during this process.

[0076] It is important to note that if no field-of-view transition occurs, the observation error parameters, prediction error parameters, and motion model of the Kalman filter model set in the camera's common field of view are maintained, and the target maintains the same tracking trajectory in the camera's non-common field of view.

[0077] In the above method, considering that the position of the target detected at the edge of the camera's field of view, i.e., the common viewing area, changes, while the position is relatively stable in the non-field of view range, i.e., the non-common viewing area, the positioning result can maintain a certain accuracy. Furthermore, the position of the target detected by each camera is transformed to the UTM coordinate system according to the pixel values ​​corresponding to latitude and longitude. Therefore, the target can be predicted and tracked by the Kalman filter algorithm and the Hungarian matching algorithm in the non-field of view range (common viewing area) and a trajectory can be established. In the non-common viewing area, the position can be corrected based on tracking and prediction. The position and velocity of the detected target are added as state variables to the Kalman filter to reduce position and velocity jumps.

[0078] like Figure 2 As shown, three types of cameras—a close-up camera, a long-range camera, and a fisheye camera—are used in the roadside unit, each responsible for target detection in a different field of view. Within the detection fields of multiple cameras, there exists a shared viewing area. However, since a monocular camera cannot directly obtain the target's depth information, the depth information of the target detected by each camera relies on the latitude and longitude calibration of the corresponding pixels on the ground within the pixel plane to obtain the target's depth information. The aforementioned target tracking and prediction method specifically includes the following steps:

[0079] S1 determines the division of the detection field of view for each camera within the detection range of this road test unit.

[0080] S2. Establish a Kalman filter model, taking the target's position (x, y), velocity (Vx, Vy), and heading angle (YAW) as state variables. The state variables are predicted and corrected by the established Kalman filter. Here, it is necessary to establish motion models and observation models for the state variables.

[0081] S3. Establish a motion model for the state variables of the detected target. Use a uniform velocity model at the edge of the camera's field of view and a uniform acceleration model at the edge of the camera's non-field of view.

[0082] S4. Establish an observation model for detecting the state variables of the target.

[0083] S4 sets the motion variance and observation variance of the state variables.

[0084] S5 uses a Kalman filter to predict state variables.

[0085] S6 involves using the Hungarian matching algorithm to correlate and track the predicted target in the current frame with the observed target in the next frame. Tracking is required based on whether the target in the current frame and the next frame is the same.

[0086] S7 increases the variance of the observed state variables and decreases the variance of the predicted state variables at the edge of the camera's field of view.

[0087] S8, for the target being tracked, combines the variance set in S7 and then uses Kalman filtering for position correction.

[0088] S9. Use Kalman filtering to obtain the optimal state quantity estimate.

[0089] S10 outputs stable target position and velocity in the common viewing area.

[0090] Prediction and tracking are achieved using Kalman filters and Hungarian matching algorithms to solve the problem of position and velocity jumps caused by excessive edge detection of targets in multi-camera field of view.

[0091] In one embodiment of this application, the method further includes: when the target enters the common field of view of the camera from the non-common field of view of the camera, switching the observation error, prediction error, and motion model in the Kalman filter model, while maintaining the same target tracking trajectory as the target in the non-common field of view of the camera; and / or, when the target is in the common field of view of the camera, continuing to predict the state quantity of the target at the next moment of the current moment and tracking the target, while maintaining the observation error, prediction error, and motion model in the Kalman filter model, while maintaining the same target tracking trajectory as the target in the non-common field of view of the camera.

[0092] Please continue to refer to this. Figure 2 Typically, the target first enters the camera's non-common field of view, and then moves from the non-common field of view into the camera's common field of view. The following classifications are based on different scenarios:

[0093] Case 1: When the target enters the common field of view of the camera from the non-common field of view of the camera, it can be determined whether the target has entered the common field of view of the camera based on the target tracking trajectory.

[0094] If the target enters the camera's common field of view (from the camera's non-common field of view), the observation error, prediction error, and motion model in the Kalman filter model are switched, while maintaining the same target tracking trajectory as the target in the camera's non-common field of view.

[0095] Typically, the target is located outside the camera's common field of view. The Kalman filter model employs a uniform motion model, increasing the observation error of the Kalman filter while decreasing the prediction error. Specifically, when the target is at the edge of the camera's detection field of view (the camera's common field of view), the variance of the observed state variables is increased, while the variance of the predicted state variables is decreased. Simultaneously, the uniform motion model is replaced with a uniform acceleration model.

[0096] Case 2: When the target is in the common field of view of the camera, it can be determined whether the target is still in the common field of view of the camera based on the target tracking trajectory.

[0097] When the target is in the common field of view of the camera, the state quantity of the target at the current moment and the state quantity at the next moment continue to be predicted and the target is tracked. The observation error, prediction error and motion model in the Kalman filter model are maintained. That is, the error of the observation quantity, the error of the prediction quantity and the motion model set for the target in the common field of view are maintained and the target maintains the same tracking trajectory in the non-common field of view of the camera.

[0098] It is understandable that the above situations 1 and 2 occur alternately, with the target entering or leaving the camera's common field of view and non-common field of view alternately.

[0099] That is, based on the tracking trajectory, it is determined whether the target has left the common field of view and entered the non-common field of view of the next camera. If it has entered the non-common field of view of the next camera, the position prediction result can be corrected based on the tracking and prediction of the target in the common field of view of the camera. The detected position and velocity of the target are added as state variables to the Kalman filter to reduce the jumps in position and velocity.

[0100] In one embodiment of this application, switching the observation error, prediction error, and motion model in the Kalman filter model includes: increasing the observation error of the Kalman filter model, decreasing the prediction error, and changing the motion model from a uniform motion model to a uniformly accelerated motion model.

[0101] Generally, increasing the observation error and decreasing the prediction error of the Kalman filter model means increasing the variance of the observed state variables while decreasing the variance of the predicted state variables. The motion model for detecting the target state variables established in the Kalman filter model uses a uniform velocity model when the target is in the camera's non-common field of view and a uniform acceleration model when the target is in the camera's common field of view. Then, for the tracked target in the camera's non-common field of view, based on the variance calculated above, Kalman filtering is used for position correction.

[0102] Based on the tracking trajectory, it can be determined whether the target has entered the common field of view. If it has entered the common field of view, the observation error of the Kalman filter is increased, the prediction error is reduced, and the motion model is changed to a uniform acceleration model. The measurement value of the target's state quantity at the current moment in the camera's common field of view is obtained. Based on the state quantity at the previous moment, the predicted value of the state quantity at the current moment is predicted by the Kalman filter. Hungarian matching is performed to complete the target association and obtain the optimal estimate of the state quantity at the current moment. This process also completes the tracking.

[0103] In one embodiment of this application, the method further includes: determining whether the target has entered a new non-common field of view region of the camera based on the target tracking trajectory; if so, continuing to predict the optimal estimate of the target's state quantity at the current moment in the non-common field of view region of the camera using a Kalman filter model, and simultaneously establishing the target tracking trajectory.

[0104] Since the roadside unit includes multiple cameras, there are multiple non-common viewing areas for each camera.

[0105] Based on the target tracking trajectory, if it is determined that the target has entered a new non-common field of view of the camera, the optimal estimate of the target's state at the current moment in the non-common field of view of the camera is predicted using the Kalman filter model, and the target tracking trajectory is established simultaneously.

[0106] It can be understood that the optimal estimate is obtained by matching the estimated state of the target in the non-common field of view of the camera at the current moment, which is predicted by the Kalman filter model, with the measured value at the current moment.

[0107] In one embodiment of this application, obtaining the optimal estimate of the target's state quantity at the current moment in the non-common field of view of the camera and establishing the target tracking trajectory includes: performing Hungarian matching by using the measured value of the target's state quantity at the current moment in the non-common field of view of the camera and the predicted value of the target's state quantity at the current moment in the non-common field of view of the camera; based on the matching result, completing the tracking target association and obtaining the optimal estimate of the target's state quantity at the current moment, and establishing the tracking trajectory; repeating the above steps for the prediction of the target's state quantity at the next moment.

[0108] The establishment of the target tracking trajectory mainly includes the following methods:

[0109] The measurement value of the target's state at the current moment in the non-common field of view of the camera is obtained. The measurement value can be obtained from the actual scene.

[0110] The estimated state of the target in the non-common field of view region of the camera at the current moment, predicted by the Kalman filter model.

[0111] Hungarian matching is performed on the measured value of the target's state quantity at the current moment in the non-common field of view of the camera and the predicted value of the target's state quantity at the current moment in the non-common field of view of the camera. Based on the matching result, the tracking target association is completed and the optimal estimate of the target's state quantity at the current moment is obtained, and a tracking trajectory is established. Then, if the target is still in the non-common field of view of the camera, the above steps are repeated for the prediction of the target's state quantity at the next moment.

[0112] In one embodiment of this application, the target includes multiple targets. Obtaining the predicted value of the state quantity of the target at the current moment in the non-common field of view of the camera includes: obtaining the state quantity of the multiple targets in the non-common field of view of the camera at the previous moment; and estimating and predicting the current state quantity of the multiple targets using a Kalman filter model based on the state quantity of the multiple targets in the non-common field of view of the camera at the previous moment, thereby obtaining the predicted value of the state quantity of the multiple targets at the current moment in the non-common field of view of the camera.

[0113] During target tracking, there may be multiple targets.

[0114] For each target, the state quantities of multiple targets in the non-common field of view of the camera at the previous moment are obtained. Then, based on the state quantities of the multiple targets in the non-common field of view of the camera at the previous moment, the current state quantities of the multiple targets are estimated and predicted using a Kalman filter model. Finally, the predicted values ​​of the state quantities of the multiple targets in the non-common field of view of the camera at the current moment are obtained.

[0115] In one embodiment of this application, the step of predicting the current state value based on the optimal estimate of the target's state value in the camera's shared field of view at the previous moment to obtain a predicted value of the current state value, and obtaining the optimal estimate of the target's current state value in the camera's shared field of view by comparing the predicted value of the current state value with the measured value of the target's current state value in the camera's shared field of view, and then tracking the target, includes: obtaining the measured value of the target's current state value in the camera's shared field of view; using the optimal estimate of the target's state value in the camera's non-shared field of view at the previous moment to predict the predicted value of the target's current state value using a Kalman filter model, and using the predicted value as the estimated value of the target's current state value in the camera's shared field of view; and performing Hungarian matching based on the estimated value of the target's current state value in the camera's shared field of view and the measured value of the target's current state value in the camera's shared field of view.

[0116] Based on the state variables of multiple targets in the non-common field of view of the camera at the previous moment, a Kalman filter is used to estimate and predict the current state variables of the multiple targets in the non-common field of view of the camera, so as to obtain the predicted value of the state variables of the multiple targets in the non-common field of view of the camera at the current moment.

[0117] By performing Hungarian matching on the measured and predicted values ​​of the state variables of the multiple targets in the non-common field of view of the camera at the current moment, target association is completed and the optimal estimate of the state variables at the current moment is obtained, and a tracking trajectory is established. The state variables at the next moment are repeated in the above steps. A stable trajectory can maintain a stable Kalman prediction.

[0118] This application also provides a target tracking and prediction device 300, such as... Figure 3 The diagram shows a schematic representation of the target tracking and prediction device in an embodiment of this application. The target tracking and prediction device 300 includes at least: an observation module 310, a Kalman prediction module 320, and a tracking module 330, wherein:

[0119] In one embodiment of this application, the observation module 310 is specifically used to: obtain measurement values;

[0120] In one embodiment of this application, the Kalman prediction module 320 is specifically used to: obtain predicted values ​​based on the motion model and the observation model, and combine them with measured values ​​to obtain the optimal state quantity;

[0121] In one embodiment of this application, the tracking module 330 is specifically used for: target association to obtain a tracking trajectory.

[0122] The optimal estimate of the target's state at the current moment is predicted by the Kalman wave module in the non-common field of view of the camera, and the target tracking trajectory is established by the tracking module.

[0123] like Figure 2 As shown, the target being tracked is illustrated using a "vehicle" as an example. The "non-common field of view" of a camera refers to the individual detection range of each camera. As a target passes through a roadside unit, it sequentially enters the different cameras on the roadside unit, such as a fisheye camera, a close-up camera, and a distant camera in that order. Therefore, the non-common field of view of each camera is the individual detection range of each of the fisheye, close-up, and distant cameras.

[0124] If we want to obtain the optimal estimate of the target's state at the current moment in the non-common field of view of the camera, we need to determine an optimal estimate from multiple estimates of the state at the current moment.

[0125] When calculating the estimated state variables of multiple targets at the current moment, the state variables of multiple targets in the non-common field of view of the camera at the current moment can be predicted using a Kalman filter model, thus obtaining the estimated state variables of multiple targets. Then, the estimated values ​​need to be matched with the measured values ​​(observations) at the current moment to obtain an optimal estimate.

[0126] The Kalman filter model uses the best result from the previous time step to predict the current value, while simultaneously using the observed value to correct the current value, thus obtaining the optimal result.

[0127] Hungarian matching is performed on the measured and predicted state values ​​of the multiple targets in the non-common field of view region of the camera. Based on the matching results, target association is completed, and then the optimal estimate of the state value at the current moment is obtained. Finally, the tracking trajectory of the target is established. In other words, within the non-field-of-view edge region (common field of view region) of the camera, the target is predicted and tracked using a Kalman filter algorithm and a Hungarian matching algorithm, and a trajectory is established.

[0128] Hungarian matching can be used for target tracking. Typically, the Intersection over Union (IoU) between the predicted bounding box of a target and the bounding box of the target in the previous frame can be used to determine if it belongs to the same target. The specific principles are not elaborated here. The Hungarian matching algorithm is used to determine whether the predicted value of the target at the current moment and the measured value of the target at the current moment belong to the same target.

[0129] It should be noted that if no field of view transition occurs, the state variables of the next time step will repeat the above steps, and stable Kalman prediction can be maintained based on a stable target tracking trajectory.

[0130] Here, "field transition" refers to the same target moving out of the field of view of one camera while simultaneously moving into the field of view of another camera.

[0131] Based on the target tracking trajectory, determine whether the target has entered the common field of view of the camera;

[0132] The system determines whether the target has entered the camera's shared field of view (COP) by tracking its trajectory, i.e., whether a field-of-view transition has occurred. If not, target tracking and target position prediction continue. If a transition occurs, the target is considered to be within the camera's COP, and the predicted target state variables need to be optimized.

[0133] If so, the predicted value of the optimal estimate of the target's state quantity in the non-common field of view of the camera at the previous moment is used as the estimated value of the target's state quantity in the common field of view of the camera at the current moment;

[0134] The optimal estimate of the target's state in the camera's non-common field of view, obtained in the previous step, is used as the target's state in the camera's common field of view at the previous moment. The purpose of this is:

[0135] When the target is in the camera's common field of view, the Kalman filter algorithm and Hungarian matching algorithm are used to predict and track the target and establish a trajectory. When the target is in the camera's non-common field of view, the position can be corrected based on the tracking and prediction. The position and velocity of the detected target are added as state variables to the Kalman filter to reduce position and velocity jumps.

[0136] The optimal estimate of the target's state in the camera's non-common field of view at the previous moment is used to predict the target's state at the current moment using a Kalman filter model. This predicted value is then used as the estimate of the target's state in the camera's common field of view at the current moment. This estimate is matched with the measured value of the target's state in the camera's common field of view at the current moment to obtain the optimal estimate for the current moment, and the target is then tracked.

[0137] Based on the target's state variables in the camera's shared field of view at the previous moment and the measured state variables at the current moment, the estimated state variables of multiple targets in the camera's shared field of view at the current moment can be predicted. Hungarian matching is then performed using the measured and predicted state variables of these multiple targets in the camera's shared field of view. Based on the matching results, target association is completed, and the optimal estimate of the current state variables is obtained. Finally, the target's tracking trajectory is established. That is, even when the target is outside the camera's shared field of view, its position can be corrected based on previous tracking and prediction. The detected target's position and velocity are added as state variables to the Kalman filter to reduce position and velocity jumps.

[0138] The process of completing the target association based on the matching results and then obtaining the optimal estimate of the state variables at the current time is as follows:

[0139] The target's current state value in the camera's common field of view is measured. Based on the previous state value, the predicted state value at the current time is predicted using a Kalman filter. Hungarian matching is then performed to complete target association and obtain the optimal estimate of the current state value. Tracking is completed simultaneously during this process.

[0140] It is important to note that if no field-of-view transition occurs, the observation error parameters, prediction error parameters, and motion model of the Kalman filter model set in the camera's common field of view are maintained, and the target maintains the same tracking trajectory in the camera's non-common field of view.

[0141] It is understood that the target tracking prediction device described above can implement each step of the target tracking prediction method provided in the foregoing embodiments. The relevant explanations of the target tracking prediction method are applicable to the target tracking prediction device and will not be repeated here.

[0142] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 4 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0143] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0144] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0145] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming a target tracking and prediction device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0146] Obtain the optimal estimate of the target's state at the current moment in the non-common field of view of the camera, and establish the target tracking trajectory;

[0147] Based on the target tracking trajectory, determine whether the target has entered the common field of view of the camera;

[0148] If so, the predicted value of the optimal estimate of the target's state quantity in the non-common field of view of the camera at the previous moment is used as the estimate of the target's state quantity in the common field of view of the camera at the current moment;

[0149] Based on the estimated value of the target's state quantity at the current moment in the camera's common field of view and the measured value of the target's state quantity at the current moment in the camera's common field of view, the optimal estimated value of the target's state quantity at the current moment in the camera's common field of view is obtained, and the target is tracked.

[0150] The above is as stated in this application. Figure 1The target tracking and prediction device disclosed in the illustrated embodiments can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0151] The electronic device can also perform Figure 1 The method for executing a target tracking and prediction device, and the implementation of the target tracking and prediction device in... Figure 1 The functions of the embodiments shown are not described in detail here.

[0152] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform... Figure 1 The method executed by the target tracking and prediction device in the illustrated embodiment is specifically used to perform:

[0153] Obtain the optimal estimate of the target's state at the current moment in the non-common field of view of the camera, and establish the target tracking trajectory;

[0154] Based on the target tracking trajectory, determine whether the target has entered the common field of view of the camera;

[0155] If so, the predicted value of the optimal estimate of the target's state quantity in the non-common field of view of the camera at the previous moment is used as the estimate of the target's state quantity in the common field of view of the camera at the current moment;

[0156] Based on the estimated value of the target's state quantity at the current moment in the camera's common field of view and the measured value of the target's state quantity at the current moment in the camera's common field of view, the optimal estimated value of the target's state quantity at the current moment in the camera's common field of view is obtained, and the target is tracked.

[0157] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0158] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0159] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0160] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0161] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0162] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0163] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0164] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0165] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0166] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A target tracking prediction method, wherein, The method includes: Obtain the optimal estimate of the target's state at the current moment in the non-common field of view of the camera, and establish the target tracking trajectory; Based on the target tracking trajectory, determine whether the target has entered the common field of view of the camera; If so, the predicted value of the optimal estimate of the target's state quantity in the non-common field of view of the camera at the previous moment is used as the estimate of the target's state quantity in the common field of view of the camera at the current moment; Based on the estimated value of the target's state quantity at the current moment in the camera's common field of view and the measured value of the target's state quantity at the current moment in the camera's common field of view, the optimal estimated value of the target's state quantity at the current moment in the camera's common field of view is obtained, and the target is tracked.

2. The method as described in claim 1, wherein, The method further includes: When the target enters the camera's common field of view from the camera's non-common field of view, the observation error, prediction error, and motion model in the Kalman filter model are switched, while maintaining the same target tracking trajectory as the target in the camera's non-common field of view. And / or, When the target is in the camera's common field of view, the system continues to predict the target's state variables for the next time step and track the target, while maintaining the observation error, prediction error, and motion model in the Kalman filter model, and simultaneously maintaining the same target tracking trajectory as the target in the camera's non-common field of view.

3. The method as described in claim 2, wherein, The switching of observation error, prediction error, and motion model in the Kalman filter model includes: Increase the observation error of the Kalman filter model, reduce the prediction error, and change the motion model from a uniform motion model to a uniformly accelerated motion model.

4. The method of claim 1, wherein, The method further includes: Based on the target tracking trajectory, determine whether the target has entered the non-common field of view of the new camera; If so, the optimal estimate of the target's state at the current moment in the non-common field of view of the camera is then predicted using the Kalman filter model, while simultaneously establishing the target tracking trajectory.

5. The method of claim 1, wherein, The step of obtaining the optimal estimate of the target's state at the current moment in the non-common field of view of the camera, and tracking the target, includes: Hungarian matching is performed using the measured value of the target's state quantity at the current moment in the camera's non-common field of view and the predicted value of the target's state quantity at the current moment in the camera's non-common field of view. Based on the matching results, the tracking target association is completed and the optimal estimate of the target's state variables at the current moment is obtained, and a tracking trajectory is established; The above steps are repeated to predict the state of the target at the next time step from the current time step.

6. The method of claim 5, wherein, The targets include multiple targets, and the predicted values ​​of the state variables of the targets in the non-common field of view of the camera at the current time are obtained, including: Obtain the state variables of multiple targets in the non-common field of view of the camera at the previous time step; Based on the state quantities of the multiple targets in the non-common field of view of the camera at the previous moment, the current state quantities of the multiple targets are estimated and predicted using a Kalman filter model, thereby obtaining the predicted values ​​of the current state quantities of the multiple targets in the non-common field of view of the camera.

7. The method of claim 1, wherein, The step of predicting the current state value based on the optimal estimate of the target's state value in the camera's common field of view at the previous moment, obtaining a predicted state value for the current moment, and then obtaining the optimal estimate of the target's state value in the camera's common field of view at the current moment based on the predicted state value and the measured state value of the target in the camera's common field of view at the current moment, and tracking the target, includes: Obtain the measurement value of the target's current state in the camera's common field of view; The optimal estimate of the target's state in the camera's non-common field of view at the previous moment is used to predict the target's state at the current moment using a Kalman filter model. The predicted value is then used as the estimate of the target's state in the camera's common field of view at the current moment. Based on the estimated value of the target's state quantity at the current moment in the camera's common field of view and the measured value of the target's state quantity at the current moment in the camera's common field of view, a Hungarian matching is performed; The target association is completed based on the matching results, and the optimal estimate of the target's state at the current moment in the camera's common field of view is obtained, while the target tracking is completed.

8. A target tracking and prediction device, wherein, The device includes: Observation module: used to obtain measurement values; Kalman prediction module: used to obtain predicted values ​​based on the motion model and observation model, and combine them with measured values ​​to obtain the optimal state variables; Tracking module: used for target association and obtaining tracking trajectory; The device is also used for, The optimal estimate of the target's state at the current moment is predicted by the Kalman wave module in the non-common field of view of the camera, and the target tracking trajectory is established by the tracking module. Based on the target tracking trajectory, determine whether the target has entered the common field of view of the camera; If so, the predicted value of the optimal estimate of the target's state quantity in the non-common field of view of the camera at the previous moment is used as the estimated value of the target's state quantity in the common field of view of the camera at the current moment; The optimal estimate of the target's state in the camera's non-common field of view at the previous moment is used to predict the target's state at the current moment using a Kalman filter model. This predicted value is then used as the estimate of the target's state in the camera's common field of view at the current moment. This estimate is matched with the measured value of the target's state in the camera's common field of view at the current moment to obtain the optimal estimate for the current moment, and the target is then tracked.

9. An electronic device, comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of applications, cause the electronic device to perform the method of any one of claims 1 to 7.

Images