Image multi-target tracking method and system based on fusion of optical flow and kalman filter

Abstract

The present application relates to a kind of image multi-target tracking method and system based on optical flow and Kalman filtering fusion, belong to intelligent driving technical field, solve the problem of insufficient persistence and accuracy of multi-target tracking in prior art. Including obtaining the target frame detection result of each frame image according to deep learning model;The target frame detection result of each frame image is simultaneously imported into optical flow prediction model and Kalman filtering prediction model, respectively, the target frame optical flow prediction result and Kalman prediction result of the next frame image corresponding to each frame image are obtained, and the target frame fusion prediction result is obtained by weighted summation;For the target frame detection result and target frame fusion prediction result of each frame image, the Mahalanobis distance between target frame is calculated, and the association between target frame detection result and target frame fusion prediction result is established using Hungarian algorithm, and the updated target frame state is obtained by Kalman filtering algorithm as the target tracking result of each frame image. Multi-target continuous tracking is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, and in particular to an image multi-target tracking method based on fusion of optical flow and Kalman filtering. BACKGROUND

[0002] The purpose of multi-target tracking is to track the targets in adjacent image frames in the process of continuous image frames. In the field of intelligent driving, it is particularly important to continuously track the targets because the position and speed of specific targets in the image need to be estimated in real time.

[0003] Currently, there are two main research directions for multi-target tracking. One is the filtering and matching method, and the other is the deep learning-based method. The filtering and matching method generally uses the Kalman filtering algorithm to predict the position of the target, and then associates the prediction with the detection result to construct the data association. Then, the Kalman filter is used to update the position of the target in the image. The deep learning-based method for multi-target tracking essentially combines target detection and data association. Data association is usually based on appearance, and the similarity of the feature vector is calculated to perform data association and tracking between frames.

[0004] The filtering and matching method is highly dependent on the accuracy and continuity of target detection. When the target detection is not accurate enough, the prediction result of the Kalman filter will have a large error. In addition, if the same target is not continuously detected, and the number of intermediate frames is large, the Kalman filter will only have a prediction, which will result in a large difference between the predicted result and the actual target position, and the data association cannot be successfully performed. If a deep learning method is used, a high model and platform computing power is required for stable tracking. If the computing power is not enough, the frame rate cannot meet the requirements, and the model must be simplified. Once the model is simplified, the accuracy and continuity of detection will be greatly reduced, and it cannot meet the requirements of the intelligent driving application environment in terms of continuity and real-time performance. SUMMARY

[0005] In view of the above analysis, the embodiments of the present application aim to provide an image multi-target tracking method and system based on fusion of optical flow and Kalman filtering, to solve the problems of insufficient continuity and accuracy of existing multi-target tracking.

[0006] In one aspect, the embodiments of the present application provide an image multi-target tracking method based on fusion of optical flow and Kalman filtering, comprising the following steps:

[0007] Real-time video images are collected and input into a deep learning model for target detection to obtain the target box detection result of each frame of image;

[0008] The target frame detection result of each frame of image is simultaneously input into the optical flow prediction model and the Kalman filter prediction model, the target frame optical flow prediction result and the Kalman prediction result of the next frame of image corresponding to each frame of image are obtained respectively, and the target frame fusion prediction result is obtained by weighted summation;

[0009] The target frame detection result and the target frame fusion prediction result of each frame of image are associated by calculating the Mahalanobis distance between the target frames and adopting the Hungarian algorithm, and the updated target frame state is obtained by the updating method in the Kalman filter algorithm as the target tracking result of each frame of image.

[0010] Based on the further improvement of the above method, the optical flow prediction model obtains the target frame to be tracked of each frame of image and the corresponding next frame of image according to the target frame detection result of each frame of image, removes the error feature points according to the moving vectors of the feature points in the target frame to be tracked, obtains the target frame initial prediction result and the target frame scale change rate according to the moving vectors of the remaining feature points, and obtains the target frame optical flow prediction result based on the target frame initial prediction result and the target frame scale change rate.

[0011] Based on the further improvement of the above method, the target frame to be tracked of each frame of image and the corresponding next frame of image is obtained according to the target frame detection result of each frame of image, including:

[0012] Based on the position change amount of each target frame in each frame of image and the previous frame of image, the position of each target frame in each frame of image is added to the corresponding position change amount to obtain each predicted target frame in the next frame of image corresponding to each frame of image; if the corresponding target frame does not exist in the previous frame of image, the position change amount is 0; the target frame position includes the x coordinate and y coordinate of the upper left corner of the target frame, and the width and height of the target frame;

[0013] The expansion amount of the target frame is obtained according to the number of layers of the optical flow pyramid and the tracking window size;

[0014] For each target frame in each frame of image and each predicted target frame in the corresponding next frame of image, the height and width of the target frame are expanded according to the expansion amount, as the target frame to be tracked.

[0015] Based on the further improvement of the above method, the expansion amount E of the target frame is obtained according to the number of layers of the optical flow pyramid and the tracking window size by the following formula:

[0016] E=2 n ×r

[0017] Wherein, n is the number of layers of the optical flow pyramid, and r is the tracking window radius.

[0018] Further improvement based on the above method, according to the moving vector of the feature points in the target frame to be tracked, remove the error feature points, including:

[0019] Take each frame image and the corresponding next frame image as the front and rear frame images to be tracked, obtain the feature points in the target frame to be tracked in the front and rear frame images as a feature point pair, and track;

[0020] For each successfully tracked feature point pair, calculate the moving vector of the feature points; the successfully tracked feature point pair is greater than 1 pair;

[0021] Take each moving vector as a reference vector in turn, calculate the cosine distance between the other moving vectors and the reference vector, and count the number of cosine distances greater than the threshold value, and take the reference vector with the most corresponding number as the optimal moving vector;

[0022] The feature points with a cosine distance less than the threshold value from the optimal moving vector are regarded as error feature points, and the error feature points are removed.

[0023] Further improvement based on the above method, according to the moving vector of the remaining feature points to obtain the initial prediction result of the target frame and the scale change rate of the target frame, including:

[0024] According to the moving vector of the remaining feature points, calculate the average moving vector;

[0025] Add the average moving vector to the coordinates of the four first vertices of each target frame of each frame image to obtain the corresponding four second vertex coordinates in the next frame image;

[0026] Use the minimum enclosing rectangle method on the four second vertex coordinates to obtain the initial prediction result of each target frame of each frame image in the next frame image;

[0027] According to the distance ratio of the remaining feature points in each frame image and the corresponding next frame image, calculate the scale change rate of the target frame.

[0028] Further improvement based on the above method, according to the distance ratio of the remaining feature points in each frame image and the corresponding next frame image, calculate the scale change rate of the target frame, including:

[0029] Calculate the distance between any two feature points in the remaining feature points in each frame image as the first distance, and calculate the distance between the two feature points in the corresponding next frame image as the second distance; when the first distance and the second distance are both greater than the pixel threshold value, the above two feature points are regarded as a pair of effective feature points;

[0030] Sum the ratio of the first distance to the second distance of all effective feature point pairs and take the average to obtain the scale change rate of the target frame.

[0031] According to a further improvement of the above method, when the target frame fusion prediction result is obtained by weighting and summing the optical flow prediction result and the Kalman prediction result corresponding to each frame of image, the weight corresponding to the optical flow prediction result is greater than the weight corresponding to the Kalman prediction result.

[0032] According to a further improvement of the above method, the Mahalanobis distance between the target frames includes:

[0033] According to the state transition equation of the Kalman filter prediction model, the predicted covariance matrix of the state is calculated according to the target frame fusion prediction result.

[0034] According to the predicted covariance matrix of the state and the observation equation, the predicted covariance matrix of the observation is calculated.

[0035] According to the target frame detection result of each frame of image, the target frame fusion prediction result and the predicted covariance matrix of the observation, the Mahalanobis distance value between each target frame detection result of each frame of image and each target frame fusion prediction result is calculated.

[0036] All Mahalanobis distance values greater than the distance threshold value are updated to the same preset distance value, and the preset distance value is greater than the distance threshold value; and according to the updated Mahalanobis distance value, a cost matrix is constructed for use in the Hungarian algorithm.

[0037] On the other hand, the embodiment of the present application provides an image multi-target tracking system based on fusion of optical flow and Kalman filter, comprising:

[0038] An image acquisition module is configured to acquire video images in real time.

[0039] A target detection module is configured to detect targets in the video images according to a deep learning model to obtain target frame detection results of each frame of image.

[0040] An optical flow prediction module is configured to input the target frame detection results of each frame of image into an optical flow prediction model to obtain target frame optical flow prediction results of the next frame of image corresponding to each frame of image.

[0041] A Kalman filter prediction module is configured to input the target frame detection results of each frame of image into a Kalman filter prediction model to obtain target frame Kalman prediction results of the next frame of image corresponding to each frame of image.

[0042] The target tracking module is used for weighting and summing a target frame fusion prediction result of an optical flow prediction result and a Kalman prediction result of a next frame image corresponding to each frame image to obtain a target frame fusion prediction result; and for the target frame detection result of each frame image and the target frame fusion prediction result, a Mahalanobis distance between target frames is calculated, a Hungarian algorithm is used to establish a correlation between the target frame detection result and the target frame fusion prediction result, and an updating method in the Kalman filtering algorithm is used to obtain an updated target frame state as a target tracking result of each frame image.

[0043] Compared with the prior art, the present application can at least realize one of the following beneficial effects:

[0044] 1. A multi-target tracking method for images is proposed based on the optical flow method, which improves the real-time performance and accuracy of multi-target tracking through methods such as cropping, scale estimation, moving vector, removing false feature points, and scale change rate.

[0045] 2. The optical flow is used as an important part of target tracking and is fused with the Kalman filtering algorithm, which can still ensure the continuity and accuracy of tracking in the case of discontinuous deep learning detection, and the problem of increasing variance of tracking results caused by the Kalman filtering prediction is solved by adding the optical flow target frame tracking part.

[0046] In the present application, the above technical solutions can be combined with each other to realize more preferred combination solutions. Other features and advantages of the present application will be described in the subsequent specification, and some advantages will become apparent from the specification or be understood through the implementation of the present application. The purposes and other advantages of the present application can be realized and obtained through the contents specifically indicated in the specification and the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0047] The accompanying drawings are included to provide a further understanding of the application and are incorporated herein and constitute a part of the application. The same reference numerals in the drawings refer to the same elements throughout.

[0048] Figure 1 A flow chart of the multi-target tracking method for images based on the fusion of optical flow and Kalman filtering in the embodiment 1 of the present application. DETAILED DESCRIPTION

[0049] The preferred embodiments of the present application are specifically described below in combination with the drawings, wherein the drawings constitute a part of the present application and are used to explain the principles of the embodiments of the present application, but are not used to limit the scope of the present application.

[0050] Embodiment 1

[0051] One specific embodiment of the present application discloses a multi-target tracking method based on optical flow and Kalman filter fusion, which uses deep learning for target detection, uses an optical flow and Kalman filter fusion method for prediction, uses a Hungarian algorithm for data association when deep learning detects a target, and finally uses Kalman filtering for updating to obtain a target tracking result. Figure 1 As shown in the figure, the method comprises the following steps:

[0052] S11: Real-time acquisition of video images is performed, and the target detection result of each frame of image is obtained by inputting the video images into a deep learning model.

[0053] It should be noted that the image data of the camera is collected in real time as the input image of the deep learning according to the time interval. In this embodiment, the deep learning model is not limited, and exemplarily, a frame of image is acquired every 30 milliseconds, and the target frame detection result of each frame of image is obtained by inputting the frame of image into a YOLO series neural network model, which includes the pixel coordinates of the upper left corner and the lower right corner of each target frame on the image.

[0054] S12: The target frame detection result of each frame of image is simultaneously input into an optical flow prediction model and a Kalman filter prediction model to obtain the target frame optical flow prediction result and the Kalman prediction result of the next frame of image corresponding to each frame of image respectively, and the target frame fusion prediction result is obtained by weighted summation.

[0055] It should be noted that the target (such as a vehicle or a pedestrian) in the video image can be dynamic or static, the movement direction of the dynamic target relative to the camera can be different, and the distance of the target from the camera is also different, so in this embodiment, the optical flow prediction model is used to track each target to be tracked respectively, detect the feature points in each target frame, and track the feature points by using the optical flow method, and on this basis, the tracking of the target frame is realized.

[0056] (1) The prediction process of the optical flow prediction model is described below through steps S121-S124:

[0057] S121: According to the target frame detection result of each frame of image, the target frame to be tracked of each frame of image and the corresponding next frame of image is obtained.

[0058] It should be noted that the width and height of the target frame are obtained according to the target frame detection result of each frame of image, and in this step, the target frame is represented by four state variables [zx zy w h], wherein zx and zy represent the x coordinate and y coordinate of the upper left corner of the target frame, w represents the width of the target frame, and h represents the height of the target frame.

[0059] Specifically, first, based on the position change amount of each target frame in each target frame and the previous frame image, the position of each target frame in each target frame is added to the corresponding position change amount, and the position of each predicted target frame in the corresponding next frame image is obtained. That is, according to the position change amount of the target frame obtained by comparing the current frame with the previous frame, the position of the target frame in the next frame is predicted.

[0060] Exemplarily, the position of the target frame in the first frame is [zx1 zy1 w1 h1], the position of the target frame in the second frame is [zx2 zy2 w2 h2], and the change amount of the target frame between the two frames is [zx2-zx1 zy2-zy1 w2-w1 h2-h1], which is represented as [σzxσzyσwσh]. Under the condition that the change amount of the target frame between the third frame and the second frame is equal to the change amount between the second frame and the first frame, the position of the target frame in the third frame is predicted as [zx2+σzxzy2+σzy w2+σw h2+σh].

[0061] It should be noted that the position change amount of the initial first frame image is set to 0, and if the corresponding target frame does not exist in the previous frame image, the corresponding position change amount is set to 0. The previous frame image refers to the previous frame image of the tracking.

[0062] Secondly, according to the number of layers of the optical flow pyramid and the size of the tracking window, the expansion amount of the target frame is obtained;

[0063] It should be noted that the optical flow algorithm uses the image pyramid model in the tracking process, and the optical flow tracking itself needs to specify the size of the window rectangle (actually a square) to be used in the tracking process. According to the number of layers n of the optical flow pyramid and the size r of the tracking window, the expansion amount E of the target frame is obtained by the following formula:

[0064] E = 2 n × r (1)

[0065] Wherein, the values of n and r can be determined according to the test effect of tracking, generally n < 7, in order to improve the tracking efficiency, r < 30, preferably, in the embodiment, n = 3, r = 10 pixels.

[0066] Finally, for each target frame in each target frame and each predicted target frame in the corresponding next frame image, the height and width of the target frame are expanded according to the expansion amount, as the target frame to be tracked. This way does not miss the feature points on the edge of the target frame, and improves the accuracy of tracking.

[0067] Since only the feature points in the target frame need to be tracked, in order to improve the tracking efficiency, according to the size of the target frame to be tracked obtained after expansion, the image in the target frame to be tracked is cropped in each frame image and the corresponding next frame image, which is used in the subsequent steps of the optical flow prediction model.

[0068] S122: removing the error feature points according to the movement vectors of the feature points in the target frame to be tracked.

[0069] Specifically, taking each frame image and the corresponding next frame image as the front and rear frame images to be tracked, the feature points in the target frame to be tracked in the front and rear frame images are obtained as a feature point pair, and tracking is performed;

[0070] It should be noted that for each target frame to be tracked, the number of feature point pairs that are successfully tracked is greater than 1; if the number of feature point pairs that are successfully tracked is less than 2, the target frame is deleted and no longer tracked.

[0071] For each feature point pair that is successfully tracked, the movement vector of the feature point is calculated; each movement vector is taken as a reference vector, the cosine distance between the reference vector and other movement vectors is calculated, and the number of cosine distances greater than the threshold value is counted. The reference vector corresponding to the largest number is taken as the optimal movement vector.

[0072] The feature points with a cosine distance less than the threshold value from the optimal movement vector are taken as error feature points, and the error feature points are removed.

[0073] It should be noted that the movement vector of the feature point is used to represent the movement direction of the feature point, and is calculated by the following formula:

[0074]

[0075] wherein [u l v l ] is the pixel coordinate of the feature point in each frame image, and [u n v n ] is the pixel coordinate of the feature point in the corresponding next frame image tracked by the optical flow.

[0076] Since the unitization has been performed when the movement vector is calculated, the cosine distance between two movement vectors is obtained according to the dot product of the two movement vectors.

[0077] Each movement vector is taken as a reference vector, the cosine distance between the reference vector and other movement vectors is calculated, and the number of cosine distances greater than the threshold value is counted. It should be noted that when the cosine distance between two movement vectors is greater than the threshold value, it indicates that the movement directions of the two feature points are similar. Here, the reference vector when the number of cosine distances greater than the threshold value is the largest is taken as the optimal movement vector, that is, the most unified movement direction is obtained. The feature points that are not similar to this movement direction are taken as error feature points, and the corresponding feature point pairs are removed from the feature point pairs that are successfully tracked.

[0078] Preferably, the threshold value of the cosine distance is set to cos 5°.

[0079] S123: obtaining a target frame initial prediction result and a target frame scale change rate according to the movement vectors of the remaining feature points.

[0080] Specifically, the target frame initial prediction result is obtained according to the movement vectors of the remaining feature points, including:

[0081] According to the movement vectors of the remaining feature points, an average movement vector is calculated;

[0082] The four first vertex coordinates of each target frame of each image are added to the average movement vector to obtain the corresponding four second vertex coordinates in the next frame of image;

[0083] The four second vertex coordinates are used to obtain the initial prediction result of each target frame of each image in the next frame of image by using the minimum enclosing rectangle method.

[0084] According to the distance ratio of the remaining feature points in the front and back frames of image, a target frame scale change rate is calculated for further correction of the initial prediction result, including:

[0085] The distances of any two feature points in the remaining feature points in each frame of image are calculated as first distances, and the distances of the two feature points in the corresponding next frame of image are calculated as second distances; when the first distance and the second distance are both greater than a pixel threshold, the above two feature points are regarded as a pair of effective feature points.

[0086] The sum of the ratios of the first distance to the second distance of all effective feature point pairs is averaged to obtain the target frame scale change rate.

[0087] Specifically, the distance of the two feature points is calculated by the following formula:

[0088]

[0089] wherein, (tx i ,ty i ), (tx j ,ty j ) are the coordinates of the two feature points.

[0090] Preferably, the pixel threshold is 3 pixels.

[0091] The target frame scale change rate is calculated by the following formula for all effective feature point pairs:

[0092]

[0093] wherein, N represents the number of the remaining feature points, M represents the number of the effective feature point pairs, represents the first distance of the feature point i and the feature point j, denotes the second distance between the feature point i and the feature point j.

[0094] S124: obtaining the optical flow prediction result of the target frame based on the initial prediction result of the target frame and the scale change rate of the target frame.

[0095] It should be noted that the initial prediction result is multiplied by the scale change rate of the target frame to obtain the optical flow prediction result of the target frame, and each optical flow prediction result of the target frame is converted into a form represented by the x coordinate and y coordinate of the center point of the target frame, the aspect ratio r of the target frame and the height h of the target frame, i.e. [x y r h].

[0096] Since the image is cropped according to the expanded target frame in step S121, after the optical flow prediction result is obtained in step S124, it can be restored to the original image, and the target frame is labeled in the original image.

[0097] Compared with the prior art, the present step proposes a method for image multi-target tracking based on the optical flow method, which improves the real-time performance and accuracy of multi-target tracking through cropping, scale estimation, moving vector, removing false feature points, scale change rate and other methods.

[0098] (2) The target frame detection result output by the deep learning model is transmitted into the Kalman filter prediction model at the same time to obtain the Kalman prediction result.

[0099] The state of the target frame is defined as wherein c x and c y are the x coordinate and y coordinate of the center point of the target frame, r is the aspect ratio of the target frame, h is the height of the target frame, and the other four are the corresponding derivatives, respectively representing the change rates of the x coordinate, y coordinate, aspect ratio and height.

[0100] Further, under the condition of uniform motion, the state transition equation from time k to time k+1 is as follows, wherein dt represents the time interval from time k to time k+1:

[0101] The x coordinate transition equation of the center point of the target frame is:

[0102]

[0103] The y coordinate transition equation of the center point of the target frame is:

[0104]

[0105] The aspect ratio transition equation is:

[0106]

[0107] Target box height transfer equation:

[0108]

[0109] Target box center point x direction velocity transfer equation:

[0110]

[0111] Target box center point y direction velocity transfer equation:

[0112]

[0113] Width-height ratio change rate:

[0114]

[0115] Height change rate:

[0116]

[0117] According to formulas (5)-(12), the extracted coefficient matrix A is as follows:

[0118]

[0119] According to the Kalman filtering algorithm, the state transfer equation of the prediction process is:

[0120]

[0121] where ω k+1 is the random noise of the prediction error at k+1, which conforms to the normal distribution, Q k+1 is the covariance matrix of the state at k+1.

[0122] Based on formula (14), the state of the target box at k+1 is predicted according to the state X k of the target box at k, and the predicted state of the target box at k+1 is taken as the Kalman prediction result.

[0123] In the prior art, when the deep learning target detection is not accurate enough or the target detection is discontinuous, the variance of the Kalman filtering prediction result is very large, and even the result is not usable at all. Therefore, after the target is predicted by using the optical flow and the Kalman filtering respectively, the two results are fused, and by adding the target box tracking part of the optical flow, the problem that the variance of the tracking result becomes larger and larger due to the prediction by the Kalman filtering is solved.

[0124] ​The prediction result of the optical flow is obtained based on the optimization method of the image appearance, and thus is more reliable. Therefore, when the prediction result of the optical flow and the prediction result of the Kalman filter are fused, the weight corresponding to the prediction result of the optical flow is greater than the weight corresponding to the prediction result of the Kalman filter, and the formula is as follows:

[0125] P = s x r lk + (1 - s) x r kf (15)

[0126] wherein r lk is the prediction result of the optical flow, r kf is the prediction result of the Kalman filter, and s is the weight corresponding to the prediction result of the optical flow, and is preferably set to 0.85.

[0127] After the weighted summation according to the weight, the fusion prediction result of each target frame is expressed by the x coordinate and the y coordinate of the center point of the target frame, the height-width ratio r of the target frame, and the height h of the target frame, that is, in the form of [x y r h].

[0128] S13: For the target frame detection result and the target frame fusion prediction result of each frame of image, the Mahalanobis distance between the target frames is calculated, the association between the target frame detection result and the target frame fusion prediction result is established by using the Hungarian algorithm, and the updated target frame state is obtained by using the updating method in the Kalman filter algorithm as the target tracking result of each frame of image.

[0129] It should be noted that the Mahalanobis distance between the target frames is calculated, that is, the Mahalanobis distance between the target frame detection result and the target frame fusion prediction result of each frame of image is calculated, including:

[0130] ①According to the state transition equation of the Kalman filter prediction model, the predicted covariance matrix of the state is calculated according to the target frame fusion prediction result;

[0131] The target frame detection result output by the deep learning model obtained in step S11 is taken as the observed target frame, and the observation equation is established. The observation state Y = [z cx z cy z r z h ] of the observed target frame is defined, which respectively represents the x coordinate, the y coordinate, the height-width ratio and the height of the center point of the observed target frame.

[0132] Further, the observation equation of the target frame at time k is as follows:

[0133] The x coordinate observation equation of the center point of the target frame is as follows:

[0134]

[0135] The y coordinate observation equation of the center point of the target frame is as follows:

[0136]

[0137] Target box width-height ratio observation equation:

[0138]

[0139] Target box height observation equation:

[0140]

[0141] According to formula (16)-(19), the extracted coefficient matrix C is as follows:

[0142]

[0143] According to the Kalman filtering algorithm, the observation equation is:

[0144] Y k =CX k +v k v k ~N(0,R k ) (21)

[0145] Where v k is the observation noise, which conforms to the normal distribution, and R k is the covariance matrix of the observation noise at time k.

[0146] Based on the state transition equation (14) of the Kalman filtering prediction model, the state of the target box fusion prediction result at time k is predicted according to the state of the target box fusion prediction result at time k-1, and then the predicted covariance matrix of the state is obtained according to the following formula:

[0147] P k =AP k-1 A T +Q k-1 (22)

[0148] 2) According to the predicted covariance matrix of the state and the observation equation, the predicted covariance matrix of the observation is calculated;

[0149] Specifically, the predicted covariance matrix of the observation is obtained according to the following formula:

[0150] S k =CP k C T +R k (23)

[0151] ③According to the target box detection result of each frame of image, the target box fusion prediction result and the observed prediction covariance matrix, the Mahalanobis distance value between each target box detection result of each frame of image and each target box fusion prediction result is calculated;

[0152] Specifically, the formula of the Mahalanobis distance is:

[0153]

[0154] Wherein, L(a, b) is the Mahalanobis distance between the a th observation result (target box detection result) and the b th prediction result (target box fusion prediction result) in each frame of image, X b is the b th target box fusion prediction result in each frame of image, Y a is the target box detection result of the a th target box in each frame of image, S a is the observed prediction covariance matrix of the a th target box in each frame of image; the target box fusion prediction result and the target box detection result are four-dimensional state quantities in the form of [x y r h].

[0155] ④The Mahalanobis distance values greater than the distance threshold value are updated to the same preset distance value, and the preset distance value is greater than the distance threshold value; the cost matrix is constructed according to the updated Mahalanobis distance value, and is used in the Hungarian algorithm.

[0156] It should be noted that according to the chi-square distribution, the distance threshold value is set to 9.4877 in the embodiment, and when the calculated Mahalanobis distance value is greater than the threshold value, it is directly defined as mismatching. In order to avoid mismatching, all the Mahalanobis distance values greater than the distance threshold value are updated to the same preset distance value, and the preset distance value is greater than the distance threshold value.

[0157] When m target boxes are detected in the current frame of image according to step S11, n target boxes are predicted according to step S12, and the Mahalanobis distance value is used, a m*n cost matrix can be constructed for each frame of image, and in the Hungarian algorithm, the target box detection result and the target box fusion prediction result are matched and associated according to the cost matrix.

[0158] If the target box detection result and the target box fusion prediction result are associated, the updating method of the Kalman filter algorithm is used to obtain the updated target box state, and the x coordinate, y coordinate, width-height ratio and height of the center point are taken as the tracking result of the final target box on the image, and the tracking is continued. If there is a target box detection result that is not associated with the target box fusion prediction result, the target box detection result is taken as a new tracking target, and the tracking is performed when the next frame of image data arrives.

[0159] Preferably, if the number of times that any target box fusion prediction result is continuously not associated with the target box detection result exceeds the tracking number threshold, the target box fusion prediction result is deleted and tracking is no longer continued. Exemplarily, the tracking number threshold is 12 times.

[0160] Compared with the prior art, the image multi-target tracking method based on fusion of optical flow and Kalman filtering provided by the embodiment proposes a multi-target tracking method on the basis of the optical flow method, improves the real-time performance and accuracy of multi-target tracking through methods such as cropping, scale estimation, movement vector, removal of false feature points, and scale change rate; takes the optical flow as an important part of target tracking, and fuses the Kalman filtering algorithm, so that the continuity and accuracy of tracking can still be ensured in the case of discontinuous deep learning detection, and the problem of increasing variance of tracking results caused by prediction of the Kalman filtering is solved by adding the target box tracking part of the optical flow.

[0161] Embodiment 2

[0162] Another embodiment of the application discloses an image multi-target tracking system based on fusion of optical flow and Kalman filtering, so as to realize the multi-target tracking method in embodiment 1. The specific implementation modes of the modules are referred to the corresponding descriptions in embodiment 1. The system comprises:

[0163] An image acquisition module is configured to acquire video images in real time.

[0164] A target detection module is configured to perform target detection on the video images according to a deep learning model to obtain target box detection results of each frame of image.

[0165] An optical flow prediction module is configured to input the target box detection results of each frame of image into an optical flow prediction model to obtain target box optical flow prediction results of a next frame of image corresponding to each frame of image.

[0166] A Kalman filtering prediction module is configured to input the target box detection results of each frame of image into a Kalman filtering prediction model to obtain target box Kalman prediction results of the next frame of image corresponding to each frame of image.

[0167] A target tracking module is configured to perform weighted summation on the target box optical flow prediction results and the Kalman prediction results of the next frame of image corresponding to each frame of image to obtain target box fusion prediction results, and perform association between the target box detection results and the target box fusion prediction results by calculating Mahalanobis distances between the target boxes and using a Hungarian algorithm, and obtain updated target box states by using an updating method in the Kalman filtering algorithm, as target tracking results of each frame of image.

[0168] Since the image multi-target tracking system based on fusion of optical flow and Kalman filtering of the embodiment is related to the foregoing multi-target tracking method, mutual learning can be performed, and thus, repeated description is not given here. Since the system embodiment has the same principle as the foregoing method embodiment, the system also has the corresponding technical effects of the foregoing method embodiment.

[0169] Those skilled in the art can understand that all or part of the processes of the foregoing embodiments can be completed by a computer program instructing relevant hardware, and the program can be stored in a computer readable storage medium. The computer readable storage medium is a disk, an optical disk, a read-only memory, a random access memory, or the like.

[0170] The above merely describes the preferred embodiments of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered by the protection scope of the present application.

Claims

1. An image multi-target tracking method based on fusion of optical flow and Kalman filtering, characterized in that, The method comprises the following steps: real-time video image acquisition, target frame detection results of each frame image are obtained by inputting into a deep learning model for target detection; target frame detection results of each frame image are simultaneously input into an optical flow prediction model and a Kalman filter prediction model, target frame optical flow prediction results and Kalman prediction results of the next frame image corresponding to each frame image are obtained respectively, and target frame fusion prediction results are obtained by weighted summation; wherein, the optical flow prediction model obtains the target frame to be tracked of each frame image and the corresponding next frame image according to the target frame detection results of each frame image; removes the error feature points according to the moving vectors of the feature points in the target frame to be tracked; obtains the target frame initial prediction results and the target frame scale change rate according to the moving vectors of the remaining feature points; obtains the target frame optical flow prediction results based on the target frame initial prediction results and the target frame scale change rate; the weight corresponding to the optical flow prediction results is greater than the weight corresponding to the Kalman prediction results; the target frame detection results of each frame image and the target frame fusion prediction results are associated by calculating the Mahalanobis distance between the target frames and adopting the Hungarian algorithm, and the updated target frame state obtained by the updating method in the Kalman filter algorithm is taken as the target tracking results of each frame image; wherein, the calculation of the Mahalanobis distance between the target frames comprises: calculating the predicted covariance matrix of the state according to the target frame fusion prediction results based on the state transition equation of the Kalman filter prediction model; calculating the predicted covariance matrix of the observation according to the predicted covariance matrix of the state and the observation equation; calculating the Mahalanobis distance values between each target frame detection result and each target frame fusion prediction result of each frame image according to the target frame detection results of each frame image, the target frame fusion prediction results and the predicted covariance matrix of the observation; updating all Mahalanobis distance values greater than the distance threshold value to the same preset distance value, and the preset distance value is greater than the distance threshold value; the cost matrix is constructed according to the updated Mahalanobis distance values and is used in the Hungarian algorithm.

2. The image multi-target tracking method based on the fusion of optical flow and Kalman filtering according to claim 1, characterized in that, The target frame to be tracked of each frame image and the corresponding next frame image is obtained according to the target frame detection results of each frame image, which comprises: based on the position change amount of each target frame in each frame image and the previous frame image, the position of each target frame in each frame image is added to the corresponding position change amount to obtain each predicted target frame in the corresponding next frame image of each frame image; if the corresponding target frame does not exist in the previous frame image, the position change amount is 0; the target frame position comprises the x-coordinate and y-coordinate of the upper left corner of the target frame, and the width and height of the target frame; the expansion amount of the target frame is obtained according to the number of layers of the optical flow pyramid and the size of the tracking window; for each target frame in each frame image and each predicted target frame in the corresponding next frame image, the height and width of the target frame are enlarged according to the expansion amount, and the target frame to be tracked is obtained.

3. The image multi-target tracking method based on fusion of optical flow and Kalman filter according to claim 2, characterized in that, The expansion amount E of the target frame is obtained according to the number of layers of the optical flow pyramid and the size of the tracking window by using the following formula: E=2 n ×r wherein, n is the number of layers of the optical flow pyramid, and r is the radius of the tracking window.

4. The image multi-target tracking method based on fusion of optical flow and Kalman filter according to claim 1, characterized in that, The method comprises the following steps of: The method comprises the following steps of: The method comprises the following steps of: The method comprises the following steps of: The method comprises the following steps of:

5. The image multi-target tracking method based on fusion of optical flow and Kalman filter according to claim 4, characterized in that, The method comprises the following steps of: The method comprises the following steps of: The method comprises the following steps of: The system comprises the following modules: An image acquisition module is configured to acquire video images in real time; 6. The image multi-target tracking method based on fusion of optical flow and Kalman filter according to claim 5, characterized in that, A target detection module is configured to detect targets in the video images based on a deep learning model to obtain target frame detection results of each frame of image; An optical flow prediction module is configured to input the target frame detection results of each frame of image into an optical flow prediction model to obtain target frame optical flow prediction results of a next frame of image corresponding to each frame of image; A Kalman filter prediction module is configured to input the target frame detection results of each frame of image into a Kalman filter prediction model to obtain target frame Kalman prediction results of the next frame of image corresponding to each frame of image.

7. An image multi-target tracking system based on fusion of optical flow and Kalman filter, characterized in that, ​ ​ ​ ​ ​ The target tracking module is configured to: perform weighted summation on a target frame box optical flow prediction result and a Kalman prediction result of a next frame image corresponding to each frame image to obtain a target frame box fusion prediction result; perform, on a target frame box detection result of the each frame image and the target frame box fusion prediction result, the following operations: calculating Mahalanobis distance between target frame boxes, establishing association between the target frame box detection result and the target frame box fusion prediction result by using a Hungarian algorithm, and obtaining an updated target frame box state by using an updating method in a Kalman filtering algorithm, so as to obtain a target tracking result of the each frame image.

Images