A real-time multi-target tracking method integrating visual optical flow feature point tracking and motion trend estimation

Abstract

This invention provides a real-time multi-target tracking method for intelligent driving scenarios that integrates visual optical flow feature point tracking and motion trend estimation. This method extracts optical flow features from the target area tracked in the previous frame to predict the target's position in the current frame. Simultaneously, a rectangular box motion trend estimation algorithm is used to predict the rectangular box motion trend. The two predictions are compared to obtain the predicted position of the target in the previous frame in the current frame. The accuracy of target tracking is then determined by comparing the similarity, overlap, and difference between the target detection results and the current frame. Finally, the successfully tracked targets are positionally filtered to improve the stability of the target rectangular box, which is then output to the intelligent driving decision-making module. This invention is primarily used in intelligent driving visual target tracking modules, providing the system with stable and reliable target output.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving visual multi-target tracking, and in particular to a multi-target real-time tracking method that integrates visual optical flow feature point tracking and motion trend estimation. Background Art

[0002] The research on visual target tracking technology is of great significance in the field of intelligent driving perception. Through this technology, the motion state and trend of the target can be estimated, thereby calculating the target's motion direction and speed, providing a basis for the intelligent driving decision-making module.

[0003] Visual object tracking technology currently has a substantial research foundation. Based on the algorithm type, it can be categorized into techniques such as correlation filtering, optical flow, motion state estimation, and deep learning. Early object tracking algorithms included Median Flow and Kalman filtering. Correlation filtering-based algorithms include MOSSE, CSK, and KCF. In recent years, deep learning-based object tracking algorithms, such as ECO, MDNet, SANet, and DeepSORT, have become increasingly popular.

[0004] However, in applications in the field of intelligent driving, due to the rapid movement of vehicles and complex scenes, how to solve problems such as target occlusion, rapid movement, and light changes is a major difficulty. At the same time, the large number of targets, high real-time requirements, and limited chip computing power also put higher demands on the performance of tracking algorithms.

[0005] Currently, many algorithms fail to meet actual needs during deployment. For example, algorithms such as Median Flow and KCF are designed for single-target tracking. When expanded to multi-target tracking, the algorithms cannot achieve real-time performance. Deep learning-based algorithms require separately labeled video tracking datasets, which is very costly and does not meet expected performance.

[0006] Therefore, there is an urgent need for a multi-target real-time tracking algorithm that can be applied to intelligent driving scenarios. Summary of the Invention

[0007] In order to solve the above-mentioned defects in the prior art, the purpose of the present invention is to provide a multi-target real-time tracking method that integrates visual optical flow feature point tracking and motion trend estimation, aiming to solve the problem of complex scene tracking, improve tracking accuracy and algorithm operation efficiency, and make tracking results smoother.

[0008] According to a first aspect of the present invention, a multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation is provided, characterized by comprising:

[0009] Step 10: Obtain the image M of the current i-th frame in real time through the image acquisition device i, use the YOLO target detection algorithm to detect M i The target in the i-th frame is obtained, and the rectangular frame position of the target is obtained. Each target in the i-th frame is added to the target detection result list Objects of the i-th frame. i .

[0010] Step 20: If the i-th frame is the first frame, i For each target in the target, configure a new ID, set the target first parameter tracked_frames to 1, set the target second parameter patched_frames to 0, and set Objects i Assigned to the target tracking result list of the current i-th frame Jump to step 70; if the i-th frame is not the first frame, continue to execute step 30.

[0011] Step 30: List of target tracking results for frame i-1 Each target is processed using the rectangular box motion trend estimation algorithm to obtain the target motion state prediction position list of the current frame Use the rectangular frame area image optical flow feature point tracking algorithm to obtain the target optical flow prediction position list of the current frame

[0012] Step 40: Contrast and The position of the rectangular box of the target j with the same ID is calculated, and the first rectangular box overlap IoU1 is calculated. The threshold value is preset according to the first overlap Judge, if The target is tracked successfully and the target is added to the target position tracking prediction result list of the target in the i-1 frame in the i frame. And add 1 to the target's patched_frames; if If target tracking fails, set tracked_frames of the target to 0.

[0013] Step 50: Objects i Each target in The matching degree of each target in is calculated, including the second overlap matching degree IoU2, the normalized center point distance Dist, the difference Diff, and the similarity SimM of the image area. If the matching degree meets the hyperparameter threshold, IoU2 is used as the weight of the matching degree of the two targets. If the matching degree does not meet the hyperparameter threshold, the weights corresponding to the matching degrees of the two targets are set to 0, and the matching degree weight matrix is ​​constructed based on this weight.

[0014] Step 60: According to the matching weight matrix, the KM optimal matching algorithm is used to obtain the matching relationship between the targets in the i-1 frame and the i-th frame. For the successfully matched targets, the ID of the target in the i-1 frame is assigned to the matching target in the i-th frame, tracked_frames is increased by 1, and patched_frames is set to 0; for the unmatched targets in the i-th frame, a new ID is assigned, tracked_frames is set to 1, and patched_frames is set to 0; for the unmatched targets in the i-1 frame, the ID remains unchanged and patched_frames is increased by 1; each target is added List.

[0015] Step 70: Right Analyze and judge the target in the target, if patched_frames is greater than the maximum target prediction frame number Pn, then the target is removed from Remove it and recycle its ID; if patched_frames is less than the maximum target prediction frame number Pn, and tracked_frames is greater than the minimum target tracking frame number Tn, then the target is removed from Remove and add List.

[0016] Step 80: Right Each target is filtered using the rectangular frame smoothing filter algorithm and the filtered result is output. The target tracking of the current i-th frame is completed; The target in is added to the cache list for tracking in the next frame; and the process returns to step 10 to process the next frame image.

[0017] Furthermore, the multi-target real-time tracking method provided by the present invention that integrates visual optical flow feature point tracking and motion trend estimation is characterized in that the rectangular frame of the target includes the coordinates (x, y) of the upper left corner of the rectangular frame area of ​​the object in the image as well as the width w and height h.

[0018] Furthermore, the multi-target real-time tracking method provided by the present invention that integrates visual optical flow feature point tracking and motion trend estimation is characterized in that the rectangular frame motion trend estimation algorithm includes: The rectangular box position of target j in Use the Kalman filter for tracking prediction, predict the rectangular frame position of the target in the current frame based on the historical rectangular frame position of the target, and calculate the predicted motion state position of target j And add to the list

[0019] The optical flow feature point tracking algorithm for the rectangular frame area image includes: i and M i-1 Construct a grayscale image pyramid. The rectangular box position of target j in Internal uniform selection M i-1 The K coordinate points in M ​​are calculated using the LK optical flow point matching algorithm based on the grayscale image pyramid. i The position in M, count all corresponding points i and M i-1 The position offset between them is averaged as the optical flow tracking offset of target j. Calculate the optical flow tracking results of target j for And add to the list

[0020] Furthermore, the multi-target real-time tracking method provided by the present invention that integrates visual optical flow feature point tracking and motion trend estimation is characterized in that for the rectangular frame R a (x a ,y a ,w a ,h a ) and rectangular frame R b (x b ,y b ,w b ,h b ), the rectangular box overlap is: IoU = intersection area of ​​two rectangular boxes / union area of ​​two rectangular boxes.

[0021] The normalized center point distance is: in: Rectangular frame R a and R b The center point coordinates of W m Indicates the image width, H m Indicates the image height.

[0022] The difference is: Diff = abs(log(w a / w b ))+abs(log(h a / h b )), w a ,h a ,w b ,h b Rectangular frame R a and R b width and height.

[0023] The calculation of the image region similarity SimM includes: a and M b The RGB channels are separated, the color histogram is statistically analyzed, and normalized to obtain Among them, S {a,b} Represents the rectangular area image M a or M b The pixel area, Represents image M a or M b The pixel value at position (i, j). For the color histogram vector and The cosine similarity formula is used to calculate the similarity of the image areas where the two rectangular boxes are located: SimM=V a `V b / (||V a ||×||V b ||), where V a `V b Represents the dot product of two vectors, ||V a || and ||V b || represents the modulus of two vectors respectively.

[0024] Furthermore, the multi-target real-time tracking method provided by the present invention that integrates visual optical flow feature point tracking and motion trend estimation is characterized in that the matching degree satisfies the hyperparameter threshold, including: Dist <thresh dist 、Diff <thresh diff 、SimM>thresh sim ,in Preset threshold for the second overlap, thresh dist Preset threshold for center point distance, thresh diff Preset threshold for difference, thresh sim Preset threshold for similarity.

[0025] Furthermore, the multi-target real-time tracking method provided by the present invention that integrates visual optical flow feature point tracking and motion trend estimation is characterized in that the rectangular frame smoothing filter algorithm is:

[0026] The rectangular box positions of target j from the i-2th frame to the i-th frame are respectively recorded as R i-2,j (x i-2,j ,y i-2,j ,w i-2,j ,h i-2,j ), R i-1,j (x i-1,j ,y i-1,j ,w i-1,j ,h i-1,j ), R i,j (x i,j ,y i,j ,w i,j ,h i,j); The positions of the rectangular frames after filtering from the i-3th frame to the i-1th frame are respectively recorded as R' i-3,j (x' i-3,j ,y' i-3,j ,w' i-3,j ,h' i-3,j ), R' i-2,j (x' i-2,j ,y' i-2,j ,w' i-2,j ,h' i-2,j )、R, i-1,j (x' i-1,j ,y' i-1,j ,w' i-1,j ,h, i-1,j ).

[0027] The four parameters of the rectangular frame are filtered using a 2nd-order Butterworth low-pass filter:

[0028]

[0029] Among them, (a1, a2, a3) and (b1, b2, b3) are the parameters of the Butterworth low-pass filter, which are calculated by setting the sampling frequency and cutoff frequency. The target rectangular box filtering result is obtained: R' i,j (x' i,j ,y' i,j ,w' i,j ,w' i,j ).

[0030] Furthermore, the multi-target real-time tracking method provided by the present invention that integrates visual optical flow feature point tracking and motion trend estimation is characterized in that the image acquisition device is installed on a vehicle, and the perception area covers the front of the vehicle; the targets include: vehicles, pedestrians, and non-motor vehicles.

[0031] According to a second aspect of the present invention, there is provided a computer device, comprising:

[0032] a memory for storing instructions; and

[0033] The processor is configured to call the instructions stored in the memory to execute the multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation according to the first aspect.

[0034] According to a third aspect of the present invention, a computer-readable storage medium is provided, storing instructions, which, when executed by a processor, execute the multi-target real-time tracking method that integrates visual optical flow feature point tracking and motion trend estimation according to the first aspect.

[0035] Compared with the prior art, the above technical solution conceived by the present invention has at least the following beneficial effects:

[0036] Combining the Kalman state filter with the optical flow tracking algorithm, the position of the target in the previous frame in the current frame is predicted, which makes up for the problem that a single predictor cannot cover complex scenes.

[0037] To address the problem of calculating the similarity of targets in previous and next frames, the present invention evaluates the problem from two dimensions. First, an evaluation algorithm is designed for the overlap, center point distance, and size of the rectangular frames. Second, a cosine similarity evaluation algorithm based on color histogram is proposed for the image similarity in the rectangular frame area. This allows for a more comprehensive calculation of target similarity and improves target tracking accuracy.

[0038] In the target bipartite graph matching problem, compared with the Hungarian matching algorithm commonly used in the industry, this algorithm can only achieve maximum matching. The KM matching algorithm adopted in this invention further considers the matching weight on the basis of the Hungarian algorithm to achieve optimal matching.

[0039] The target tracking algorithm implemented by the present invention solves the problem of target rectangular frame jitter and proposes a rectangular frame filtering algorithm based on a Butterworth low-pass filter to make the target tracking result smoother.

[0040] The present invention ultimately achieves an operating efficiency of 30 frames per second in an embedded system and can track up to 64 targets; the tracking performance is advantageous.

[0041] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0043] Figure 1 It is a flowchart according to an exemplary embodiment.

[0044] Figure 2 FIG. 4 is a schematic diagram of optimal bipartite matching of a rectangular frame according to an exemplary embodiment. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0046] Algorithm notation description:

[0047] Initialize the target label array, recorded as ListID; the list adopts a queue structure.

[0048] The current frame is recorded as i; the current frame image is recorded as M i ; The current target detection result is recorded as Objects i ; The current target tracking result is recorded as

[0049] The previous frame image is recorded as M i-1 , the target tracking result of the previous frame is recorded as The target motion state prediction result of the previous frame is recorded as The target optical flow prediction result of the previous frame is recorded as The predicted position of the target in the previous frame in the current frame is recorded as

[0050] Current target tracking results use express.

[0051] Each tracking target contains parameters: target label id, target tracking frame number tracked_frames, target prediction frame number patched_frames, and target rectangular frame coordinates R (x, y, w, h).

[0052] The parameter Tn represents the minimum target tracking frame number, and the parameter Pn represents the maximum target prediction frame number.

[0053] like Figure 1 As shown, in one embodiment, the multi-target real-time tracking method provided by the present invention that integrates visual optical flow feature point tracking and motion trend estimation consists of the following steps:

[0054] 1) System Setup: Install the camera on the vehicle's front windshield, covering the forward sensing area, connect it to the controller via a video transmission cable, and power the entire system.

[0055] 2) System initialization: The system starts, loads the driver, and performs a hardware function self-test. If a hardware failure occurs, an alarm is sounded and the system exits. If the system self-test is normal, proceed to the next step.

[0056] 3) The algorithm collects the camera image of the current frame i in real time, denoted as M i , and use the YOLO target detection algorithm to detect targets in the image, including vehicles, pedestrians, and non-motor vehicles, recorded as Objects i ;

[0057] 4) If the current frame is the first frame, assign the target detection result to the current tracking result Go to step 10); if not, go to the next step;

[0058] 5) Tracking results for frame i-1 The rectangular box motion trend estimation algorithm (see the key module of the algorithm (4) Rectangular box motion trend estimation algorithm) is used to obtain the predicted motion state position of each target in the current frame, which is recorded as

[0059] 6) Tracking results for frame i-1 Use the rectangular frame area image optical flow feature point tracking algorithm (see Section 5 Key Modules of the Algorithm Section 5, Section 5, Rectangular Frame Area Image Optical Flow Feature Point Tracking Algorithm) to obtain the optical flow predicted position of each target in the current frame, denoted as

[0060] 7) Comparison and The position of the rectangular box corresponding to the target in the list is used to calculate the overlap IoU. If Then add the target List, and increase the number of patched_frames parameters of each target by 1, otherwise the target tracking fails;

[0061] 8) Objects detected for the i-th frame i Each target in the i-1 frame is tracked and predicted respectively For each target in the algorithm, the rectangular box matching degree is calculated (see the key module of the algorithm (1) rectangular box matching degree calculation) and the image region similarity SimM where the rectangular box is located (see the key module of the algorithm (2) rectangular box region image similarity calculation). If the matching degree between the two rectangular boxes satisfies: Dist <thresh dist 、Diff <thresh diff 、SimM>thresh sim , then the matrix is ​​constructed based on IoU as weight;

[0062] 9) Using the weight matrix, execute the rectangular box optimal bipartite matching algorithm (see the key module of the algorithm (6) rectangular box optimal bipartite matching algorithm) to obtain the corresponding relationship; for the target that is successfully matched, assign the corresponding target ID in the i-1 frame to the target of the current i-th frame, increase the target parameter tracked_frames by 1, and set patched_frames to 0; for the target that is not matched in the i-th frame, assign a new ID in the target label array ListID, set the target parameter tracked_frames to 1, and set patched_frames to 0; for the target that is not matched in the i-1 frame, the target label remains unchanged, and patched_frames increases by 1; add these three targets List;

[0063] 10) Yes Analyze and judge the targets in the list. If patched_frames is greater than the maximum target prediction frame number Pn, the target is removed from the list and its id is recycled to the target label list ListID; if patched_frames is less than the maximum target prediction frame number Pn, and tracked_frames is greater than the minimum target tracking frame number Tn, the target is removed and join list; Cache the targets in the cache for subsequent target tracking calculations;

[0064] 11) Yes Each target in the list is filtered using a rectangular frame smoothing filter algorithm (see the key module of the algorithm (3) Rectangular frame smoothing filter algorithm) to reduce the jitter of the rectangular frame, and the filtered result is added to the cache list for tracking in the next frame; at the same time, the target tracking result is output to the intelligent driving decision module for decision-making, and the target tracking of the current frame is completed.

[0065] 12) Return to step 3).

[0066] The key algorithm modules involved in the above steps include the following parts:

[0067] (1) Rectangular box matching calculation: Rectangular box R a (x a ,y a ,w a ,h a ) and rectangular frame R b (x b ,y b ,w b ,h b ) are calculated using three methods:

[0068] Overlap IoU = intersection area of ​​two rectangular boxes / union area of ​​two rectangular boxes.

[0069] Normalized center point distance in: Rectangular frame R a and R b The center point coordinates of W m Indicates the image width, H m Indicates the image height.

[0070] Difference Diff = abs(log(w a / w b ))+abs(log(h a / hb )).

[0071] (2) Calculation of image similarity SimM in the rectangular area:

[0072] For the rectangular area image M a and M b The RGB channels are separated, the color histogram is statistically analyzed, and normalized. The calculation formula is as follows:

[0073]

[0074] Among them S {a,b} Represents the rectangular area image M a or M b The pixel area, Represents image M a or M b The pixel value at position (i, j).

[0075] Corresponding to forming two color histogram vectors of the same dimension and The cosine similarity formula is used to calculate the similarity between two rectangular images:

[0076] SimM=V a ·V b / (||V a ||×||V b ||)

[0077] Where V a ·V b Represents the dot product of two vectors, ||V a || and ||V b || represents the modulus of two vectors respectively.

[0078] (3) Rectangular frame smoothing filter algorithm: To make the target tracking rectangle more stable, the present invention uses a smoothing filter algorithm to adjust the position of the target rectangle after tracking. The current frame is recorded as i, and the target labeled j is smoothed by filtering. The steps are as follows:

[0079] The target rectangle position from the i-2th frame to the i-th frame is recorded as R i-2,j (x i-2,j ,y i-2,j ,w i-2,j ,h i-2,j ), R i-1,j (x i-1,j ,y i-1,j ,w i-1,j ,h i-1,j ), R i,j (x i,j ,y i,j,w i,j ,h i,j ); The positions of the rectangular frames after filtering from the i-3th frame to the i-1th frame are respectively recorded as R' i-3,j (x' i-3,j ,y' i-3,j ,w' i-3,j ,h' i-3,j ), R' i-2,j (x' i-2,j ,y' i-2,j ,w' i-2,j ,h' i-2,j )、R, i-1,j (x' i-1,j ,y' i-1,j ,w' i-1,j ,h, i-1,j ).

[0080] The four parameters of the rectangular frame are filtered using a second-order Butterworth low-pass filter. The calculation formula is as follows:

[0081]

[0082] Where (a1, a2, a3) and (b1, b2, b3) are the parameters of the Butterworth low-pass filter, which are calculated by setting the sampling frequency and cutoff frequency.

[0083] Return and save the target rectangle filtering result: R, i,j (x' i,j ,y' i,j ,w' i,j ,w' i,j ).

[0084] (4) Rectangular frame motion trend estimation algorithm: In order to improve the accuracy of target tracking, the present invention predicts the rectangular frame position of each target and adopts a 4-dimensional Kalman state filter, including the (x, y, w, h) of the rectangular frame. Based on the historical rectangular frame position of the target, the rectangular frame position of the target in the current frame is predicted.

[0085] (5) Optical flow feature point tracking algorithm for rectangular frame area images: The present invention constructs a grayscale pyramid for the two frames of images and calculates the position of the rectangular frame area image of the previous frame in the current frame through the LK optical flow tracking algorithm, thereby achieving optical flow tracking. The algorithm steps are as follows:

[0086] Construct grayscale pyramids for the two frames of images respectively.

[0087] Target list for the previous frame K points are uniformly selected from the rectangular area of ​​each target in .

[0088] Use the LK optical flow point matching algorithm to calculate the positions of these K points in the current frame image.

[0089] Delete the points that failed to match, and calculate the horizontal and vertical offsets of the two frames before and after for the remaining points.

[0090] The offset is counted and the median is taken as the target offset for target optical flow tracking; and the position of the target rectangle in the current frame is calculated.

[0091] Add the optical flow tracking results to List and return.

[0092] (6) Rectangular box optimal bipartite matching algorithm: For the target matching problem between rectangular box list A and rectangular box list B, the present invention converts it into a bipartite graph matching problem and uses the KM optimal matching algorithm to calculate the optimal matching result.

[0093] First, calculate the IoU overlap between the rectangles in the two lists.

[0094] The overlap degree is used as the weight to construct the weight matrix.

[0095] The weight matrix is ​​calculated using the KM algorithm to obtain the matching relationship between the rectangular boxes in list A and list B under the optimal matching condition.

[0096] The successfully matched rectangles are considered to be the same target and the matching results are returned.

[0097] like Figure 2 As shown in the figure, there are 3 targets in list A and 4 targets in list B. For each target rectangle in the list, the overlap IoU is calculated, and the value range is (0-1). The weight matrix is ​​obtained, and then the KM optimal bisection matching algorithm is used to obtain the matching result. After calculation, Figure 2 A1, A2, and A3 in the list are matched successfully with B1, B2, and B4 in list B respectively.

[0098] Specifically, in some embodiments, the steps of implementing the invention are:

[0099] 1) System Setup: Install the camera on the vehicle's front windshield, covering the forward sensing area, connect it to the controller via a video transmission line, and power the entire system.

[0100] 2) System initialization: The system starts, loads the driver, and performs a hardware function self-test. If a hardware failure occurs, an alarm will be issued and the system will exit; if the system self-test is normal, proceed to the next step.

[0101] 3) Real-time acquisition of the camera image of the current frame i, denoted as M iThe YOLO target detection algorithm is used to detect targets in the image. Each target position contains the pixel coordinates x, y, width and height w, h of the upper left corner of the rectangular area of ​​the object in the image. The target categories include: vehicles, pedestrians, and non-motor vehicles. The recognition results are recorded as Objects i .

[0102] 4) If the current frame i is the first frame, initialize the target label queue ListID, the queue length is 128, the queue data range is 0 to 127, the data input and output in the queue adopt the first-in-first-out principle for the detection result Objects i Assign an ID to each target in the target and set the target parameters tracked_frames to 1 and patched_frames to 0, and put the target into the list Go to step 12); if the current frame i is not the first frame, go to the next step.

[0103] 5) Take out the target tracking result of frame i-1 from the cache For each tracking target j, the rectangular box position Use the Kalman filter to perform tracking prediction and obtain the predicted motion state position of each target in the current frame And add to the list This module is named Rectangular Box Motion Trend Estimation Algorithm.

[0104] 6) Take out the i-1th frame image M from the cache i-1 , and respectively for the image M i and M i-1 Build a grayscale image pyramid and then track the result of frame i-1 The rectangular box position of each tracking target j in Take K coordinate points in the row and column directions respectively, and use the LK optical flow point matching algorithm based on the grayscale image pyramid to calculate these K points. 2 The position of each point in the current frame image, and the median of the position offset and distance offset between the corresponding points is recorded as The optical flow tracking result of each tracking target in the current frame is calculated using the following formula: And add to the list This module is named the rectangular frame area image optical flow feature point tracking algorithm:

[0105]

[0106] 7) Comparison and The position of the rectangular box corresponding to the target in the list is used to calculate the overlap IoU between the two. If Then add the target List, and add 1 to the patched_frames parameter value of each target, otherwise the target tracking fails and the tracked_frames parameter is set to 0.

[0107] 8) Detection results of current frame i Objects i Each target is tracked and predicted by the i-1 frame The matching degree of the rectangular box of each target in is calculated, including the overlap IoU, the normalized center point distance Dist, and the difference Diff. The calculation formula is as follows:

[0108] a.IoU = intersection area of ​​two rectangular boxes / union area of ​​two rectangular boxes.

[0109] b. in: Rectangular frame R a and R b The center point coordinates of W m Indicates the image width, H m Indicates the image height.

[0110] c.Diff=abs(log(w a / w b ))+abs(log(h a / h b )).

[0111] 9) Recalculate Objects i Each target is respectively The similarity SimM of the image area where the target is located is calculated in two steps:

[0112] a. For two rectangular frame area images M a and M b The RGB channels are separated, the color histogram is statistically analyzed, and normalized. The calculation formula is as follows:

[0113]

[0114] Among them S {a,b} Represents the rectangular area image M a or M b The pixel area, Represents image M a or M b The pixel value at position (i, j).

[0115] b. Corresponding to forming two color histogram vectors of the same dimension and The cosine similarity formula is used to calculate the similarity between two rectangular images:

[0116] SimM=V a ·V b / (||V a ||×||V b ||)

[0117] Where V a ·V b Represents the dot product of two vectors, ||V a || and ||V b || represents the modulus of two vectors respectively.

[0118] 10) If Objects i and The matching degree between the targets in satisfies the hyperparameter threshold: Dist <thresh dist 、Diff <thresh diff 、SimM>thresh sim , then the IoU is used as the weight to construct the matrix; if the hyperparameter threshold is not met, the weight in the matrix is ​​set to 0.

[0119] 11) For the weight matrix, use the KM optimal bipartite matching algorithm to calculate Objects i and The correspondence between the targets; for the successfully matched target, the i-1 frame The corresponding target id is assigned to the Objects in the current i-th frame i For the target, the target parameter tracked_frames is increased by 1, and patched_frames is set to 0; for Objects i For unmatched targets, assign a new ID to the target number array ListID, set the target parameter tracked_frames to 1, and patched_frames to 0; For unmatched targets, the target number remains unchanged and patched_frames increases by 1. Add these three targets to List.

[0120] 12) Clear List, yes Analyze and judge the targets in the list. If patched_frames is greater than the maximum target prediction frame number Pn, the target is removed from the list and its id is recycled to the target label queue ListID; if patched_frames is less than the maximum target prediction frame number Pn, and tracked_frames is greater than the minimum target tracking frame number Tn, the target is removed and join and List; if patched_frames is less than the maximum target prediction frame number Pn, and tracked_frames is less than the minimum target tracking frame number Tn, then the target is added to List, used for subsequent target tracking calculations.

[0121] 13) Yes Each target in the list is filtered using a rectangular frame smoothing filter algorithm to reduce the jitter of the rectangular frame. The calculation steps are as follows:

[0122] a. The original position of the rectangular frame of the target numbered j from the i-2th frame to the i-th frame is recorded as R i-2,j (x i-2,j ,y i-2,j ,w i-2,j ,h i-2,j ), R i-1,j (x i-1,j ,y i-1,j ,w i-1,j ,h i-1,j ), R i,j (x i,j ,y i,j ,w i,j ,h i,j ); The positions of the rectangular frames after filtering from the i-3th frame to the i-1th frame are respectively recorded as R, i-3,j (x' i-3,j ,y' i-3,j ,w' i-3,j ,h, i-3,j )、R, i-2,j (x' i-2,j ,y' i-2,j ,w' i-2,j ,h, i-2,j )、R, i-1,j (x' i-1,j ,y' i-1,j ,w' i-1,j ,h, i-1,j ).

[0123] b. Filter the four parameters of the rectangular frame using a second-order Butterworth low-pass filter. The calculation formula is as follows:

[0124]

[0125] Where (a1, a2, a3) and (b1, b2, b3) are the parameters of the Butterworth low-pass filter, which are calculated by setting the sampling frequency and cutoff frequency.

[0126] c. Return and save the target rectangular box filtering result: R,i,j (x' i,j ,y' i,j ,w' i,j ,w' i,j ), and update The target rectangle position in .

[0127] 14) The target tracking list is added to the cache for tracking in the next frame and output to the intelligent driving decision module for decision-making. Target tracking in the current frame is completed.

[0128] 15) Return to step 3).

[0129] It should be understood that the present invention is not limited to the exact construction described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation, characterized in that: include: Step 10: Obtain the image M of the current i-th frame in real time through the image acquisition device i , use the YOLO target detection algorithm to detect M i The target in the i-th frame is obtained, and the rectangular frame position of the target is obtained. Each target in the i-th frame is added to the target detection result list Objects of the i-th frame. i ; Step 20: If the i-th frame is the first frame, i For each target in the target, configure a new ID, set the target first parameter tracked_frames to 1, set the target second parameter patched_frames to 0, and set Objects i Assigned to the target tracking result list of the current i-th frame Jump to step 70; If the i-th frame is not the first frame, continue to step 30; Step 30: List of target tracking results for frame i-1 Each target is processed using the rectangular box motion trend estimation algorithm to obtain a list of target motion state prediction positions in the current frame. Use the rectangular frame area image optical flow feature point tracking algorithm to obtain the target optical flow prediction position list of the current frame Step 40: Contrast and The position of the rectangular box of the target j with the same ID is calculated, and the first rectangular box overlap IoU1 is calculated. The threshold value is preset according to the first overlap Judge, if The target is tracked successfully and the target is added to the target position tracking prediction result list of the target in the i-1 frame in the i frame. And add 1 to the target's patched_frames; if If target tracking fails, the tracked_frames of the target is set to 0; Step 50: Objects i Each target in Calculate the matching degree of each target in, including the second overlap matching degree IoU2, the normalized center point distance Dist, the difference degree Diff, and the similarity of the image area SimM. If the matching degree meets the hyperparameter threshold, IoU2 is used as the weight of the matching degree of the two targets. If the matching degree does not meet the hyperparameter threshold, the weights corresponding to the matching degrees of the two targets are set to 0, and the matching degree weight matrix is ​​constructed based on this weight; Step 60: According to the matching weight matrix, the KM optimal matching algorithm is used to obtain the matching relationship between the targets in the i-1 frame and the i-th frame. For the successfully matched targets, the ID of the target in the i-1 frame is assigned to the matching target in the i-th frame, tracked_frames is increased by 1, and patched_frames is set to 0; for the unmatched targets in the i-th frame, a new ID is assigned, tracked_frames is set to 1, and patched_frames is set to 0; for the unmatched targets in the i-1 frame, the ID remains unchanged and patched_frames is increased by 1; each target is added List; Step 70: Right Analyze and judge the target in the target, if patched_frames is greater than the maximum target prediction frame number Pn, then the target is removed from Remove it and reclaim its ID; If patched_frames is less than the maximum target prediction frame number Pn, and tracked_frames is greater than the minimum target tracking frame number Tn, then the target is removed from Remove and add List; Step 80: Right Each target is filtered using the rectangular frame smoothing filter algorithm and the filtered result is output. The target tracking of the current i-th frame is completed; The target in is added to the cache list for tracking in the next frame; and the process returns to step 10 to process the next frame image.

2. The multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation according to claim 1 is characterized in that: The rectangular frame of the target includes the coordinates (x, y) of the upper left corner of the rectangular frame area of ​​the object in the image as well as the width w and height h.

3. The multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation according to claim 2 is characterized in that: The rectangular frame motion trend estimation algorithm includes: The rectangular box position of target j in Use the Kalman filter for tracking prediction, predict the rectangular frame position of the target in the current frame based on the historical rectangular frame position of the target, and calculate the predicted motion state position of target j And add to the list The optical flow feature point tracking algorithm for the rectangular frame area image includes: i and M i-1 Construct a grayscale image pyramid. The rectangular box position of target j in Internal uniform selection M i-1 The K coordinate points in M ​​are calculated using the LK optical flow point matching algorithm based on the grayscale image pyramid. i The position in M, count all corresponding points i and M i-1 The position offset between them is averaged as the optical flow tracking offset of target j. Calculate the optical flow tracking results of target j for And add to the list 4. The multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation according to claim 3 is characterized in that: For the rectangular box R a (x a ,y a ,w a ,h a ) and rectangular frame R b (x b ,y b ,w b ,h b ), the rectangular box overlap is: IoU = intersection area of ​​two rectangular boxes / union area of ​​two rectangular boxes; The normalized center point distance is: in: Rectangular frame R a and R b The center point coordinates of W m Indicates the image width, H m Indicates the image height; The difference is: Diff = abs(log(w a / w b ))+abs(log(h a / h b )), w a ,h a ,w b ,h b Rectangular frame R a and R b width and height; The calculation of the image region similarity SimM includes: For two rectangular frame area images M a and M b The RGB channels are separated, the color histogram is statistically analyzed, and normalized to obtain Among them, S {a,b} Represents the rectangular area image M a or M b The pixel area, Represents image M a or M b The pixel value at position (i, j); For the color histogram vector V a : and V b : The cosine similarity formula is used to calculate the similarity of the image areas where the two rectangular boxes are located: SimM=V a ·V b / (||V a ||×||V b ||), where V a ·V b Represents the dot product of two vectors, ||V a || and ||V b || represents the modulus of two vectors respectively.

5. The multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation according to claim 4 is characterized in that: The matching degree meets the hyperparameter threshold including: Dist <thresh dist 、Diff <thresh diff 、SimM>thresh sim ,in Preset threshold for the second overlap, thresh dist Preset threshold for center point distance, thresh diff Preset threshold for difference, thresh sim Preset threshold for similarity.

6. The multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation according to claim 5 is characterized in that: The rectangular frame smoothing filter algorithm is: The rectangular box positions of target j from the i-2th frame to the i-th frame are respectively recorded as R i-2,j (x i-2,j ,y i-2,j ,w i-2,j ,h i-2,j ), R i-1,j (x i-1,j ,y i-1,j ,w i-1,j ,h i-1,j ), R i,j (x i,j ,y i,j ,w i,j ,h i,j ); The positions of the rectangular frames after filtering from the i-3th frame to the i-1th frame are respectively recorded as R' i-3,j (x' i-3,j ,y' i-3,j ,w' i-3,j ,h' i-3,j ), R' i-2,j (x' i-2,j ,y' i-2,j ,w' i-2,j ,h' i-2,j ), R' i-1,j (x' i-1,j ,y' i-1,j ,w' i-1,j ,h' i-1,j ); The four parameters of the rectangular frame are filtered using a 2nd-order Butterworth low-pass filter: Where (a1, a2, a3) and (b1, b2, b3) are the parameters of the Butterworth low-pass filter, which are calculated by setting the sampling frequency and cutoff frequency; Get the target rectangular frame filtering result: R' i,j (x' i,j ,y' i,j ,w' i,j ,w' i,j ).

7. The multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation according to any one of claims 1 to 6, characterized in that: The image acquisition device is installed on a vehicle, and the sensing area covers the front of the vehicle; the targets include: vehicles, pedestrians, and non-motor vehicles.

8. A computer device, characterized in that: include: a memory for storing instructions; and a processor for calling the instructions stored in the memory to execute the multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that Instructions are stored, and when the instructions are executed by the processor, the multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation according to any one of claims 1 to 7 is executed.

Images