A Vehicle Target Detection and Tracking Method Based on Multi-Source Information Fusion

Abstract

The present invention discloses a vehicle target detection and tracking method based on multi-source information fusion, which includes: Step 1, obtaining a vehicle detection data set, where the vehicle detection data set includes multiple first detection images and multiple second detection images; Step 2, annotating the first detection images and the second detection images; Step 3, performing temporal and spatial synchronous registration on the first detection images and the second detection images; Step 4, using the Gaussian mixture model and Bayesian theory to perform fusion detection on the first detection images and the second detection images to obtain a fused detection result; Step 5, based on the fused detection result, using a detection and tracking algorithm of multi-source information fusion to achieve the final detection and tracking of vehicle targets. The method of the present invention has high detection accuracy while achieving fast detection, and has a good detection and tracking effect on vehicle targets in complex scenarios.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection and tracking, and particularly relates to a vehicle target detection and tracking method based on multi-source information fusion. Background Art

[0002] In order to form a safe, efficient, convenient and green modern transportation system, a large number of video monitoring systems have been deployed on highways in China at present, so as to achieve the purposes of supervising drivers' safe driving, detecting road conditions, monitoring transportation risks, identifying and preventing and controlling. However, in the currently deployed video monitoring systems, videos can only play the role of detection and tracking in good environments. Due to the limitations of videos, they cannot be detected and identified in complex environments such as rain, snow, fog, haze, and dust, resulting in the inability to supervise safe driving and detect road conditions in bad environments. Therefore, it is also necessary to add detection devices with strong penetration, such as millimeter-wave radars, lidar, etc. With the rapid development of computer technology, great progress has been made in technologies such as image recognition, radar detection, target tracking, information fusion, vehicle networking, and vehicle-road coordination. Further research and technology integration based on these technologies will help to break through the technical bottlenecks of road transportation risk monitoring, identification and prevention and control, and achieve rapid identification and effective prevention and control of road operation risks. Integrating radar and video in an integrated manner and giving full play to the advantages of both to solve the detection and tracking of multi-source information fusion is the primary task.

[0003] Videos and radars are basic devices for realizing traffic risk monitoring. The progress of radar and video detection technologies and the development of new technologies such as artificial intelligence have brought greater vitality to the traffic behavior monitoring system of the road transportation network. However, the current detection technologies based on radar and video are still restricted by complex road traffic scenarios, multi-source heterogeneous traffic risks, etc. Occlusion problems, size change problems, and data fusion problems are common, which all restrict the accuracy of all-weather online recognition of traffic behavior and risk identification.

[0004] Therefore, it is urgent to solve the problems of stable tracking and intelligent perception of traffic targets, improve the accuracy of traffic behavior identification, and enhance the effectiveness of identifying road network operation risks. Summary of the Invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a vehicle target detection and tracking method based on multi-source information fusion. The technical problems to be solved by the present invention are realized through the following technical solutions:

[0006] A vehicle target detection and tracking method based on multi-source information fusion, the vehicle target detection and tracking method comprising:

[0007] Step 1: Obtain a vehicle detection data set, where the vehicle detection data set includes multiple first detection images and multiple second detection images. The first detection images are images corresponding to video monitoring devices, and the second detection images are images corresponding to radar devices;

[0008] Step 2: Label the first detection images and the second detection images;

[0009] Step 3: Perform synchronous registration in terms of time and space on the first detection images and the second detection images;

[0010] Step 4: Use the Gaussian mixture model and Bayesian theory to perform fusion detection on the first detection images and the second detection images to obtain a fused detection result;

[0011] Step 5: Based on the fused detection result, use a detection and tracking algorithm for multi-source information fusion to achieve the final detection and tracking of vehicle targets.

[0012] In an embodiment of the present invention, step 2 includes:

[0013] Label the vehicles in the first detection images and the second detection images using bounding boxes, and label the vehicles in the first detection images and the second detection images according to vehicle size, lane position, and appearance order.

[0014] In an embodiment of the present invention, step 3 includes:

[0015] Step 3.1: Perform time registration on the first detection images and the second detection images;

[0016] Step 3.2: Perform spatial registration on the first detection images and the second detection images.

[0017] In an embodiment of the present invention, step 3.1 includes:

[0018] Judge whether the frame numbers and times of the first detection images and the second detection images are consistent. If they are consistent, the time registration is completed.

[0019] In an embodiment of the present invention, step 3.2 includes:

[0020] Step 3.21: Convert the coordinates in the radar device coordinate system to the world coordinate system centered on the video monitoring device;

[0021] Step 3.22: Convert the coordinates of the world coordinate system to the video monitoring device coordinate system;

[0022] Step 3.23: Convert the coordinates of the video monitoring device coordinate system to the image coordinate system.

[0023] In one embodiment of the present invention, step 4 includes:

[0024] Step 4.1: Based on the Gaussian distribution, integrate the position distribution f(x1, y1) of the radar detection result relative to the true value to obtain the probability distribution F(x1, y1), and integrate the position distribution f(x2, y2) of the video detection result relative to the true value to obtain the probability distribution F(x2, y2);

[0025] Step 4.2: Based on Bayes' formula, obtain the true position B of the target according to the probability distribution F(x1, y1) and the probability distribution F(x2, y2) i The probability P(B i │A) relative to the detection result A;

[0026] Step 4.3: Select a preset area, where the preset area includes the target positions in the first detection image and the second detection image, and perform joint probability calculation on each pixel in the preset area and the probability P(B i │A), and output the position with the highest joint probability as the fused detection result.

[0027] In one embodiment of the present invention, step 5 includes:

[0028] Step 5.1: Use the object detection algorithm to obtain the detection target set detections of the k-th frame;

[0029] Step 5.2: Use the data association algorithm to establish the association matrix between the target and the trajectory;

[0030] Step 5.3: Use the tracker template to perform loop detection on the current k-th frame, calculate the maximum response value, determine the target prediction position, and achieve trajectory tracking.

[0031] In one embodiment of the present invention, step 5.3 includes:

[0032] Step 5.31: Initialize the KCF tracker;

[0033] Step 5.32: Update the target position;

[0034] Step 5.33: Update the tracker template.

[0035] In one embodiment of the present invention, after step 5.3, it further includes:

[0036] Step 5.4: Use Kalman filtering to predict the position of the occluded target.

[0037] In one embodiment of the present invention, step 5.4 includes:

[0038] Step 5.41: Predict the state of the k-th frame using the state value of the (k - 1)-th frame;

[0039] Step 5.42: Calculate the Kalman gain K k ;

[0040] Step 5.43: Update the prediction result with the observation value Z k Perform weighted averaging on the prediction result and the observation result to obtain the state estimation at the current moment. Meanwhile, update the covariance P k .

[0041] Advantages of the present invention:

[0042] 1. The present invention uses a multi-source information fusion method to make a more accurate estimation of vehicle target detection in different scenarios, enhancing the detection and tracking ability of vehicle targets in complex scenarios; for video radars at different distances and in different weather conditions, their advantages and disadvantages are complemented. In the area with the advantages of video monitoring, a fusion detection strategy mainly based on video detection information is formulated; in the area with the advantages of radar monitoring, a fusion detection strategy mainly based on radar detection information is formulated; in the area with the common advantages of radar and video, a balanced fusion detection strategy is formulated.

[0043] 2. After obtaining the detection and tracking results of the radar and the camera respectively, in order to achieve the fusion detection and tracking of the radar and the camera, a radar-video integrated detection and tracking fusion algorithm based on the Gaussian mixture model and Bayesian theory is designed. After obtaining the distribution of the true value relative to the detection result and combining the prediction information of the true value by Kalman filtering, the target position after the fusion of radar and video information can be obtained.

[0044] 3. For the problems of missed detection and occlusion existing in the acquired data frames, we respectively adopt KCF prediction and tracking, and use Kalman filtering for occlusion, achieving complete vehicle target detection and tracking with good results.

[0045] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Description of the Drawings

[0046] Figure 1 is a flowchart of a vehicle target detection and tracking method based on multi-source information fusion provided by an embodiment of the present invention;

[0047] Figure 2 is a time synchronization flowchart provided by an embodiment of the present invention;

[0048] Figure 3 is a space synchronization flowchart provided by an embodiment of the present invention;

[0049] Figure 4 is a radar space coordinate conversion plan view provided by an embodiment of the present invention;

[0050] Figure 5 is an analysis diagram of the detection situation of two-source data provided by an embodiment of the present invention;

[0051] Figure 6 is a schematic diagram of two-source data fusion provided by an embodiment of the present invention;

[0052] Figure 7 is a fusion detection and tracking flow chart provided by an embodiment of the present invention;

[0053] Figure 8 is a schematic diagram of the KCF tracking process provided by an embodiment of the present invention. Specific Embodiments

[0054] The following further describes the present invention in detail with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0055] Embodiment 1

[0056] Please refer to Figure 1 , Figure 1 is a flow schematic diagram of a vehicle target detection and tracking method based on multi-source information fusion provided by an embodiment of the present invention. An embodiment of the present invention provides a vehicle target detection and tracking method based on multi-source information fusion, and the vehicle target detection and tracking method based on multi-source information fusion includes:

[0057] Step 1: Obtain a vehicle detection data set, which includes multiple first detection images and multiple second detection images. The first detection images are images corresponding to video monitoring devices, and the second detection images are images corresponding to radar devices.

[0058] Specifically, traffic data of road vehicles in different scenarios and weather conditions are collected to construct a vehicle detection data set, ensuring sufficient vehicle detection data in complex weather to study various targets.

[0059] Step 1.1: Install a road monitoring device and a millimeter-wave radar at the same location to collect vehicles on the same road. That is, the road traffic vehicle data is collected by using the road monitoring device (i.e., the video monitoring device) and the millimeter-wave radar. The amount of road video data finally collected in this embodiment reaches 2T, and the video data covers various complex weather conditions such as sunny days, rainy days, foggy days, and snowy days. In terms of time, it also includes two time periods: day and night.

[0060] Step 1.2: Convert the video data collected by the video monitoring device and the radar device into image data, namely the first detection image and the second detection image. The obtained image resolution is, for example, 1920×1080.

[0061] Step 1.3: Subdivide according to the day, night, and different weather conditions of the traffic video, and compare the detection and tracking effects under different weather conditions. In this embodiment, a total of 12,000 different data are obtained.

[0062] Step 2: Label the first detection image and the second detection image.

[0063] Specifically, label the vehicles in each image to obtain the label data corresponding to each image, which is convenient for subsequent use in tracking and detection. During the labeling process, it is necessary to closely fit the labeling box to the edge of the vehicle, and classify different vehicle types in detail, and equip different symbols, which are divided into large, medium, and small (bms), left and right lanes (lr), and the order of appearance (1, 2, 3....), such as the first small car in the left lane (sl1). The finally framed area is the position of the real trajectory point of the vehicle.

[0064] In this embodiment, the Labellmg software is used for labeling. When labeling the vehicle image, an xml format file will be generated, and the xml file will contain information such as the frame number, vehicle ID, vehicle type, and the obtained pixel center point position corresponding to the target image.

[0065] Step 3: Perform time and space synchronization registration on the first detection image and the second detection image.

[0066] Step 3.1: Perform time registration on the first detection image and the second detection image.

[0067] Specifically, judge whether the frame numbers and times of the first detection image and the second detection image are the same. If they are the same, the time registration is completed. If they are not the same, adjust the frame numbers to be the same to complete the time registration.

[0068] In this embodiment, the time for the video monitoring device and the radar device to obtain signals can correspond to each other. For example, in this embodiment, the millimeter-wave radar device specification is 100ms, the scanning frequency is 10Hz, and the frame rate of the selected video monitoring device is 10fps. Therefore, the time for the video monitoring device and the radar device to obtain signals can correspond to each other. Among them, the data obtained by the radar device contains the frame number and the time mark of each group, that is, it contains hours, minutes, seconds, and milliseconds, and the video sequence collected by the video monitoring device is named by this time. In this way, time fusion only needs to judge whether the frame number and time are the same. For example Figure 2As shown, the next step can be carried out only when the frame number and time are the same. Otherwise, adjust according to the frame number until the condition is met.

[0069] Step 3.2: As Figure 3 shown, perform spatial registration on the first detection image and the second detection image.

[0070] Step 3.21: Convert the coordinates in the radar device coordinate system (i.e., the millimeter-wave coordinate system) to the world coordinate system centered on the video monitoring device (i.e., the camera);

[0071] Step 3.22: Convert the coordinates of the world coordinate system to the video monitoring device coordinate system (i.e., the camera coordinate system);

[0072] Step 3.23: Convert the coordinates of the video monitoring device coordinate system to the image coordinate system.

[0073] The detection scanning plane of the millimeter-wave radar is a two-dimensional plane, and the (x, y) coordinate information of the target can be obtained, without the z coordinate information of the target. Establish a coordinate system with the radar plane (as Figure 4 shown), and the conversion from the millimeter-wave radar coordinate system O r to the world coordinate system O w can be regarded as the conversion of a two-dimensional X-Y coordinate system. The relationship between O r and O w is nothing more than translation and rotation. The following formula:

[0074] Conversion of the radar coordinate system to the world coordinate system centered on the camera:

[0075]

[0076] Conversion of the image plane coordinate system to the pixel coordinate system:

[0077]

[0078] Conversion of the camera coordinate system to the image coordinate system:

[0079]

[0080] Conversion of the world coordinate system to the camera coordinate system:

[0081]

[0082] Conversion of the world coordinate system to the pixel coordinate system:

[0083]

[0084] Step 4: Use the Gaussian mixture model and Bayesian theory to perform fusion detection on the first detection image and the second detection image to obtain the fused detection result.

[0085] Step 4.1: Based on the Gaussian distribution, integrate the position distribution f(x1, y1) of the radar detection result relative to the true value to obtain the probability distribution F(x1, y1), and integrate the position distribution f(x2, y2) of the video detection result relative to the true value to obtain the probability distribution F(x2, y2).

[0086] Assume that the individual detection results of the radar and the video relative to the true position of the target both follow a Gaussian distribution. Since the detection results can be represented in the form of x - y - z coordinates on the road plane, and it can be considered that z is 0 on the road plane, the detection results of the sensor relative to the true position should follow a two - dimensional Gaussian distribution in the x - y plane.

[0087] The function of the one - dimensional Gaussian distribution is shown as follows:

[0088]

[0089] Calculate the mathematical expectation and standard deviation of the vehicle target center position according to the data calibrated from the video image and the data output by the radar respectively. In the function expression of the Gaussian distribution, first assume that every two variables are relatively independent and both follow a Gaussian distribution, that is, for the probability distribution function f(x0, x1, …, x n ) there is the following equation:

[0090] f(x0, x1, ···· x n ) = f(x0)·f(x1) ···· f(x n )

[0091] In the formula, f(x i ) follows a one - dimensional Gaussian distribution, that is:

[0092]

[0093] where δ i and μ i are the standard deviation and mean of the i - th variable.

[0094] When describing the two - dimensional Gaussian distribution, the value of n is 2. Since x and y are independent of each other, the two - dimensional Gaussian distribution function can be expressed as:

[0095]

[0096] If it is assumed that the target detection results of the radar and the video both follow a Gaussian distribution, let f(x1,y1) represent the position distribution of the radar detection result relative to the true value, and f(x2,y2) represent the position distribution of the video detection result relative to the true value. Then, the probability distributions F(x1,y1) and F(x2,y2) of the two can be obtained through integration.

[0097] Step 4.2: Based on Bayes' formula, obtain the true position B of the target according to the probability distributions F(x1,y1) and F(x2,y2). i The probability P(B i │A) of the true position B of the target relative to the detection result A.

[0098] The true position B of the target i The probability P(B i │A) of the true position B of the target relative to the detection result A can be calculated through Bayes' formula:

[0099]

[0100] Among them, the probability of event B i is P(B i ), which is equal at every pixel in the area where the actual target may appear; the probability of event A under the condition that event B i has occurred is P(A│B i ) ∈ [F(x1,y1), F(x2,y2)]; The value of is 1.

[0101] Step 4.3: Select a preset area, which includes the target position in the first detection image and the target position in the second detection image. Calculate the joint probability of each pixel in the preset area and the probability P(Bi│A), and output the position with the highest joint probability as the fused detection result.

[0102] After obtaining the distribution of the true value relative to the detection result, the fusion detection process is mainly divided into three cases: (1) Both the video detection result and the radar detection result exist, as shown in Figure 5 (a); (2) Only the video detection result exists, as shown in Figure 5 (b); (3) Only the radar detection result exists, as shown in Figure 5 (c).

[0103] In the first case, as shown in Figure 6 (a), we take the pixels in a certain area (i.e., the preset area) around the video detection result and the radar detection result, as shown in Figure 6 (b). Each pixel is regarded as an event where the true target may exist, and according to the probability P(B i│A) Calculate the joint probability one by one. After calculating the event probabilities of all pixel positions in the region, output the position with the highest joint probability as the fused detection result, which is used as an estimated value of the true position of the target, as shown in Figure 6 (c).

[0104] Step 5. As shown in Figure 7 , based on the fused detection result, use the detection and tracking algorithm of multi-source information fusion to achieve the final detection and tracking of the vehicle target.

[0105] The detection process of the vehicle target using the multi-source information fusion algorithm is as shown in Figure 6 . Assume that the current frame being processed is the k-th frame, and the target trajectory set traces is formed from the previous k - 1 frames. First, use the target detection algorithm to obtain the detection target set detections of the k-th frame, then use the data association algorithm to establish the association matrix between the target and the trajectory, and adopt corresponding trajectory processing strategies for different association results to update the target trajectory.

[0106] In the detection traffic scenario of radar-camera fusion, missed detection and false detection events cannot be completely avoided. Based on this, how to design a robust data association algorithm has become one of the core issues in multi-target tracking.

[0107] Step 5.1: Use the target detection algorithm to obtain the detection target set detections of the k-th frame.

[0108] Data association algorithm: The main research content is multi-target tracking in highway traffic data to obtain the complete fused trajectory information of vehicles. Based on the target detection results, the multi-target tracking problem can be simplified to the association and matching problem between the target and the trajectory. Therefore, the research on multi-target tracking methods can be divided into two aspects: namely, the data association between the target and the trajectory and the target-trajectory processing for different association results.

[0109] Data association based on the detection result is the first step to achieve multi-target tracking. The associated data is the target detection result of the current k-th frame and the target trajectories formed from the previous k - 1 frames Among them, the detection result includes the coordinate position of the fused target box, the target category, and the target confidence; the existing target trajectory includes the trajectory unique ID, the target information (target video radar coordinate position, target category, and confidence) in each frame of the trajectory, the trajectory direction, the image of the previous frame of the trajectory, etc.

[0110] IoU is a method for representing data correlation based on target position information. Based on the IoU idea, a similarity measurement method is established, and its calculation method is as follows:

[0111]

[0112] Among them, represents the i-th target box of the detection result of the k-th frame, represents the last target box of the existing j-th trajectory . The higher the correlation between two target boxes, the greater the corresponding similarity metric value. The most ideal situation is that the two targets completely overlap, that is, the ratio is 1.

[0113] Step 5.2: Use a data association algorithm to establish an association matrix between targets and trajectories.

[0114] Use the IoU similarity metric method to obtain the association matrix A mn between targets and trajectories, as shown in the following formula. Each row represents the similarity metric value between the target detection box and the target box of each tracking trajectory, and each column represents the similarity metric value between the target box of this tracking trajectory and the current detection boxes. Based on this, find the best association result and determine the tracking trajectory. The association matrix A mn is:

[0115]

[0116] Among them, the m rows represent the similarity metric values between the target detection box and the target boxes of each tracking trajectory, and the n columns represent the similarity metric values between the target box of this tracking trajectory and the current detection boxes

[0117] Due to the instability of the detection algorithm, there are missed detections in target detection. Therefore, the method of simply relying on the detection results to complete target tracking is unreliable. To solve this problem, the KCF (Kernel Correlation Filter) algorithm is used to predict the target position. The KCF tracking algorithm based on correlation filtering has a fast tracking speed and high tracking accuracy when the target scale remains unchanged (the occurrence of missed detections is short-term, generally only one frame, and it can be considered that the target scale remains unchanged).

[0118] Step 5.3: Use the tracker template to perform cyclic detection on the current k-th frame, calculate the maximum response value, determine the target prediction position, and achieve trajectory tracking.

[0119] Specifically, the essence of KCF is to extract HOG (Histogram of Oriented Gradient) features of the tracking target to construct a tracker, use the tracker template to perform cyclic detection on the current k-th frame, calculate the maximum response value, determine the target prediction position, and achieve trajectory tracking.

[0120] Step 5.31: Initialize the KCF tracker: As Figure 8As shown, the rectangular box area in Figure a represents the target detected at the (k - 1)-th frame. This target is successfully associated with a trajectory, and the target is in a tracking state. At this time, the trajectory information is updated. At the k-th frame, due to missed detection, the target is in a lost state, and at this time, the KCF tracker needs to be initialized. The initialization process means: for the (k - 1)-th frame image (this frame image is included in the trajectory information), circular sampling is performed near the target position to extract HOG features, and a target tracker is trained using ridge regression.

[0121] Step 5.32, Update the target position: Use the target position in the (k - 1)-th frame to select a candidate region in the k-th frame image, such as Figure 8 (the larger rectangular box area in (b) (in the KCF algorithm, the candidate box is 2.5 times the size of the original target)), circular sampling is performed within the candidate region, and the response value corresponding to each sampling sample is calculated using the target tracker trained in the first step. The position with the strongest response is used as the predicted position of the k-th frame, and the yellow box is the target position of the calculated maximum response value;

[0122] Step 5.33, Update the tracker template: In order to adapt to the changes of the target, after each prediction result is obtained in the KCF algorithm, the template and parameters are iteratively updated.

[0123] Step 5.4, Due to the change in the pixel gray value of the occluded vehicle, it can be determined whether the vehicle is occluded. When occlusion is successfully detected, the KCF tracker needs to stop template update and propose a new tracking strategy for the occlusion situation. In the highway scenario, the driving process of the target vehicle in a short period of time can be regarded as uniform linear motion. Therefore, Kalman filtering is used to predict the position of the occluded target.

[0124] Kalman filtering uses the current state of the target to estimate its next state. The specific process of target tracking by Kalman filtering is as follows:

[0125] Step 5.41, Predict the state of the k-th frame using the state value of the (k - 1)-th frame.

[0126]

[0127]

[0128] Where represents the state value of the k-th frame, represents the control quantity of the (k - 1)-th frame, and A and B are system parameters. is a vector, which can include speed in addition to coordinates, such as coordinate x, coordinate y, speed x, and speed y. is the covariance of the error of the predicted value. The prediction process adds new uncertainty Q, plus the existing uncertainty.

[0129] Step 5.42: Calculate the Kalman gain K k .

[0130]

[0131] where H is a parameter of the measurement system.

[0132] Step 5.43: Use the observation value Z k to update the prediction result, perform a weighted average on the prediction result and the observation result to obtain the state estimate at the current moment. Meanwhile, update the covariance P k .

[0133]

[0134]

[0135] When the target leaves the occlusion and is detected again, when the data association algorithm is used again, the target successfully matches the fusion trajectory, so the target switches to the tracking state and the fused target trajectory continues to track.

[0136] 1. The present invention uses the method of multi-source information fusion to make a more accurate estimation of the target detection of vehicles in different scenarios, enhancing the detection and tracking ability of vehicle targets in complex scenarios; for the video radar in different distances and weather conditions, it makes up for each other's advantages and disadvantages. In the area with the advantages of video monitoring, a fusion detection strategy mainly based on video detection information is formulated; in the area with the advantages of radar monitoring, a fusion detection strategy mainly based on radar detection information is formulated; in the area with the common advantages of radar and video, a balanced fusion detection strategy is formulated.

[0137] 2. After obtaining the detection and tracking results of the radar and the camera respectively, in order to achieve the fusion detection and tracking of the radar and the camera, a radar-video integrated detection and tracking fusion algorithm based on the Gaussian mixture model and Bayesian theory is designed. After obtaining the distribution of the true value relative to the detection result and combining the prediction information of the true value by Kalman filtering, the target position after the fusion of radar and video information can be obtained.

[0138] 3. For the problems of missed detection and occlusion existing in the acquired data frames, we respectively adopt KCF prediction tracking and use Kalman filtering for occlusion, achieving complete detection and tracking of vehicle targets with good results.

[0139] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, "a plurality" means two or more unless otherwise specifically defined.

[0140] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or specific data points described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or specific data points described may be combined in a suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification.

[0141] The above content is a further detailed description of the present invention in conjunction with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited only to these descriptions. For those of ordinary skill in the technical field to which the present invention pertains, without departing from the concept of the present invention, several simple deductions or substitutions can be made, and all should be regarded as belonging to the protection scope of the present invention.

Claims

1. A vehicle target detection and tracking method based on multi-source information fusion, characterized in that, The vehicle target detection and tracking method includes: Step 1: Obtain a vehicle detection data set, which includes multiple first detection images and multiple second detection images. The first detection images are the images corresponding to the video monitoring device, and the second detection images are the images corresponding to the radar device; Step 2: Annotate the first detection images and the second detection images; Step 3: Perform synchronous registration of time and space on the first detection images and the second detection images; Step 4: Use the Gaussian mixture model and Bayesian theory to perform fusion detection on the first detection images and the second detection images to obtain a fused detection result; The said Step 4 includes: Step 4.1: Based on the Gaussian distribution, integrate the position distribution f(x1, y1) of the detection result of the radar relative to the true value to obtain the probability distribution F(x1, y1), and integrate the position distribution f(x2, y2) of the detection result of the video relative to the true value to obtain the probability distribution F(x2, y2); Step 4.

2. Based on Bayes' formula, obtain the true position B of the target according to the probability distribution F(x1, y1) and the probability distribution F(x1, y2). i The probability P(B i │A) relative to the detection result A; True position B of the target i Probability P(B i |A) relative to the detection result A can be statistically calculated by Bayes' formula: Among them, event B i has a probability of P(B i ), and the pixels are equal everywhere in the area where the actual target may appear; given that event B i has occurred, the probability of event A is P(A|B i ) ∈ [F(x1, y1), F(x2, y2)]; The value of is 1; Step 4.3: Select a preset area, where the preset area includes the target positions in the first detection image and the second detection image, and calculate the joint probability of each pixel in the preset area and the probability P(B i │A), and output the position with the highest joint probability as the fused detection result; Step 5: Based on the fused detection result, use a multi-source information fusion detection and tracking algorithm to achieve the final detection and tracking of the vehicle target.

2. The vehicle target detection and tracking method based on multi-source information fusion according to claim 1, characterized in that, The said Step 2 includes: Use annotation boxes to annotate the vehicles in the first detection images and the second detection images, and annotate the vehicles in the first detection images and the second detection images according to the vehicle size, lane position, and appearance order.

3. The vehicle target detection and tracking method based on multi-source information fusion according to claim 1, characterized in that The said Step 3 includes: Step 3.1: Perform time registration on the first detection images and the second detection images; Step 3.2: Perform spatial registration on the first detection images and the second detection images.

4. The vehicle target detection and tracking method based on multi-source information fusion according to claim 3, characterized in that The said Step 3.1 includes: Judge whether the frame numbers and times of the first detection images and the second detection images are consistent. If they are consistent, the time registration is completed.

5. The vehicle target detection and tracking method based on multi-source information fusion according to claim 3, characterized in that, The said Step 3.2 includes: Step 3.21: Convert the coordinates in the radar device coordinate system to the world coordinate system centered on the video monitoring device; Step 3.22: Convert the coordinates of the world coordinate system to the video monitoring device coordinate system; Step 3.23: Convert the coordinates of the video monitoring device coordinate system to the image coordinate system.

6. The vehicle target detection and tracking method based on multi-source information fusion according to claim 1, characterized in that The said Step 5 includes: Step 5.1: Use a target detection algorithm to obtain the detection target set detections of the k-th frame; Step 5.2: Use a data association algorithm to establish an association matrix between the target and the trajectory; Step 5.3: Use the tracker template to perform cyclic detection in the current k-th frame, calculate the maximum response value, determine the target prediction position, and achieve trajectory tracking.

7. The vehicle target detection and tracking method based on multi-source information fusion according to claim 6, characterized in that, The said Step 5.3 includes: Step 5.31: Initialize the KCF tracker; Step 5.32: Update the target position; Step 5.33: Update the tracker template.

8. The vehicle target detection and tracking method based on multi-source information fusion according to claim 6, characterized in that After the said Step 5.3, it further includes: Step 5.4: Use Kalman filtering to predict the position of the occluded target.

9. The vehicle target detection and tracking method based on multi-source information fusion according to claim 8, characterized in that The said Step 5.4 includes: Step 5.41: Use the state value of the (k - 1)-th frame to predict the state of the k-th frame; Step 5.42, calculate the Kalman gain K k ; Step 5.43: Use the observed value Z k to update the prediction result, perform a weighted average on the prediction result and the observed result to obtain the state estimate at the current moment. Meanwhile, update the covariance P k .

Images