Target tracking method based on 4D radar and camera fusion

Abstract

The invention provides a target tracking method based on 4D radar and camera fusion, and belongs to the technical field of intelligent traffic, and the method comprises the steps: carrying out the target detection of obtained 4D radar data and camera data, and obtaining a radar target detection result and a camera target detection result; matching the two results with the current track respectively; when matching fails, target initialization is carried out based on the two results, and an initialization track marked as a radar and an initialization track marked as a camera are obtained; when the matching is successful, updating the current track by using a target detection result; after updating, judging whether the two initialized trajectories conform to a trajectory fusion rule or not, and when the two initialized trajectories conform to the trajectory fusion rule, fusing the two initialized trajectories to generate a fused trajectory; and during updating, when the current track is a fusion track, updating the fusion track by using at least one of a radar target detection result and a camera target detection result. According to the processing scheme, the precision, continuity and stability of target tracking are improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent transportation technology, and in particular to a target tracking method based on 4D radar and camera fusion. Background Art

[0002] With the acceleration of urbanization and the booming new energy vehicle market, traffic safety and efficiency have become major issues that need to be solved urgently. Intelligent transportation systems have effectively improved traffic conditions and traffic safety by collecting, processing and applying traffic information. Among them, vehicle detection and tracking technology, as a key component of the front-end perception function of intelligent transportation systems, plays a vital role in traffic flow monitoring and other aspects. In recent years, with the advancement of deep learning research and the development of computer vision technology, camera-based detection and tracking technology has made significant progress. Cameras can provide rich visual information and have a natural advantage in identifying details such as the location and size of vehicles. However, its performance is greatly affected by factors such as light conditions and weather changes. In contrast, 4D millimeter-wave radar, as an emerging sensing technology, has emerged in recent years. Compared with traditional 3D radar, 4D radar has increased the observation range in the vertical direction. It can not only measure distance, speed and azimuth, but also provide pitch angle information, greatly improving detection accuracy. The characteristics of radar enable it to work stably under various severe weather conditions and is not affected by night lighting conditions. Therefore, it shows great application potential in all-weather intelligent transportation systems.

[0003] Due to the different characteristics of radar data and image data, traditional target detection methods have different processing methods. Traditional radar tracking usually directly uses signal processing algorithms such as constant false alarm rate (CFAR) to obtain point cloud data, and performs target cluster detection and tracking. Camera tracking usually uses deep learning networks for detection, and then uses the detection results in the tracking algorithm. Although 4D radar and camera have their own advantages in tracking, the use of a single sensor often cannot meet the needs of high precision and high robustness in complex traffic environments. Specifically, radar data has a high noise level, making it difficult to obtain very accurate target size information; in addition, when the radar encounters a stationary or frequently paused target, it is easy to lose the static tracking target because its algorithm cannot distinguish between similar stationary targets and background noise targets. Due to its planar imaging characteristics, the camera cannot directly detect the depth information of the target, and the prior assumptions are often inaccurate, resulting in poor target position and speed estimation; at close range, the drastic change of the target pixel position caused by the field of view may also lead to the loss of the target. In contrast, 4D radar has a higher credibility for the detection of target position and speed, while the camera can intuitively and accurately estimate the size of the target.

[0004] In order to more effectively utilize the advantages of the two sensors, a fusion method can be used to combine 4D radar and camera to effectively cope with various environmental challenges, especially vehicle tracking tasks in complex traffic scenarios, such as parking detection. The most common approach is to simply match the detection results of the two modalities and use them for tracking. Some existing studies often input the two data into the deep learning network for fusion at the same time, but this deep learning method usually has a slower detection speed and cannot meet real-time requirements. These approaches do not take into account that when one of the single source observation data is unreliable, simple fusion often causes interference and contamination of the original more accurate estimate of a single sensor, thereby compromising the tracking accuracy. These methods are also unable to cope with the situation where a single sensor is missing under certain conditions, such as overexposure of the camera or inability to detect in a foggy environment. Summary of the invention

[0005] In view of this, an embodiment of the present application provides a target tracking method based on 4D radar and camera fusion, which is used for multi-target vehicle tracking in a road environment. It improves the robustness of the system while utilizing the characteristics of the 4D radar for all-weather operation, at least partially solves the limitations of unreliable single-source observation data in the prior art, and improves the accuracy and stability of vehicle tracking.

[0006] The present application embodiment provides a target tracking method based on 4D radar and camera fusion, including:

[0007] Performing target detection on the 4D radar data and the camera data acquired in real time, respectively, to obtain target detection results, wherein the target detection results include radar target detection results and camera target detection results;

[0008] Matching the radar target detection result and the camera target detection result with the current trajectory respectively;

[0009] When at least one of the radar target detection result and the camera target detection result does not match the current trajectory, performing target initialization based on the radar target detection result and the camera target detection result to obtain an initialization trajectory marked as radar and an initialization trajectory marked as camera;

[0010] When the radar target detection result and the camera target detection result respectively match the current trajectory, updating the current trajectory using the target detection results;

[0011] After the update, it is determined whether the initialization track marked as radar and the initialization track marked as camera meet the track fusion rule. If they meet the track fusion rule, the initialization track marked as radar and the initialization track marked as camera are fused to generate a fused track, and the initialization track marked as radar and the initialization track marked as camera are deleted.

[0012] During the updating process, when the current trajectory is the initialization trajectory marked as radar, the initialization trajectory marked as radar is updated using the radar target detection result; when the current trajectory is the initialization trajectory marked as camera, the initialization trajectory marked as camera is updated using the camera target detection result; when the current trajectory is the fused trajectory, the fused trajectory is updated using at least one of the radar target detection result and the camera target detection result.

[0013] According to a specific implementation of the embodiment of the present application, the real-time acquired 4D radar data and camera data are respectively subjected to target detection to obtain target detection results, including:

[0014] Preprocessing the 4D radar data acquired in real time to obtain radar point cloud data;

[0015] Performing target detection after clustering the radar point cloud data to obtain the radar target detection result, wherein the radar target detection result includes position information, speed information and size information of the radar target;

[0016] The camera data acquired in real time is subjected to target detection through an image target detection network to obtain the camera target detection result on the image plane, wherein the camera target detection result includes position information and size information of the camera target.

[0017] According to a specific implementation of an embodiment of the present application, the target initialization is performed based on the radar target detection result and the camera target detection result, respectively, to obtain an initialization trajectory marked as a radar and an initialization trajectory marked as a camera, including:

[0018] Perform EKF modeling based on the radar target detection result to obtain a radar EKF model;

[0019] Initializing the radar EKF model according to the first initialization parameter to obtain the initialization trajectory marked as radar;

[0020] Perform EKF modeling based on the camera target detection result to obtain a camera EKF model;

[0021] The camera EKF model is initialized according to the second initialization parameter to obtain the initialization trajectory marked as the camera.

[0022] According to a specific implementation of an embodiment of the present application, initializing the radar EKF model according to the first initialization parameter to obtain the initialization trajectory marked as radar includes:

[0023] Setting a first variance for the position information of the radar target, setting a second variance for the speed information of the radar target, and setting a third variance for the size information of the radar target, wherein the third variance is greater than the first variance and the second variance;

[0024] The position information of the radar target, the speed information of the radar target, the size information of the radar target, the first variance, the second variance, the third variance and the first noise parameter are used as the first initialization parameters, and the radar EKF model is initialized according to the first initialization parameters to obtain the initialized trajectory marked as radar.

[0025] According to a specific implementation of the embodiment of the present application, the camera EKF model is initialized according to the second initialization parameter to obtain the initialization trajectory marked as the camera, including:

[0026] Obtaining an initialization speed of a camera target based on the camera target detection result of a current frame and the camera target detection result of a previous frame;

[0027] estimating an initialization position and an initialization size of a camera target according to an initialization speed of the camera target;

[0028] Setting a fourth variance for the initialization speed, setting a fifth variance for the initialization position, and setting a sixth variance for the initialization size;

[0029] The initialization speed, the initialization position, the initialization size, the fourth variance, the fifth variance, the sixth variance and the second noise parameter are used as the second initialization parameters, the camera EKF model is initialized according to the second initialization parameters, and the initialization trajectory marked as the camera is obtained.

[0030] According to a specific implementation of the embodiment of the present application, the trajectory fusion rule includes:

[0031] Calculate the IoU value of the 3D detection frame of the initialization track marked as the radar and the initialization track marked as the camera in the current frame and multiple consecutive frames adjacent to the current frame;

[0032] When the IoU values ​​of the current frame and the multiple consecutive frames adjacent to the current frame are greater than a first threshold, the initialized track marked as the radar and the initialized track marked as the camera are fused.

[0033] According to a specific implementation of the embodiment of the present application, the updating of the fused trajectory by using at least one of the radar target detection result and the camera target detection result includes:

[0034] When the 4D radar data is missing, updating the fusion trajectory using the camera target detection result;

[0035] When the camera data is missing, updating the fusion trajectory using the radar target detection result;

[0036] When both the 4D radar data and the camera data exist, and both the radar target detection result and the camera target detection result match the fused trajectory, the fused trajectory is updated using the radar target detection result and the camera target detection result.

[0037] According to a specific implementation of the embodiment of the present application, the updating of the fused trajectory by using the radar target detection result and the camera target detection result includes:

[0038] Setting different weight coefficients for the position information, speed information and size information of the radar target in the radar target detection result to obtain a first set of weight coefficients;

[0039] Setting different weight coefficients for the position information of the camera target, the size information of the camera target, and the speed information calculated according to the position information of the camera target in the camera target detection result, respectively, to obtain a second set of weight coefficients;

[0040] Based on the first set of weight coefficients and the second set of weight coefficients, the fused trajectory is updated by using the radar target detection result and the camera target detection result.

[0041] According to a specific implementation of the embodiment of the present application, the method further includes:

[0042] Counting the number of consecutive matches of the initialization track marked as a radar or the initialization track marked as a camera;

[0043] When the number of consecutive matches exceeds the second threshold, the trajectory is a credible target;

[0044] When the number of consecutive matches is lower than the second threshold, the track is considered an untrustworthy target and is deleted.

[0045] According to a specific implementation of the embodiment of the present application, the method further includes:

[0046] Counting the number of consecutive mismatches of the fused trajectory, the initialization trajectory marked as a radar, or the initialization trajectory marked as a camera;

[0047] When both 4D radar data and camera data are missing, the time interval between the previous frame calculation and the next frame calculation is calculated using the number of consecutive mismatches;

[0048] When the number of consecutive mismatches exceeds a third threshold, the track is deleted.

[0049] Beneficial effects:

[0050] The target tracking method based on 4D radar and camera fusion in the embodiment of the present application performs target detection on the real-time acquired 4D radar data and camera data respectively. According to the target detection results, the radar and camera each perform target initialization to obtain their own corresponding initialization trajectories; during the tracking process, the tracking trajectories of the radar and camera that meet certain matching conditions are fused, and a new fused trajectory is obtained by combining their respective advantage observations; the trajectory is updated during the observation process using the observation values ​​of the radar and camera, and is jointly updated when it matches the observations of both sensors, and can also be updated independently when a certain sensor is absent; the tracking target is maintained to obtain the final tracking result. Therefore, the design of the trajectory fusion method of the present application improves the accuracy, continuity and stability of target tracking, and can better utilize the characteristics of the sensor to effectively avoid the problem of multimodal interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 is a flow chart of a target tracking method based on 4D radar and camera fusion according to an embodiment of the present invention;

[0053] Figure 2 is another flow chart of a target tracking method based on 4D radar and camera fusion according to an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of initializing the speed state of a target trajectory using a camera target detection result according to an embodiment of the present invention;

[0055] Figure 4Schematic diagram of initializing the position and size state of a target track using a camera target detection result according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0057] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0058] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0059] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The illustrations only show the components related to the present application rather than being drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0060] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0061] The present application embodiment provides a target tracking method based on 4D radar and camera fusion. Figures 1 to 4 Describe in detail.

[0062] Reference Figure 1 The target tracking method based on 4D radar and camera fusion in this embodiment includes the following steps:

[0063] Step S101: performing target detection on the 4D radar data and camera data acquired in real time, respectively, to obtain target detection results, wherein the target detection results include radar target detection results and camera target detection results;

[0064] Step S102, matching the radar target detection result and the camera target detection result with the current trajectory respectively;

[0065] Step S103: when at least one of the radar target detection result and the camera target detection result does not match the current trajectory, target initialization is performed based on the radar target detection result and the camera target detection result to obtain an initialization trajectory marked as radar and an initialization trajectory marked as camera;

[0066] Step S104: when the radar target detection result and the camera target detection result respectively match the current trajectory, the current trajectory is updated using the target detection result;

[0067] Step S105: after updating, judging whether the initialization track marked as radar and the initialization track marked as camera meet the track fusion rule; when meeting the track fusion rule, fusing the initialization track marked as radar and the initialization track marked as camera to generate a fused track, and deleting the initialization track marked as radar and the initialization track marked as camera;

[0068] Step S106: During the updating process, when the current trajectory is the initialization trajectory marked as radar, the initialization trajectory marked as radar is updated by using the radar target detection result; when the current trajectory is the initialization trajectory marked as camera, the initialization trajectory marked as camera is updated by using the camera target detection result; when the current trajectory is the fused trajectory, the fused trajectory is updated by using at least one of the radar target detection result and the camera target detection result.

[0069] In this embodiment, the real-time acquired 4D radar data and camera data are respectively used for target detection. According to the target detection results, the radar and camera each perform target initialization to obtain their own corresponding initialization trajectories; during the tracking process, the tracking trajectories of the radar and camera that meet certain matching conditions are fused, and a new fused trajectory is obtained by combining their respective advantage observations; the trajectory is updated during the observation process using the radar and camera observations, and a joint update is performed when it matches the observations of both sensors, and it can also be updated independently when a sensor is absent; the tracking target is maintained to obtain the final tracking result. Therefore, the design of the trajectory fusion method of the present application improves the accuracy, continuity and stability of target tracking, and can better utilize the characteristics of the sensor to effectively avoid the problem of multimodal interference.

[0070] In one embodiment, performing target detection on the real-time acquired 4D radar data and camera data respectively to obtain target detection results includes:

[0071] Preprocessing the 4D radar data acquired in real time to obtain radar point cloud data;

[0072] Performing target detection after clustering the radar point cloud data to obtain the radar target detection result, wherein the radar target detection result includes position information, speed information and size information of the radar target;

[0073] The camera data acquired in real time is subjected to target detection through an image target detection network to obtain the camera target detection result on the image plane, wherein the camera target detection result includes position information and size information of the camera target.

[0074] In the specific implementation, first install 4D millimeter wave radar sensing equipment and traffic monitoring cameras on appropriate roadside traffic poles. The two paired devices need to have the same orientation and roughly the same sensing field of view, and perform time synchronization and spatial calibration on the devices to ensure that the two devices can sample synchronous data and that the sampled data can be converted to the same world coordinate system. At the same time, it is necessary to determine the position of the radar and camera in the world coordinate system to meet the calculation premise of the subsequent tracking model.

[0075] The real-time synchronized 4D radar data and camera image data are sent to their respective target detection modules. Through the detection algorithms of their respective branches, the target detection results are obtained and sent to the target tracking system for processing. Figure 2 The steps of radar input, radar target detection, radar target detection results and camera input, camera target detection network and camera target detection results in the above figure. After installing the two sensors, they need to be calibrated to determine the external parameters of the radar and the internal and external parameters of the camera so that the subsequent tracking steps can perform coordinate conversion.

[0076] The raw data measured by 4D radar will first go through the radar signal processing step, and the signal will be converted from the time domain to the frequency domain through the fast Fourier transform of multiple radar feature dimensions to better separate and identify the target; and then the CFAR filter will be further performed to dynamically adjust the threshold to adapt to the changing noise environment, and the radar signal will be converted into radar point cloud data. The radar point cloud data will be uniformly converted to the world coordinate system through the external parameter matrix of the radar calibration, and then the target detection and tracking of the radar data will also be performed in the world coordinate system. The point cloud data is sent to the target detection module of the radar, and clustering algorithms such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise, DBSCAN) are applied to identify target clusters in a complex background, and obtain 3D target detection results in the world coordinate system. In addition to using density as the clustering dimension, the clustering algorithm in the radar often uses speed information as an auxiliary, so that it is easier to identify dynamic targets. Therefore, these detection results contain the position and speed information of the target, as well as rough target size information. This traditional radar target detection method that does not use deep learning has a great advantage in detection speed. Usually, these processes can be completed in the radar equipment to obtain a target detection list.

[0077] The camera performs deep learning target detection through a commonly used target detection network such as YOLO, and obtains a 2D target detection result on the image plane, including the 2D position and size information of the target on the image. Specifically, the camera image does not require too many preprocessing steps and can be directly sent to an image target detection network such as YOLO. YOLO is a commonly used end-to-end real-time target detection network that achieves a good balance between accuracy and detection speed. The detection result is a 2D target detection box on the image plane.

[0078] The image and radar detection results are sent to the target tracking system together. When the tracking system completes the processing of the current detection results, it will request new input from the system. The radar and camera will then detect the real-time data and send it to the tracking system, and the cycle will repeat.

[0079] In one embodiment, matching the radar target detection result and the camera target detection result with the current trajectory respectively includes:

[0080] According to the input target detection results from the radar and camera, the existing target trajectory models are matched respectively for the status update of the existing trajectory. The current detection box (the detection box in the target detection result) needs to calculate the intersection over union (IoU) with the existing trajectory target bounding box, and filter the match according to a certain minimum threshold. If it is higher than the set threshold, the match is successful, and if it is lower than the set threshold, the match fails. Radar detection matching is performed directly in 3D space, while matching with the image 2D detection box first requires projecting the 3D bounding box of the trajectory target onto the 2D image plane. The Hungarian algorithm is used for the calculated target IoU to match the detection results with the existing trajectory one by one.

[0081] In one embodiment, performing target initialization based on the radar target detection result and the camera target detection result to obtain an initialization track marked as a radar and an initialization track marked as a camera respectively includes:

[0082] Perform EKF modeling based on the radar target detection result to obtain a radar EKF model;

[0083] Initializing the radar EKF model according to the first initialization parameter to obtain the initialization trajectory marked as radar;

[0084] Perform EKF modeling based on the camera target detection result to obtain a camera EKF model;

[0085] The camera EKF model is initialized according to the second initialization parameter to obtain the initialization trajectory marked as the camera.

[0086] In the specific implementation, the target detection results obtained by the radar and camera are input into the target tracking framework. At this time, the algorithm will match the existing trajectory in the system with the current detection result. The detection result that fails to match is regarded as a new target and becomes a new trajectory. EKF is used to model it, corresponding to Figure 2 Unmatched detections and initialization trajectories. The Extended Kalman Filter (EKF) is an extended form of the Kalman filter and is used for the problem of state estimation of nonlinear systems. In target tracking, EKF is widely used to estimate the target's state variables such as position and velocity. It continuously updates the estimate of the target state by combining historical observations with current observations, thereby providing more accurate position predictions and state estimates.

[0087] The vehicle targets on the road are modeled in 3D space, with hidden states, including the position, velocity vector, acceleration vector and size of the vehicle target in the world coordinate system. The target detection results of radar and image are independently modeled by EKF, and the radar EKF model and camera EKF model are obtained respectively, in order to select appropriate EKF initialization parameters according to the characteristics of their respective sensors.

[0088] Specifically, to perform EKF modeling on the target, we first need to determine the hidden state of the target and simply model the target as a 3D target with uniformly accelerated motion. Then:

[0089] x=[x,y,z,l,w,h,v x ,v y ,a x ,a y ] T ,

[0090] Among them, (x, y, z) is the center point of the target in the world coordinate system, (l, w, h) is the size of the target in 3D space, (v x ,v y ,a x ,a y ) are the speed and acceleration of the target on the x-axis and y-axis. Here we do not consider the speed and acceleration components of the target on the z-axis, because the target is assumed to move on the ground plane, so the z-axis components of these attributes are implicitly set to 0. At the same time, the orientation information of the target is implicit in the velocity information. In this embodiment, the default orientation of the target velocity vector is the front orientation of the target. These attributes are set as the hidden state of the target. When adding a target trajectory using EKF, these state quantities and corresponding covariances need to be initialized.

[0091] The target tracking system maintains a target list and adds, updates, and deletes target tracks. In road scenarios, the targets of interest are limited to vehicle targets. When a target detected by the radar and camera does not match any existing target track, the target is initialized as a new target track. At this time, it is necessary to distinguish between targets from radar and camera, because the measurements of 4D radar and camera have different characteristics, and different initialization strategies need to be used for the targets.

[0092] In one embodiment, the initializing the radar EKF model according to the first initialization parameter to obtain the initialization trajectory marked as radar includes:

[0093] Setting a first variance for the position information of the radar target, setting a second variance for the speed information of the radar target, and setting a third variance for the size information of the radar target, wherein the third variance is greater than the first variance and the second variance;

[0094] The position information of the radar target, the speed information of the radar target, the size information of the radar target, the first variance, the second variance, the third variance and the first noise parameter are used as the first initialization parameters, and the radar EKF model is initialized according to the first initialization parameters to obtain the initialized trajectory marked as radar.

[0095] In specific implementation, when initializing the target detected by the radar, since the 4D radar can detect the azimuth and radial velocity of the 3D space station where the target is located, the initial values ​​of the azimuth and velocity states of the target can be set directly. At the same time, the radar can detect the rough size of the target, but due to the sparse characteristics and noise level of radar detection, this detection is not particularly accurate. We directly use these values ​​to initialize the target and set its acceleration to 0, assuming that the target is initially moving at a uniform speed, and the acceleration of the camera target is the same. Therefore, when the radar target is initialized, a larger initial value will be given to the variance of the corresponding size in the covariance, indicating that the initial estimate of this part is not accurate, and a smaller variance will be set for the position and velocity.

[0096] In addition, the noise parameters (first noise parameters) of the radar target need to be initialized. The system noise is set to a fixed value a priori, and since the observation accuracy of the sensor is related to the distance from the target to the sensor, the observation accuracy of the radar and camera for farther targets will decrease, so the observation noise R is a function related to the distance from the target to the sensor:

[0097] R=R(d)=θ+βd γ ,

[0098] Among them, θ, β and γ are all adjustable hyperparameters, and d is the distance from the target to the sensor. The observation noise of different observation dimensions has different coefficients, but the observation noise will increase with the increase of distance.

[0099] In one embodiment, the initializing the camera EKF model according to the second initialization parameter to obtain the initialization trajectory marked as the camera includes:

[0100] Obtaining an initialization speed of a camera target based on the camera target detection result of a current frame and the camera target detection result of a previous frame;

[0101] estimating an initialization position and an initialization size of a camera target according to an initialization speed of the camera target;

[0102] Setting a fourth variance for the initialization speed, setting a fifth variance for the initialization position, and setting a sixth variance for the initialization size;

[0103] The initialization speed, the initialization position, the initialization size, the fourth variance, the fifth variance, the sixth variance and the second noise parameter are used as the second initialization parameters, the camera EKF model is initialized according to the second initialization parameters, and the initialization trajectory marked as the camera is obtained.

[0104] In specific implementation, when initializing the target detected by the camera, since the target detected by the camera is on the image plane, it belongs to 2D target detection, so the 3D state cannot be initialized directly, and the coordinates of the image plane need to be converted to the world coordinate system through the image's external and internal parameter matrices. Since the image detection has no depth information, we can only make a rough estimate of the position and size of the target. The initialization of speed information is also indirectly inferred from the displacement of the target in two consecutive frames of the image, and this speed estimate is also inaccurate. Therefore, when initializing the covariance, it is necessary to set a larger variance for the position, speed, and size.

[0105] Specifically, refer to Figure 3 First, you need to use the bounding box of the same target in two consecutive frames to initialize the speed. At this time, a certain IoU (Intersection over Union) threshold can be used to calculate and judge the same target. Take the center points of the two targets and transform them from the image coordinate system to the world coordinate system through the camera's intrinsic and extrinsic matrix. Although we cannot know the specific positions of these two points in 3D space because the camera cannot measure depth, we can extend the ray direction along the two points until it intersects with the ground plane. Here we assume that the ground is a horizontal plane. In fact, this prior assumption is inaccurate, but its calculation error is within the tolerance range. Using these two intersection points and the interval between the two frames, we can calculate the speed of the target:

[0106]

[0107] Among them, v x is the speed of the target in the x-axis direction, v y is the speed of the target in the y-axis direction, (X1, Y1) is the coordinate of the center point of the target in the previous frame of two consecutive frames, (X2, Y2) is the coordinate of the center point of the target in the next frame of two consecutive frames, and Δt is the time interval between two consecutive frames.

[0108] The speed calculated here is an initial estimate with errors and will not be very accurate. Then the speed can be used to calculate the direction of the target, combined with the target's 2D target detection box, such as Figure 4 As shown, the position and size information of the target can be further estimated. When predicting, the image detection model needs to additionally predict the tail points of the two vehicles. Select one of the two tail points as a vertical line, and make a ray along the target direction with one of the four corner points of the bounding box. You can get an intersection point, where the target direction has been transformed from the world coordinate system to the image coordinate system through the internal and external parameters. Through this intersection point and geometric relationship, a 3D detection box consisting of eight points can be constructed on the image plane. After these eight points are converted to the world coordinate system, the size and center point of the target can be calculated. Due to this indirect estimation method, the initialization information of the camera target is not very accurate, and the covariance of these state quantities needs to be set to a larger value.

[0109] In addition, the noise parameters of the camera target need to be initialized. For details, refer to the setting method of the noise parameters of the radar target, which will not be repeated here.

[0110] In one embodiment, a flag attribute is additionally set for the radar initialization trajectory and the camera initialization trajectory obtained in the above steps to mark whether they are from the radar or the camera, thereby obtaining the initialization trajectory marked as the radar and the initialization trajectory marked as the camera. When the states of two targets from the radar and the camera are in a relatively close range within a certain period of time, the system regards the two targets as the same target, merges them, and marks their state quantities as fused targets, and the corresponding trajectory is the fused trajectory.

[0111] When fusing two targets into a new target, it is necessary to set new initialization states and noise parameters for the target. Since the two sensors have different characteristics and measurements they are good at, the new initial parameters cannot be simply set to the average of the two old models, but a weighted calculation is required. Since 4D radar is better at measuring the orientation and speed of the target, the state quantities (position and speed) from the 4D radar target are given a higher weight in the new model, while the size state of the image target is given a higher weight. Therefore, the fused target can be updated using observations from the two sensors, making full use of more accurate measurements without setting too high a covariance parameter for the fused target.

[0112] In one embodiment, the trajectory fusion rule includes:

[0113] Calculate the IoU value of the 3D detection frame of the initialization track marked as the radar and the initialization track marked as the camera in the current frame and multiple consecutive frames adjacent to the current frame;

[0114] When the IOU values ​​of the current frame and the continuous multiple frames adjacent to the current frame are greater than the first threshold, the initialization track marked as the radar and the initialization track marked as the camera are fused.

[0115] In specific implementation, targets from radar and camera will be matched in the tracking system and merged into the same target, that is, the corresponding Figure 2 Fusion judgment and fusion trajectory. The initialized trajectory will be marked as coming from the radar or camera by an additional flag. The trajectory in the target list will continue to calculate the IoU value of the 3D detection box between the radar target detection result and the camera target detection result. When the IoU between the radar and the camera continues to be greater than the set first threshold for a period of time, such as 5 frames of continuous updates, the system will regard the two targets as the same target and merge them to obtain a new fusion trajectory. The merged target needs to set new initialization parameters:

[0116] x=αx r +(1-α)x c ,

[0117] x is the state of the merged target, x r is the state quantity in the radar target, α is the weight coefficient of the state quantity in the radar target, x c is the state quantity in the camera target, and 1-α is the weight coefficient of the state quantity in the camera target. The new parameters use weighted calculations instead of simple averages, and these weight coefficients are different in different state dimensions. This is because the radar and camera sensors have their own strengths in measurement, so the radar target will have a more accurate azimuth and speed estimate, while the camera target will have a more accurate size estimate after a period of observation and update. The weight coefficient is used to take advantage of the more accurate estimates of the two targets. At the same time, due to this fusion method, the covariance of all states does not need to be set too high. The fused target will also have its own unique state quantity mark.

[0118] In one embodiment, the updating of the fused trajectory using at least one of the radar target detection result and the camera target detection result includes:

[0119] When the 4D radar data is missing, updating the fusion trajectory using the camera target detection result;

[0120] When the camera data is missing, updating the fusion trajectory using the radar target detection result;

[0121] When both the 4D radar data and the camera data exist, and both the radar target detection result and the camera target detection result match the fused trajectory, the fused trajectory is updated using the radar target detection result and the camera target detection result.

[0122] In specific implementation, the trajectories in the target list maintained by the system will be matched with the input target detection results and updated accordingly. Figure 2 The trajectory matched with the detection and the update steps. The IoU is calculated as the cost by comparing the target detection box with the radar detection box in 3D space and the target detection box and image detection box projected to the plane image in the 2D image plane. The Hungarian algorithm is used to match the target in the target detection result with the current trajectory. When the IoU does not exceed the set threshold, the match is rejected (matching failure).

[0123] The matched target needs to be updated. In this embodiment, a complete EKF model prediction update formula is given, where the prediction formula is as follows:

[0124]

[0125] in, is the prior estimate at the current moment, is the posterior estimate of the previous moment, f(x,u) is the state equation, u k is the system input at the current moment (here equivalent to the time interval Δt from the last update), is the covariance matrix of the prior estimate at the current moment, Q is the process noise covariance, k is the current moment, and k-1 is the previous moment.

[0126] The updated formula is as follows:

[0127]

[0128] Among them, K k is the Kalman gain calculated at the current moment, R is the measurement noise covariance, Z k is the observed value at the current moment, h(x) is the observation equation, and P k The posterior estimate and its covariance calculated for the current time.

[0129] Based on the set target state quantity and the acceleration motion model, the state equation of the model can be obtained as follows:

[0130]

[0131] Among them, the subscript k+1 is the next moment, the subscript k is the current moment, the superscript x is the x-axis, and the superscript y is the y-axis.

[0132] Through the relative relationship between radar measurement and camera measurement and the target, the observation equation h(x) is also given as:

[0133]

[0134] Among them, the camera observation (d,θ a ,θ e ,v r ) are the distance to the target, the observed horizontal angle, the pitch angle and the measured radial velocity of the target, respectively. Suppose the position of the 4D radar in the world coordinate system is (x r ,y r ,0). The 2D detection frame obtained by the camera can be represented by the upper left and lower right corner points as (x0, y0, x1, y1). The specific observation equation from the camera to the target is (x0, y0, x1, y1) T =I(x) is given below.

[0135] To associate the target with the camera detection, we can project the 3D target detection box in the world coordinate system onto the image plane and enclose the detection box with a minimum bounding rectangle, which is the image detection box. From the target's velocity, we can calculate the orientation of the image detection box ψ i :

[0136]

[0137] From the target direction, center point position (x i ,y i ,z i ) and size (l i ,w i ,h i ) can calculate the distance p from the center point to the corner point in the 3D detection box i '=(x',y',z'), where:

[0138]

[0139] where x′ i ±(l i / 2)cos(ψ i ) term and y′ i ±(l i / 2)sin(ψ i ) have the same sign, x′ i (w i / 2)sin(ψ i ) term and y′ i (w i / 2)cos(ψ i) items have opposite signs. Add the distance to the center point to get eight 3D detection box corner points p i :

[0140] p i =(x i ,y i , z i )+p′ i ,

[0141] Then it needs to be transformed into the image plane and normalized to homogeneous coordinate form through the external parameter matrix E and the internal parameter matrix K:

[0142]

[0143] The coordinates of the upper left corner (x0, y0) and the lower right corner (x1, y1) of the 2D image detection box can be calculated through the coordinates of the eight corner points projected onto the image plane:

[0144] x0=min{u i |i=1,2,...,8}

[0145] y0=min{v i |i=1,2,...,8}

[0146] x1=max{u i |i=1,2,...,8}

[0147] y2=max{v i |i=1,2,...,8}

[0148] Through the established EKF model, the target tracking system regards the existing trajectory as a state quantity, regards the matched detection result as an observation, and uses the observation to continuously update the existing trajectory. Corresponding to the previously established flag, the result from the radar initialization (radar initial trajectory) only uses the radar target detection result to update the model, and the result from the camera initialization (camera initial trajectory) only uses the camera target detection result to update the model. The fused trajectory can be matched with the detection results of the two sensors at the same time. When the simultaneous matching is successful, the fused trajectory uses the radar and camera observations to update the trajectory together, that is, the two detection results are input into the model as observations for EKF update, and a more accurate trajectory result is obtained. At the same time, the established EKF nonlinear model allows the system to update the target using only the observation from a single sensor. When the radar or camera misses the detection, the observation of another sensor can normally update the current trajectory independently. Therefore, the method of this embodiment is different from the existing common practice in that the method of the present application can still track when a certain sensor is absent. If the fused target only matches the detection result of one of the sensors, it can be updated using only a single detection result, thereby combating unreliable single-source observation data. For example, when a vehicle target suddenly stops, the radar may not detect the target due to the sudden change, but at this time it can only rely on the detection results of the camera, thereby improving the accuracy and stability of target tracking and better utilizing the characteristics of the sensor without causing multimodal interference problems.

[0149] Trajectories are continuously created, updated, and deleted in the target detection system. The target trajectory list during this period is the result of target tracking, corresponding to Figure 2 The target tracking system mainly performs three operations on the managed targets: adding new targets based on the detection results, updating existing targets with the current predicted matches, and deleting redundant existing targets. When the new detection result cannot match all the tracks in the list, the system will use the detection to create a new tracking target according to certain conditions. The system will continue to match the two different tracks marked as non-fused types. When the tracks created by the radar and the camera meet the conditions for continuous matching, the system will merge the two tracks to create a new fused track. When a fused track maintained in the list continues to not match a target for a period of time, the system will abandon the maintenance of the track and delete the track from the list. The track that is continuously maintained in the system is the tracking result.

[0150] In one embodiment, the updating of the fused trajectory by using the radar target detection result and the camera target detection result includes:

[0151] Setting different weight coefficients for the position information, speed information and size information of the radar target in the radar target detection result to obtain a first set of weight coefficients;

[0152] Setting different weight coefficients for the position information of the camera target, the size information of the camera target, and the speed information calculated according to the position information of the camera target in the camera target detection result, respectively, to obtain a second set of weight coefficients;

[0153] Based on the first set of weight coefficients and the second set of weight coefficients, the fused trajectory is updated by using the radar target detection result and the camera target detection result.

[0154] In specific implementation, since 4D radar is better at measuring the position and speed of the target, the new model gives higher weight to the position and speed of the target from the 4D radar, while giving higher weight to the size state of the image target. The fused trajectory can be updated using observations from both sensors, making full use of more accurate measurements without setting too high covariance parameters for the fused target. Therefore, the characteristics of the sensor can be better utilized without causing multimodal interference problems.

[0155] In one embodiment, the method further comprises:

[0156] Counting the number of consecutive matches of the initialization track marked as a radar or the initialization track marked as a camera;

[0157] When the number of consecutive matches exceeds the second threshold, the trajectory is a credible target;

[0158] When the number of consecutive matches is lower than the second threshold, the track is considered an untrustworthy target and is deleted.

[0159] In one embodiment, the method further comprises:

[0160] Counting the number of consecutive mismatches of the fused trajectory, the initialization trajectory marked as a radar, or the initialization trajectory marked as a camera;

[0161] When both 4D radar data and camera data are missing, the time interval between the previous frame calculation and the next frame calculation is calculated using the number of consecutive mismatches;

[0162] When the number of consecutive mismatches exceeds a third threshold, the track is deleted.

[0163] In specific implementation, when the system manages the deletion of targets, the first ones are the targets that have just been initialized by the radar and camera. These initialized targets will also have an additional initialization count, which represents the number of consecutive matches of the track, and is used to confirm whether the track will remain in the target list for a long time. When a track is accidentally created due to a false detection, the target will not be continuously matched with subsequent detections, and the track will be quickly deleted by the system. When this track is matched continuously for multiple frames to reach a second threshold, such as three frames, the track will be confirmed by the system as a credible target and retained in the target list. This setting can also ensure that when a sensor detects and initializes the same target twice, subsequent matches will only be performed with one of them, resulting in the deletion of the other repeatedly created target.

[0164] The trajectory retained in the target list will also have a count, which represents the number of frames that the trajectory has not been matched continuously. This count (number of consecutive mismatches) can be used to calculate the time elapsed in the next frame when both sensors fail in a certain frame. For example, let the interval time of each frame be Δt, and the time interval with the previous frame is set to T when updating. If it matches every time, then T = Δt; if there is a frame where both sensor detection and trajectory are not matched, then the time interval between the previous frame calculation and the next frame calculation is T = 2Δt, and so on. The number of consecutive mismatches is related to the time interval T, which is a system input for updating the EKF model. The number of consecutive mismatches can also be used to delete a target when it has not been matched in the system for a long time. This third threshold can be set to five to ten frames. In addition, the system will set the maximum detection space range of the camera and radar, and targets outside this space union range in the world coordinate system will also be automatically deleted by the system.

[0165] Finally, these target tracking results can be sent to downstream applications of the intelligent transportation system for further processing, visualization and statistics of vehicle flow information in the image. For example, in the parking detection system, considering the characteristics of vehicles constantly stopping, this method can solve the problem of uncertain single-source observation data caused by radar characteristics and provide more accurate tracking results.

[0166] In the embodiment provided by the present invention, the real-time acquired 4D radar data and camera data are respectively used for target detection. According to the target detection results, the radar and camera respectively perform target initialization to obtain their respective corresponding initialization trajectories; during the tracking process, the tracking trajectories of the radar and camera that meet certain matching conditions are fused, and a new fused trajectory is obtained by combining their respective advantageous observations; the trajectory is updated during the observation process using the radar and camera observations, and a joint update is performed when it matches the observations of both sensors, and it can also be updated independently when a certain sensor is absent; the tracking target is maintained to obtain the final tracking result. Therefore, the design of the trajectory fusion method of the present application improves the accuracy, continuity and stability of target tracking, and can better utilize the characteristics of the sensor to effectively avoid the problem of multimodal interference.

[0167] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A target tracking method based on 4D radar and camera fusion, characterized in that: include: Performing target detection on the 4D radar data and the camera data acquired in real time, respectively, to obtain target detection results, wherein the target detection results include radar target detection results and camera target detection results; Matching the radar target detection result and the camera target detection result with the current trajectory respectively; When at least one of the radar target detection result and the camera target detection result does not match the current trajectory, performing target initialization based on the radar target detection result and the camera target detection result to obtain an initialization trajectory marked as radar and an initialization trajectory marked as camera; When the radar target detection result and the camera target detection result respectively match the current trajectory, updating the current trajectory using the target detection results; After the update, it is determined whether the initialization track marked as radar and the initialization track marked as camera meet the track fusion rule. If they meet the track fusion rule, the initialization track marked as radar and the initialization track marked as camera are fused to generate a fused track, and the initialization track marked as radar and the initialization track marked as camera are deleted. During the updating process, when the current trajectory is the initialization trajectory marked as radar, the initialization trajectory marked as radar is updated using the radar target detection result; when the current trajectory is the initialization trajectory marked as camera, the initialization trajectory marked as camera is updated using the camera target detection result; when the current trajectory is the fused trajectory, the fused trajectory is updated using at least one of the radar target detection result and the camera target detection result.

2. The target tracking method based on 4D radar and camera fusion according to claim 1 is characterized in that: The real-time acquired 4D radar data and camera data are respectively subjected to target detection to obtain target detection results, including: Preprocessing the 4D radar data acquired in real time to obtain radar point cloud data; Performing target detection after clustering the radar point cloud data to obtain the radar target detection result, wherein the radar target detection result includes position information, speed information and size information of the radar target; The camera data acquired in real time is subjected to target detection through an image target detection network to obtain the camera target detection result on the image plane, wherein the camera target detection result includes position information and size information of the camera target.

3. The target tracking method based on 4D radar and camera fusion according to claim 2 is characterized in that: The performing target initialization based on the radar target detection result and the camera target detection result to obtain an initialization track marked as a radar and an initialization track marked as a camera respectively includes: Perform EKF modeling based on the radar target detection result to obtain a radar EKF model; Initializing the radar EKF model according to the first initialization parameter to obtain the initialization trajectory marked as radar; Perform EKF modeling based on the camera target detection result to obtain a camera EKF model; The camera EKF model is initialized according to the second initialization parameter to obtain the initialization trajectory marked as the camera.

4. The target tracking method based on 4D radar and camera fusion according to claim 3 is characterized in that: Initializing the radar EKF model according to the first initialization parameter to obtain the initialization trajectory marked as radar includes: Setting a first variance for the position information of the radar target, setting a second variance for the speed information of the radar target, and setting a third variance for the size information of the radar target, wherein the third variance is greater than the first variance and the second variance; The position information of the radar target, the speed information of the radar target, the size information of the radar target, the first variance, the second variance, the third variance and the first noise parameter are used as the first initialization parameters, and the radar EKF model is initialized according to the first initialization parameters to obtain the initialized trajectory marked as radar.

5. The target tracking method based on 4D radar and camera fusion according to claim 3 is characterized in that: Initializing the camera EKF model according to the second initialization parameter to obtain the initialization trajectory marked as the camera includes: Obtaining an initialization speed of a camera target based on the camera target detection result of a current frame and the camera target detection result of a previous frame; estimating an initialization position and an initialization size of a camera target according to an initialization speed of the camera target; Setting a fourth variance for the initialization speed, setting a fifth variance for the initialization position, and setting a sixth variance for the initialization size; The initialization speed, the initialization position, the initialization size, the fourth variance, the fifth variance, the sixth variance and the second noise parameter are used as the second initialization parameters, the camera EKF model is initialized according to the second initialization parameters, and the initialization trajectory marked as the camera is obtained.

6. The target tracking method based on 4D radar and camera fusion according to claim 1, characterized in that: The trajectory fusion rules include: Calculate the IoU value of the 3D detection frame of the initialization track marked as the radar and the initialization track marked as the camera in the current frame and multiple consecutive frames adjacent to the current frame; When the IoU values ​​of the current frame and the multiple consecutive frames adjacent to the current frame are greater than a first threshold, the initialized track marked as the radar and the initialized track marked as the camera are fused.

7. The target tracking method based on 4D radar and camera fusion according to claim 2 is characterized in that: The updating of the fused trajectory by using at least one of the radar target detection result and the camera target detection result includes: When the 4D radar data is missing, updating the fusion trajectory using the camera target detection result; When the camera data is missing, updating the fusion trajectory using the radar target detection result; When both the 4D radar data and the camera data exist, and both the radar target detection result and the camera target detection result match the fused trajectory, the fused trajectory is updated using the radar target detection result and the camera target detection result.

8. The target tracking method based on 4D radar and camera fusion according to claim 7 is characterized in that: The updating of the fusion trajectory by using the radar target detection result and the camera target detection result includes: Setting different weight coefficients for the position information, speed information and size information of the radar target in the radar target detection result to obtain a first set of weight coefficients; Setting different weight coefficients for the position information of the camera target, the size information of the camera target, and the speed information calculated according to the position information of the camera target in the camera target detection result, respectively, to obtain a second set of weight coefficients; Based on the first set of weight coefficients and the second set of weight coefficients, the fused trajectory is updated by using the radar target detection result and the camera target detection result.

9. The target tracking method based on 4D radar and camera fusion according to claim 1, characterized in that: The method further comprises: Counting the number of consecutive matches of the initialization track marked as a radar or the initialization track marked as a camera; When the number of consecutive matches exceeds the second threshold, the trajectory is a credible target; When the number of consecutive matches is lower than the second threshold, the track is considered an untrustworthy target and is deleted.

10. The target tracking method based on 4D radar and camera fusion according to claim 1, characterized in that: The method further comprises: Counting the number of consecutive mismatches of the fused trajectory, the initialization trajectory marked as a radar, or the initialization trajectory marked as a camera; When both 4D radar data and camera data are missing, the time interval between the previous frame calculation and the next frame calculation is calculated using the number of consecutive mismatches; When the number of consecutive mismatches exceeds a third threshold, the track is deleted.

Images