A lightweight target tracking method integrating millimeter-wave radar and monocular camera

Abstract

The present invention discloses a lightweight target tracking method that integrates millimeter-wave radar and a monocular camera. The method comprises the following steps: jointly calibrating the millimeter-wave radar and the monocular camera; using motion parameters as a filtering basis, and using the Euclidean clustering algorithm to filter the point cloud data of the current frame detected by the millimeter-wave radar to collect the current image of the monocular camera and perform target detection and lane line detection, and obtain the target point cloud data within the current lane line; and using Kalman filtering to perform stable target tracking. The present invention uses the clustered radar data points as the target tracking objects, introduces visual detection results to observe the class to which the forward target point belongs when the initial frame and the number of forward target points change; and simultaneously integrates the detection information of the current lane line of the YOLOP network to achieve lightweight tracking of the forward target point within the same lane.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent assisted driving, and specifically relates to a lightweight target tracking method that integrates millimeter-wave radar and a monocular camera. Background Art

[0002] With the rapid development of technologies like sensors, connected vehicles, and artificial intelligence, the upgrading and transformation of intelligent vehicles has become a key approach to reducing accident rates. In particular, given the numerous blind spots and poor braking characteristics of construction vehicles, improving collision warning capabilities is crucial for reducing the accident rate.

[0003] Vehicles use a variety of sensors, intelligent algorithms, and connected vehicles to achieve real-time perception and precise positioning of targets ahead, enabling safe decision-making for assisted and even autonomous driving. Target detection and tracking, as a key technology for environmental perception, are also crucial for improving collision warning capabilities. Millimeter-wave radar and visual cameras are the two most commonly used sensors in this technology. Millimeter-wave radar has excellent environmental adaptability and strong penetration, accurately detecting information such as a target's speed, direction, and distance. However, due to its operating principle, it suffers from high detection noise, excessive clutter interference, and an inability to obtain target geometry and category information. Visual cameras are low-cost and can capture rich data, making them widely used for target identification and classification. However, they are susceptible to environmental factors such as lighting and weather.

[0004] Patent application number 202110351803.1 discloses a vehicle detection method based on radar signal and visual fusion. It primarily utilizes millimeter-wave radar and a monocular camera, integrated with the YOLOV3-MobileNet neural network to detect vehicles and track detected vehicles using the Deep-SORT tracking algorithm. This type of method primarily utilizes deep learning-based tracking algorithms such as Deep-SORT. These algorithms incorporate CNNs to extract surface features of the detected target, thereby increasing the network's robustness to loss and obstacles. However, the inclusion of these features hinders real-time detection and tracking on embedded devices with limited computing power. Patent application number 201911095047.X discloses an ADAS front collision warning method based on monocular vision fused with millimeter-wave radar. This method primarily calculates the actual distance to the vehicle ahead using a monocular camera and fuses this information with the distance information obtained from the millimeter-wave radar. This fused information is then used to provide a warning. However, this method only uses the fusion of the two sensors for distance measurement and ignores the issue of real-time tracking of the same vehicle. Summary of the Invention

[0005] In order to solve the problem of real-time tracking of the front target in the same lane while the car is driving, the present invention proposes a lightweight target tracking method that integrates millimeter-wave radar and a monocular camera.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A lightweight target tracking method that integrates millimeter-wave radar and a monocular camera includes the following steps: jointly calibrating the millimeter-wave radar and the monocular camera; using motion parameters as a filtering basis and using a Euclidean clustering algorithm to filter point cloud data of the current frame detected by the millimeter-wave radar; capturing the current image of the monocular camera and performing target detection and lane line detection, and obtaining target point cloud data within the current lane line; and performing stable target tracking using a Kalman filter.

[0008] Preferably, the joint calibration includes the following steps: calibrating the monocular camera and obtaining a conversion model between the world coordinate system and the pixel coordinate system; converting the millimeter wave radar coordinate system to the world coordinate system; and synchronizing the sampling frequencies of the millimeter wave radar and the monocular camera.

[0009] Preferably, the point cloud data detected by the millimeter wave radar includes information of a plurality of scanning points, and any scanning point information includes position coordinates (x, y) and relative speed v. The screening includes the following steps:

[0010] Calculate the Euclidean distance D of each scan point;

[0011] Determine whether the Euclidean distance D is not greater than the empirical threshold. If so, retain it, otherwise discard it.

[0012] Preferably, capturing the current image with a monocular camera and performing target detection and lane line detection includes the following steps: obtaining target categories and lane line semantic segmentation images through a trained YOLOP network model; parsing the lane line semantic segmentation image using a density clustering algorithm, and extracting the two lane lines in the lane where the vehicle is located.

[0013] Preferably, the acquisition of target point cloud data within the current lane line includes the following steps: fusing the filtered point cloud data, the two extracted lane lines and the current image; eliminating the point cloud data in the non-current lane line area; and acquiring the target point information fused in the current image based on the target area.

[0014] Preferably, the target stable tracking comprises the following steps:

[0015] Pre-determination step: determine whether the current image is the initial image. If so, assign an initial ID to the target point of the initial image and output the target category. Otherwise, execute the consistency determination step;

[0016] The consistency determination step determines whether the number of target points in the current image is the same as that in the previous frame. If so, the similarity matching step is performed. Otherwise, a new ID is assigned to the current target point and the target category is output.

[0017] Similarity matching: The target point information of the previous frame image is predicted through the Kalman filter algorithm, and the similarity between the predicted result and the current target point information is calculated. If the similarity is greater than the threshold, the ID remains unchanged and the Kalman state equation is updated. Otherwise, a new ID is assigned to the current target point and the target category is output.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] This method uses clustered point cloud data as the target tracking object. When the initial frame and the number of forward target points change, the visual detection results (target detection network) are used to determine the class of the forward target point. Simultaneously, lane line detection is used to obtain detection information about the current lane line, enabling tracking of forward target points within the same lane. This method integrates the YOLOP network model, which includes both target detection and lane line detection, to achieve lightweight tracking of forward target points within the same lane. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of the present invention.

[0021] Figure 2 Flowchart of an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0023] The present invention implements a lightweight target tracking method that integrates millimeter-wave radar and monocular camera based on the RK1808 embedded platform. The embedded platform uses Rockchip's AI edge computing chip RK1808 as the main controller. The chip adopts a dual-core Cortex-A35 architecture, with an NPU peak computing power of up to 3.0, a VPU that supports 1080P video encoding and decoding, supports camera video signal input and has a built-in ISP. The monocular camera adopts the Weikong ES-DC1080ADAS, with a horizontal resolution of 1280 and a vertical resolution of 720. The millimeter-wave radar adopts the medium- and long-range millimeter-wave radar product CTLRR-320 developed by Chengtai Technology Co., Ltd. The product uses the CAN bus as the data interaction channel, with a refresh period of 50ms and a baud rate of 500kbps.

[0024] Reference Figure 1 and Figure 2 A lightweight target tracking method that integrates millimeter-wave radar and a monocular camera is proposed. The method specifically includes the following four steps.

[0025] Step 1: Jointly calibrate the millimeter-wave radar and monocular camera. The following three sub-steps are included.

[0026] Calibrate the monocular camera and obtain the transformation model between the world coordinate system and the pixel coordinate system;

[0027] (1-1) The present invention uses the checkerboard calibration method proposed by Zhang Zhengyou to calibrate the monocular camera to obtain the camera's intrinsic parameter matrix and extrinsic parameter matrix, and then derives the relationship between the world coordinate system and the pixel coordinate system according to formula (1):

[0028]

[0029] Where Z is the main axis of the camera; (u, v) represents the pixel coordinate system; (x, y) represents the camera coordinate system; (c x , c y ) represents the center point coordinate of the camera coordinate system; f represents the focal length; R is the rotation matrix; t is the translation matrix; (X w , Y w , Z w ) represents the world coordinate system. c x 、c y and the distortion coefficient K together constitute the camera intrinsic parameter matrix, and R and t constitute the extrinsic parameter matrix.

[0030] (1-2) Convert the millimeter wave radar coordinate system to the world coordinate system. The millimeter wave radar detection scanning plane is a two-dimensional plane, and the target coordinate information (x, y) can be obtained. Here, the coordinate information does not include the target height z. The coordinate system is established with the radar plane, and the millimeter wave radar coordinate system O t-X t Y t Z t To the world coordinate system O w -X w Y w Z w conversion.

[0031] (1-3) Synchronize the sampling frequencies of the millimeter-wave radar and monocular camera. Since the millimeter-wave radar and camera need to be synchronized within a time range, and in practical applications, the millimeter-wave radar's sampling period is slower than that of the camera, the millimeter-wave radar's acquisition frequency is used as the benchmark. For example, starting with the first identical timestamp after the start of recording in the radar data frame acquired by the millimeter-wave radar and the visual frame acquired by the camera, the radar data is sampled every 1 frame, and the video image data is sampled every 2 frames. This completes the temporal synchronization of the two sensors.

[0032] Step 2: Using the motion parameters as a filtering benchmark, the point cloud data of the current frame detected by the millimeter-wave radar is filtered using the Euclidean clustering algorithm.

[0033] In step 2 of the present invention, the point cloud data detected by the millimeter wave radar includes information of a plurality of scanning points, and any scanning point information includes position coordinates (x, y) and relative velocity v. The screening includes the following steps:

[0034] (2-1) Calculate the Euclidean distance of each scan point Among them, (x α ,y α , v α ) represents the information of the αth scanning point, α is the scanning point index, and (x0, y0, v0) represents the information of the cluster center;

[0035] (2-2) Determine whether the Euclidean distance D is not greater than the empirical threshold th. If so, classify the corresponding scanning points in the current frame; otherwise, discard them.

[0036] Step 3: Use the trained YOLOP network model to perform target detection and lane line detection on the current image captured by the monocular camera, and obtain the target point cloud data within the current lane line. Step 3 of the present invention includes the following steps:

[0037] (3-1) Obtain the target category and lane line semantic segmentation image through the trained YOLOP network model.

[0038] (3-2) Use the density clustering algorithm to analyze the lane line semantic segmentation image and extract the two lane lines in the lane where the vehicle is located.

[0039] As a network solution for panoramic driving perception, the YOLOP network can process three perception tasks in parallel: target detection, drivable area segmentation, and lane line detection. It should be noted that the specific structure of the YOLOP network model is common knowledge in this field and will not be described in detail here. The YOLOP network model is trained using images and corresponding lane line labels in the BDD100K open source dataset. Each frame of image captured by the monocular camera is used as the input of the trained YOLOP network model. The target category is obtained through the target detection network branch, and the corresponding lane line semantic segmentation image is obtained through the lane line detection network branch. All lane line coordinates in this frame of lane line semantic segmentation image are clustered by MeanShift. The vector form of the MeanShift clustering algorithm is shown in Equations (2) and (3).

[0040]

[0041]

[0042] Among them, p represents the lane line pixel coordinates (x, y) after semantic segmentation, and its grayscale value is 255. i Indicates the coordinates (x, y) of the discrete point that needs to be classified. j represents the coordinates (x, y) of the points already in the cluster; G(p) is a unit kernel function; H is a positive symmetric d*d matrix, called the bandwidth matrix, which is a diagonal matrix. w(p i )≥0,w(p i ) is the weight of each sample. The form of the diagonal matrix H is

[0043] Since the present invention is based on point cloud data tracking within the same lane, other point cloud data that are not within the current lane line are filtered out, and the lane line coordinates are fitted using the least squares method to obtain the coefficients (a, b, c) of the quadratic equation. By substituting the coefficients into the quadratic equation, the two lane lines in the lane where the vehicle is located can be extracted.

[0044] (3-3) Fusing the filtered point cloud data, the two extracted lane lines, and the current image; Here, fusing the point cloud data and the current image is common knowledge in the field, and those skilled in the art can set it up according to actual conditions, which will not be elaborated here.

[0045] (3-4) Eliminate point cloud data in areas other than the current lane line.

[0046] (3-5) Based on the target area, the target point information fused in the current image is obtained.

[0047] In the present invention, the area between the two lane lines is the target area, the scanning point mapped in the target area of ​​the current image is the target point, and the scanning point information is the target point information.

[0048] Step 4, performing stable target tracking through Kalman filtering, specifically includes the following steps.

[0049] (4-1) Pre-determination step: determine whether the current image is the initial image. If so, assign an initial ID to the target point (x, y, v) of the initial image and output the target category. Otherwise, perform the consistency determination step; the target category output here is the target category output by the target detection branch in the trained YOLOP network model.

[0050] (4-2) Consistency determination step: determine whether the number of current target points in the current image is consistent with the number of target points in the previous frame image. If so, perform the similarity matching step; otherwise, assign a new ID to the current target point and output the target category.

[0051] (4-3) Similarity matching: The target point information of the previous frame image is predicted through the Kalman filter algorithm, and the similarity between the predicted result and the current target point information is calculated. If the similarity is greater than the threshold, the ID remains unchanged and the Kalman state equation is updated. Otherwise, a new ID is assigned to the current target point and the target category is output.

[0052] Here, the “calculation of the similarity between the prediction result and the current target point information” is as follows:

[0053]

[0054] Where (x1, y1, v1) identifies the current target point, and (x2, y2, v2) represents the predicted result. The larger D is, the lower the similarity. If D > 5, the similarity is not greater than the threshold, and a new ID needs to be assigned to the current target point and the corresponding target category output.

[0055] While driving, the vehicle primarily performs lightweight tracking of targets ahead, ultimately providing real-time safe distance warnings. This approach uses a combination of millimeter-wave radar and a monocular camera to acquire forward target data. The millimeter-wave radar scans moving targets in the forward environment, while the monocular camera fuses and matches the radar's feedback point cloud data, defining the point cloud data's category (such as vehicle, person, or other target detection and classification results). The monocular camera image is processed through a network for target detection and lane line detection, and then fused with the millimeter-wave radar data to obtain forward target information within the current lane. Based on the principle that targets in consecutive frames do not change suddenly, a Kalman filter algorithm is used to predict these target points. The predicted results are then matched with the target point information obtained in the current frame. The target category and assignment are determined based on the target detection results. This approach avoids traditional tracking algorithms or depth-based tracking algorithms. Instead, it tracks point cloud data points and leverages the target detection results and lane line semantic segmentation results to achieve lightweight tracking of targets within the lane, improving computational speed and reducing tracking processing time.

Claims

1. A lightweight target tracking method integrating millimeter-wave radar and monocular camera, characterized in that: The method comprises the following steps: Jointly calibrate the millimeter-wave radar and monocular camera; Using motion parameters as the filtering benchmark, the point cloud data of the current frame detected by the millimeter-wave radar is filtered using the Euclidean clustering algorithm; Perform target detection and lane line detection on the current image captured by the monocular camera, and obtain the target point cloud data within the current lane line; The stable tracking of the target by Kalman filtering includes the following steps: Pre-determination step: determine whether the current image is the initial image. If so, assign an initial ID to the target point of the initial image and output the target category. Otherwise, execute the consistency determination step; The consistency determination step determines whether the number of target points in the current image is the same as that in the previous frame. If so, the similarity matching step is performed. Otherwise, a new ID is assigned to the current target point and the target category is output. Similarity matching: The target point information of the previous frame image is predicted through the Kalman filter algorithm, and the similarity between the predicted result and the current target point information is calculated. If the similarity is greater than the threshold, the ID remains unchanged and the Kalman state equation is updated. Otherwise, a new ID is assigned to the current target point and the target category is output.

2. The lightweight target tracking method integrating millimeter-wave radar and monocular camera according to claim 1, characterized in that: The joint calibration includes the following steps: Calibrate the monocular camera and obtain the transformation model between the world coordinate system and the pixel coordinate system; Convert the millimeter-wave radar coordinate system to the world coordinate system; Synchronize the sampling frequency of millimeter-wave radar and monocular camera.

3. The lightweight target tracking method integrating millimeter-wave radar and monocular camera according to claim 1, characterized in that: The point cloud data detected by the millimeter wave radar includes a plurality of scanning point information, and any scanning point information includes position coordinates (x, y) and relative speed v. The screening includes the following steps: Calculate the Euclidean distance D between each scan point and the cluster center; Determine whether the Euclidean distance D is not greater than the empirical threshold. If so, retain it, otherwise discard it.

4. The lightweight target tracking method integrating millimeter-wave radar and monocular camera according to claim 1, wherein: Performing object detection and lane line detection on the current image captured by the monocular camera includes the following steps: Obtain the semantic segmentation image of target categories and lane lines through the trained YOLOP network model; The lane line semantic segmentation image is analyzed using a density clustering algorithm to extract the two lane lines in the lane where the vehicle is located.

5. The lightweight target tracking method integrating millimeter-wave radar and monocular camera according to claim 4, characterized in that: The acquisition of target point cloud data within the current lane line includes the following steps: Fuse the filtered point cloud data, the two extracted lane lines, and the current image; Eliminate point cloud data in areas other than the current lane line; Obtain target point information fused in the current image based on the target area.

Images