Traffic multi-element radar vision fusion detection method

Abstract

A multi-element radar-visual fusion detection method for traffic targets acquires raw image data of traffic roads through a camera and radar point cloud data through millimeter-wave radar. The raw point cloud data detected by the radar is recoded into a grid image. A neural network is used to perform target detection on the grid image and the camera image. In the radar coordinate system, the target length, target width, and velocity information detected by the radar and camera are represented as the length, width, and height of a three-dimensional feature cube. The intersection-union ratio (IUGR) of the three-dimensional feature cube is calculated, and the target correlation degree is calculated based on the IUGR. The correlated target information is then fused using two-source information to remove false targets and redundant information. This invention fully leverages the advantages of both cameras and millimeter-wave radar, enabling the detection of richer and more accurate traffic target information, and more accurately and quickly associating targets, meeting the needs of intelligent transportation applications and providing strong support for the efficient operation of intelligent transportation systems and traffic safety.

Description

Technical Field

[0001] This invention relates to the field of traffic radar technology, and in particular to a traffic multi-element radar visual fusion detection method. Background Technology

[0002] In the field of intelligent transportation, accurate and efficient detection of traffic targets is crucial for achieving intelligent traffic management and ensuring road traffic safety. Traditional traffic target detection methods typically employ single-sensor data, such as camera images or radar point clouds. However, due to the complexity and variability of road traffic scenarios, a single sensor often cannot meet the requirements for accuracy and robustness. Camera images are affected by factors such as lighting and weather, which may result in shadows and blurring, making accurate target detection difficult. While radar point clouds can provide distance and velocity information, they are relatively weak in target classification and detail recognition.

[0003] In recent years, with the increasing complexity of traffic scenarios and the rising demands for traffic target recognition and tracking, the need for multi-sensor fusion has gradually increased. However, current multi-sensor fusion methods still face many challenges. Data from different sensors may exhibit temporal and spatial inconsistencies, affecting the fusion results. Designing suitable data fusion algorithms is crucial to integrating data from different sensors, extracting useful information, and eliminating redundancy. Traffic target recognition and tracking typically require high real-time performance, and multi-sensor fusion further complicates data processing, placing higher demands on computing resources and processing speed.

[0004] The statements herein provide only background information in relation to this invention and do not necessarily constitute prior art. Summary of the Invention

[0005] The purpose of this invention is to provide a traffic multi-element radar-visual fusion detection method that comprehensively applies both camera and millimeter-wave radar sensors, giving full play to their respective advantages. This method can detect richer and more accurate traffic target information, more accurately and quickly associate targets, meet the needs of intelligent transportation applications, and provide strong support for the efficient operation of intelligent transportation systems and traffic safety.

[0006] To achieve the above objectives, the present invention provides a traffic multi-element radar visual fusion detection method, comprising the following steps:

[0007] Step S1: Collect raw image data of traffic roads through a camera and record the collection time. At the same time, collect point cloud position, velocity, cross section (RCS), and collection time through millimeter-wave radar to form radar point cloud data. Calculate the coordinate transformation matrix of the camera and radar, unify the pixel coordinates into the radar coordinate system, and align the data time axis.

[0008] Step S2: Recode the raw point cloud data detected by the radar into a grid image, and use a neural network to perform target detection on the grid image to obtain the position, velocity and size of the radar target;

[0009] Step S3: Use a neural network to perform target detection on the camera image, extract the position, speed, and size of the target from the image, and detect the category, color, and brand. Use the coordinate transformation matrix obtained in step S1 to transform the target detection information of the camera image into the radar coordinate system.

[0010] Step S4: In the radar coordinate system, the target length, target width and velocity in the target information detected by the radar and camera are represented as the length, width and height of a three-dimensional feature cube. The crossover ratio of the three-dimensional feature cube is calculated, and the target correlation degree is calculated based on the crossover ratio.

[0011] Step S5: Perform two-source information fusion on the associated target information, remove false targets and redundant information, and convert it into video for display on the user interface.

[0012] In step S1, a 4D millimeter-wave radar and a 4K resolution camera are selected as sensors to acquire information about the traffic scene. The radar and camera are fixed at close positions on the traffic poles to achieve overlap of the field of view and cover the entire traffic area to be observed.

[0013] In step S1, the Zhang calibration algorithm is used to calibrate the camera, estimate its intrinsic and extrinsic parameters, and use the angle reflection to calibrate the extrinsic parameters of the radar and video equipment. By repeatedly detecting the coordinates of the angle reflection in the radar and video coordinate systems, the coordinate transformation matrix that transforms the points in the camera coordinate system to the radar coordinate system is calculated, and the pixel coordinates are unified into the radar coordinate system.

[0014] In step S1, NTP is used for time synchronization to ensure that the time of the radar and video equipment is synchronized with the NTP time.

[0015] In step S2, the radar point cloud data is converted into a grid with a fixed height h, width w, and cell size s. Each cell of the grid contains three channels, which represent the radar point's RCS, x-direction velocity, and y-direction velocity, respectively. If multiple radar points fall into the same cell, the three channel values ​​of that cell are the average of these radar points. If a cell is not mapped to a radar point, the three channel values ​​of that cell will be set to zero.

[0016] In step S2, the YOLOv5 neural network is used to detect two-dimensional objects in the radar grid image. During the training of the YOLOv5 neural network, each vehicle is divided into m classes according to its size. The YOLOv5 neural network will output B predicted target boxes. Each target box contains the x and y positions of the target center, height h, width w, and corresponding class information. It also contains the probability of the target's class and the target confidence. The average speed within the nearest a grid cells to the target center is calculated as the target's measured speed.

[0017] In step S3, the detected vehicle target is used as input, and the YOLOv5 neural network is used to detect two-dimensional objects in the camera image, thereby realizing the detection and localization of vehicle targets in the image. The neural network will output C predicted target boxes. Each target box contains the x and y positions of the target center, height h, width w, and corresponding category information, as well as the probability of the target's category and the target confidence level. Between adjacent video frames, the Lucas-Kanade optical flow algorithm is used to track the pixels of the target box, and the target's motion speed is estimated based on the pixel displacement.

[0018] In step S4, the horizontal direction of the road plane is used as the x-axis, the vertical direction as the y-axis, and the speed corresponding to the target as the z-axis, so that the target information detected by the radar and camera is represented as a three-dimensional feature cube.

[0019] For the target information detected by the radar, the coordinates (Px1, Py1) of the bottom center point of each target box are used as the position of the radar feature cube CU1, the length Δy1 and width Δx1 of the target box are used as the length L1 and width W1 of the feature cube, and the velocity v1 corresponding to the target box is used as the height H1 of the feature cube.

[0020] For the target information detected by the camera, the coordinates (Px2, Py2) of the bottom center point of each target box in the radar coordinate system are used as the position of the camera feature cube CU2, the length Δy2 and width Δx2 of the target box in the radar coordinate system are used as the length L2 and width W2 of the feature cube, and the velocity v2 corresponding to the target box is used as the height H2 of the feature cube.

[0021] The correlation between targets is calculated based on the Intersection over Union (IOU) ratio of the feature cubes:

[0022]

[0023] Where U is the union of the radar-view feature cubes and I is the intersection of the radar-view feature cubes;

[0024] When the IOU is greater than the threshold th, the radar and video detection targets are considered to be related targets.

[0025] In step S5, target pairs with high correlation are selected based on target correlation and are considered as real correlated targets; radar targets with low correlation are excluded as false targets; when multiple adjacent radar and video targets are correlated with each other, they are merged and deduplicated as duplicate targets, and only the target with the highest confidence is retained.

[0026] In step S5, for the target corresponding to the associated target, the position and speed information detected by the radar are used as the true position and speed of the target, and the size, color and brand attributes detected by the camera are added to the target attributes. All the target attribute information and motion state are converted into video and displayed intuitively on the user interface.

[0027] This invention employs both cameras and millimeter-wave radar for road traffic target detection. The raw point cloud data detected by radar is recoded into a grid image. A neural network is used to simultaneously detect targets in both the radar grid image and the camera image, obtaining information such as the target's position, speed, and size. For the camera image, attributes such as category, color, and brand can be detected. The target's length, width, speed, and size are represented as the three-dimensional feature length of a feature cube. By calculating the intersection-union ratio (IUU) between the radar-viewed feature cubes, the correlation between targets is obtained, achieving faster and more effective target association. Finally, the target attribute information of the associated radar-viewed targets is integrated, false targets and redundant information are removed, and the information is displayed on the user interface, allowing users to intuitively understand the detected traffic target information. Attached Figure Description

[0028] Figure 1 This is a flowchart of a traffic multi-element radar visual fusion detection method provided by the present invention.

[0029] Figure 2 This is a schematic diagram of the radar and camera installation in an embodiment of the present invention. Detailed Implementation

[0030] The following is based on Figure 1 and Figure 2 The preferred embodiments of the present invention will be described in detail below.

[0031] like Figure 1 As shown, this invention provides a traffic multi-element radar visual fusion detection method, comprising the following steps:

[0032] Step S1: The camera and radar simultaneously acquire data.

[0033] Raw image data of traffic roads is acquired by a camera and the acquisition time is recorded. At the same time, radar point cloud data is acquired by millimeter-wave radar, including point cloud location, velocity, cross section (RCS), acquisition time, etc.

[0034] Calculate the extrinsic transformation matrix of the camera and radar, unify the pixel coordinates to the radar coordinate system, and align the data time axis to ensure that the data are in the same coordinate system and time axis.

[0035] In this embodiment, to obtain richer point cloud information and high-definition images, a 4D millimeter-wave radar and a 4K resolution camera are selected as sensors to acquire traffic scene information. For example... Figure 2 As shown, the radar and camera are fixed at close positions on traffic poles to achieve overlapping fields of view, covering the entire traffic area to be observed. A 4K resolution camera acquires raw image data of the traffic road and records the acquisition time, while a 4D millimeter-wave radar acquires radar point cloud data of the road scene, including point cloud position, velocity, RCS (radio cross section), and acquisition time. Zhang's calibration algorithm is used to calibrate the camera, estimating its intrinsic parameters (including focal length, principal point, etc.) and extrinsic parameters (camera position and attitude). Simultaneously, an inverted angle is used to calibrate the extrinsic parameters of the radar and video equipment. By repeatedly detecting the coordinates of the inverted angle in the radar and video coordinate systems, a coordinate transformation matrix is ​​calculated to convert points in the camera coordinate system to the radar coordinate system, unifying pixel coordinates to the radar coordinate system. To achieve time alignment, NTP (Network Time Protocol) is used for time synchronization, ensuring that the time of the radar and video equipment is synchronized with NTP time.

[0036] Step S2: Radar point cloud target detection.

[0037] The raw point cloud data detected by radar is recoded into a grid image, and a neural network is used to detect targets in the grid image to obtain information such as the position, velocity, and size of radar targets.

[0038] In this embodiment, the radar point cloud data is converted into a grid with a fixed height h, width w, and cell size s. By mapping the positions of radar points to the grid, the radar point cloud data can be converted into an image-like form; radar points exceeding the grid size are discarded. Each cell in the grid contains three channels, representing the radar point's radar cross-section (RCS), x-direction velocity, and y-direction velocity, respectively. If multiple radar points fall into the same cell, the three channel values ​​of that cell are the average of those radar points. If a cell is not mapped to any radar point, the three channel values ​​of that cell are set to zero to maintain the integrity and consistency of the grid.

[0039] Two-dimensional object detection in radar cell images was performed using the YOLOv5 neural network. In radar perception, the size of a target is related to the number of radar point clouds and the radar cross section (RCS), thus the target size can be determined. During network training, each vehicle was classified into m classes based on size. The network outputs B predicted bounding boxes, each containing the x and y positions of the target center, height h, width w, and corresponding class information, as well as the probability of the target belonging to its class and the target confidence score. Furthermore, the average velocity within the nearest a cells to the target center was calculated as the target's measured velocity.

[0040] Step S3: Camera image target detection.

[0041] Neural networks are used to detect targets in camera images, extracting information such as the target's position, speed, and size, and detecting attributes such as category, color, and brand.

[0042] Transform the camera target to the radar coordinate system to ensure that the data are in the same coordinate system.

[0043] In this embodiment, a YOLOv5 neural network is used to detect two-dimensional objects in camera images, enabling the detection and localization of targets in the image. The neural network outputs C predicted bounding boxes, each containing the x and y positions of the target center, height h, width w, and corresponding category information, as well as the probability of the target belonging to the category and the target confidence level. Subsequently, between adjacent video frames, the Lucas-Kanade optical flow algorithm is used to track the pixels of the bounding boxes, and the target's motion velocity is estimated based on the pixel displacement. Finally, the image target detection information is transformed into the radar coordinate system according to the coordinate transformation matrix calculated in step S1.

[0044] The detected vehicle targets are used as input, and the YOLOv5 neural network is used to detect vehicle targets in the image. The output of the YOLOv5 neural network contains multiple heads, each corresponding to a prediction of a target attribute. The output of the YOLOv5 neural network will contain multiple prediction results, each corresponding to a prediction of a target attribute, ultimately obtaining attribute information such as vehicle category, color, and brand.

[0045] Step S4: Calculate the association between the target and the radar.

[0046] In the radar coordinate system, the target length, target width, and velocity in the target information detected by the radar and camera are represented as the length, width, and height of a three-dimensional feature cube. The crossover ratio (CRI) of the three-dimensional feature cube is calculated, and the target correlation degree is calculated based on the CRI.

[0047] In this embodiment, the horizontal direction of the plane containing the road is used as the x-axis and the vertical direction as the y-axis. Specifically, the speed corresponding to the target is used as the z-axis, and the target information detected by the radar and camera is represented as a three-dimensional feature cube.

[0048] For the target information detected by the radar, the coordinates (Px1, Py1) of the bottom center point of each target box are used as the position of the radar feature cube CU1, the length Δy1 and width Δx1 of the target box are used as the length L1 and width W1 of the feature cube, and the velocity v1 corresponding to the target box is used as the height H1 of the feature cube.

[0049] For the target information detected by the camera, the coordinates (Px2, Py2) of the bottom center point of each target box in the radar coordinate system are used as the position of the camera feature cube CU2. The length Δy2 and width Δx2 of the target box in the radar coordinate system are used as the length L2 and width W2 of the feature cube. The velocity v2 corresponding to the target box is used as the height H2 of the feature cube.

[0050] Then, the correlation between targets is calculated based on the intersection-union ratio (IOU) of the feature cubes, using the following formula:

[0051]

[0052] Where U is the union of the radar-view feature cubes and I is the intersection of the radar-view feature cubes.

[0053] When the Intersection over Union (IOU) is greater than the threshold th, the radar and video detection targets are considered to be related targets. By calculating the intersection-union ratio of the feature cubes, fast and effective target association is achieved.

[0054] Step S5: Target two-source information fusion and video conversion.

[0055] The associated target information is fused from two sources to remove false targets and redundant information, and then converted into video for display on the user interface, so that users can intuitively understand the detected traffic target information.

[0056] In this embodiment, target pairs with high correlation are selected based on target correlation and are considered as truly correlated targets. Radar targets with low correlation are excluded as false targets. When multiple adjacent radar and video targets are correlated, they can be merged and deduplicated as duplicate targets, retaining only the target with the highest confidence level.

[0057] For targets corresponding to associated targets, the position and velocity information detected by radar are used as the target's true position and velocity, while the size, color, and brand attributes detected by the camera are added to the target attributes. All target attribute information and motion status are converted into video and displayed intuitively on the user interface, enabling users to better understand the target's environment and motion status, as well as their movement on the road, providing more comprehensive and accurate information for traffic monitoring and management.

[0058] This invention employs both cameras and millimeter-wave radar for road traffic target detection. The raw point cloud data detected by radar is recoded into a grid image. A neural network is used to simultaneously detect targets in both the radar grid image and the camera image, obtaining information such as the target's position, speed, and size. For the camera image, attributes such as category, color, and brand can be detected. The target's length, width, speed, and size are represented as the three-dimensional feature length of a feature cube. By calculating the intersection-union ratio (IUU) between the radar-viewed feature cubes, the correlation between targets is obtained, achieving faster and more effective target association. Finally, the target attribute information of the associated radar-viewed targets is integrated, false targets and redundant information are removed, and the information is displayed on the user interface, allowing users to intuitively understand the detected traffic target information.

[0059] It should be noted that, in the embodiments of the present invention, the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential," etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the accompanying drawings and are only for the convenience of describing the embodiments. They do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0060] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0061] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A traffic multi-element radar visual fusion detection method, characterized in that, Includes the following steps: Step S1: Collect raw image data of traffic roads through a camera and record the collection time. At the same time, collect point cloud position, velocity, cross section (RCS), and collection time through millimeter-wave radar to form radar point cloud data. Calculate the coordinate transformation matrix of the camera and radar, unify the pixel coordinates into the radar coordinate system, and align the data time axis. Step S2: Recode the raw point cloud data detected by the radar into a grid image, and use a neural network to perform target detection on the grid image to obtain the position, velocity and size of the radar target; Step S3: Use a neural network to perform target detection on the camera image, extract the position, speed, and size of the target from the image, and detect the category, color, and brand. Use the coordinate transformation matrix obtained in step S1 to transform the target detection information of the camera image into the radar coordinate system. Step S4: In the radar coordinate system, the target length, target width and velocity in the target information detected by the radar and camera are represented as the length, width and height of a three-dimensional feature cube. The crossover ratio of the three-dimensional feature cube is calculated, and the target correlation degree is calculated based on the crossover ratio. Step S5: Perform two-source information fusion on the associated target information, remove false targets and redundant information, and convert it into video for display on the user interface.

2. The traffic multi-element radar visual fusion detection method as described in claim 1, characterized in that, In step S1, a 4D millimeter-wave radar and a 4K resolution camera are selected as sensors to acquire information about the traffic scene. The radar and camera are fixed at close positions on the traffic poles to achieve overlap of the field of view and cover the entire traffic area to be observed.

3. The traffic multi-element radar visual fusion detection method as described in claim 2, characterized in that, In step S1, the Zhang calibration algorithm is used to calibrate the camera, estimate its intrinsic and extrinsic parameters, and use the angle reflection to calibrate the extrinsic parameters of the radar and video equipment. By repeatedly detecting the coordinates of the angle reflection in the radar and video coordinate systems, the coordinate transformation matrix that transforms the points in the camera coordinate system to the radar coordinate system is calculated, and the pixel coordinates are unified into the radar coordinate system.

4. The traffic multi-element radar visual fusion detection method as described in claim 3, characterized in that, In step S1, NTP is used for time synchronization to ensure that the time of the radar and video equipment is synchronized with the NTP time.

5. The traffic multi-element radar visual fusion detection method as described in claim 1, characterized in that, In step S2, the radar point cloud data is converted into a grid with a fixed height h, width w, and cell size s. Each cell of the grid contains three channels, which represent the radar point's RCS, x-direction velocity, and y-direction velocity, respectively. If multiple radar points fall into the same cell, the three channel values ​​of that cell are the average of these radar points. If a cell is not mapped to a radar point, the three channel values ​​of that cell will be set to zero.

6. The traffic multi-element radar visual fusion detection method as described in claim 5, characterized in that, In step S2, the YOLOv5 neural network is used to detect two-dimensional objects in the radar grid image. During the training of the YOLOv5 neural network, each vehicle is divided into m classes according to its size. The YOLOv5 neural network will output B predicted target boxes. Each target box contains the x and y positions of the target center, height h, width w, and corresponding class information. It also contains the probability of the target's class and the target confidence. The average speed within the nearest a grid cells to the target center is calculated as the target's measured speed.

7. The traffic multi-element radar visual fusion detection method as described in claim 1, characterized in that, In step S3, the detected vehicle target is used as input, and the YOLOv5 neural network is used to detect two-dimensional objects in the camera image, thereby realizing the detection and localization of vehicle targets in the image. The neural network will output C predicted target boxes. Each target box contains the x and y positions of the target center, height h, width w, and corresponding category information, as well as the probability of the target's category and the target confidence level. Between adjacent video frames, the Lucas-Kanade optical flow algorithm is used to track the pixels of the target box, and the target's motion speed is estimated based on the pixel displacement.

8. The traffic multi-element radar visual fusion detection method as described in claim 1, characterized in that, In step S4, the horizontal direction of the road plane is used as the x-axis, the vertical direction as the y-axis, and the speed corresponding to the target as the z-axis, so that the target information detected by the radar and camera is represented as a three-dimensional feature cube. For the target information detected by the radar, the coordinates (Px1, Py1) of the bottom center point of each target box are used as the position of the radar feature cube CU1, the length Δy1 and width Δx1 of the target box are used as the length L1 and width W1 of the feature cube, and the velocity v1 corresponding to the target box is used as the height H1 of the feature cube. For the target information detected by the camera, the coordinates (Px2, Py2) of the bottom center point of each target box in the radar coordinate system are used as the position of the camera feature cube CU2, the length Δy2 and width Δx2 of the target box in the radar coordinate system are used as the length L2 and width W2 of the feature cube, and the velocity v2 corresponding to the target box is used as the height H2 of the feature cube. The correlation between targets is calculated based on the Intersection over Union (IOU) ratio of the feature cubes: Where U is the union of the radar-view feature cubes and I is the intersection of the radar-view feature cubes; When the IOU is greater than the threshold th, the radar and video detection targets are considered to be related targets.

9. The traffic multi-element radar visual fusion detection method as described in claim 1, characterized in that, In step S5, target pairs with high correlation are selected based on target correlation and are considered as real correlated targets; radar targets with low correlation are excluded as false targets; when multiple adjacent radar and video targets are correlated with each other, they are merged and deduplicated as duplicate targets, and only the target with the highest confidence is retained.

10. The traffic multi-element radar visual fusion detection method as described in claim 9, characterized in that, In step S5, for the target corresponding to the associated target, the position and speed information detected by the radar are used as the true position and speed of the target, and the size, color and brand attributes detected by the camera are added to the target attributes. All the target attribute information and motion state are converted into video and displayed intuitively on the user interface.

Images