A method for camera and 4D millimeter-wave radar data fusion and target detection

Abstract

This invention relates to a method for target detection through camera and 4D millimeter-wave radar data fusion, comprising the following steps: A camera acquires an image of the foreground, performing target perception on the environment, including personnel and vehicles; the 4D millimeter-wave radar illuminates the environment, outputting data on the target's distance, velocity, horizontal angle, and altitude; the camera and 4D millimeter-wave radar undergo joint calibration, using a spatiotemporal synchronization algorithm to synchronize the image data and the point cloud data acquired by the 4D millimeter-wave radar; after spatiotemporal synchronization, a neural network detects targets in the image, and a fusion decision algorithm determines the target information. This invention calibrates, fuses, and analyzes image target data with 4D millimeter-wave radar data to determine the spatial location and altitude information of the image target, not only achieving fusion detection but also avoiding safety issues caused by unclear target altitude.

Description

Technical Field

[0001] This invention relates to the technical fields of autonomous driving, target detection, and video surveillance, and in particular to a method for fusion of camera and 4D millimeter-wave radar data and target detection. Background Technology

[0002] Taking the field of autonomous driving as an example, the current mainstream perception fusion technology uses cameras, LiDAR, and millimeter-wave radar to fuse data.

[0003] Using camera image data alone cannot accurately obtain the spatial distance of the target; a single visual camera sensor has insufficient detection and recognition accuracy, poor stability, and a relatively short detection range in the longitudinal direction. The camera is also easily affected by factors such as light and weather, especially at night or on rainy days.

[0004] Using a combination of lidar and camera data fusion, the identification and positioning effects are relatively good. Although lidar can detect three-dimensional spatial information and has a higher accuracy, it is easily constrained by rainy or foggy weather conditions, has a very high cost, and is also prone to damage.

[0005] Millimeter-wave radar is less affected by lighting and weather conditions, has high stability, and offers high ranging accuracy and long range. However, current millimeter-wave radar has relatively low resolution, is sensitive to metal, resulting in poor identification performance and an inability to identify target features. Currently, 3D millimeter-wave radar is widely used in the market, which can output information such as target speed, distance, and horizontal angle, but cannot collect target height information. This data fusion method often fails to identify stationary targets or targets at high altitudes, posing certain security risks. Summary of the Invention

[0006] The present invention aims to address the shortcomings of the prior art by providing a method for camera and 4D millimeter-wave radar data fusion and target detection.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for camera and 4D millimeter-wave radar data fusion and target detection includes the following steps:

[0009] S1. The camera captures images of the front and performs target perception of the environment in front, including people and vehicles.

[0010] The S2 and 4D millimeter-wave radars illuminate the environment ahead. The 4D millimeter-wave radar outputs data on the target's range, speed, horizontal angle, and altitude in four dimensions.

[0011] The S3, camera, and 4D millimeter-wave radar are jointly calibrated, and the image data and point cloud data collected by the 4D millimeter-wave radar are synchronized in time and space through a spatiotemporal synchronization algorithm.

[0012] S4. After spatiotemporal synchronization, targets in the image are detected by a neural network, and the target information is determined by a fusion decision algorithm.

[0013] In step S3, the spatiotemporal synchronization algorithm used for the joint calibration of the camera and 4D millimeter-wave radar synchronizes data time using the timestamps of the camera and 4D millimeter-wave radar data. Specifically:

[0014] The camera and 4D millimeter-wave radar are spatially calibrated using a calibration board. A camera spatial coordinate system is constructed at the camera end, with the optical center of the camera lens as the origin of the camera spatial coordinate system. The X-axis and Y-axis are parallel to the horizontal and vertical axes of the image, respectively, and the Z-axis passes through the lens along the optical center outward.

[0015] A radar spatial coordinate system is constructed at the 4D millimeter-wave radar end, with the center of the radar transmission plane as the origin of the radar coordinate system. The X-axis and Y-axis are parallel to the X-axis and Y-axis of the camera coordinate system, respectively, and the Z-axis is perpendicular to the radar transmission plane.

[0016] The camera acquires image data of the calibration board, and the 4D millimeter-wave radar acquires spatial data of the calibration board, including distance, velocity, horizontal angle, and height. The position of the calibration board is moved multiple times to acquire multiple pairs of images and 4D millimeter-wave radar data. The relationship between the calibration board and each pair of images and 4D millimeter-wave radar data is corresponding. The 4D millimeter-wave radar data is dimensionality reduced on two defined spatial planes to obtain two sets of dimensionality-reduced radar data. Spatial calibration is performed using the dimensionality-reduced radar data and images.

[0017] The reduced-dimensional radar data is coupled with two defined spatial planes, namely:

[0018] The first set of dimensionality-reduced radar data calculates the distance from all radar target points to the center line of the radar transmission plane using the horizontal angle dimension in the data. Then, the 4D millimeter-wave radar data is projected onto the X / Y axis plane in the radar spatial coordinate system. The X / Y axis plane in this radar spatial coordinate system is a defined spatial plane, called the radar plane.

[0019] The second set of dimensionality-reduced radar data removes the height dimension from the data and projects the 4D millimeter-wave radar data onto the Earth's surface, which is a defined spatial plane called the Earth's surface.

[0020] Spatial calibration is performed using the reduced-dimensional radar data and images, following these steps:

[0021] P1. In the acquired multiple pairs of images and 4D millimeter-wave radar data, perform the following operations:

[0022] P11. Locate the calibration board in the camera image and record the pixel coordinates of the center of the calibration board. The pixel coordinates of the calibration board in multiple images form a pixel matrix.

[0023] P12. Locate the point cloud information of the calibration board in the 4D millimeter-wave radar, and reduce the dimensionality of this point in two defined planes to obtain its information in the two defined planes. After the point cloud coordinates of multiple calibration boards are reduced in dimensionality, they form a point cloud matrix in the two planes respectively.

[0024] P2. By calibrating the camera's intrinsic parameters in advance using Zhang Zhengyou's calibration method, the correspondence between image pixel coordinates and camera spatial coordinates can be obtained. Then, the homography transformation relationship between the image and the radar plane is calculated using the pixel matrix and the point cloud matrix in the radar plane. The homography transformation relationship between the image and the ground plane is also calculated using the pixel matrix and the point cloud matrix in the ground plane, thus completing the calibration work.

[0025] In step S4, the fusion decision algorithm is performed according to the following steps:

[0026] The camera and 4D millimeter-wave radar have completed spatiotemporal calibration, identified targets in the images and bounded them, and reduced the 4D millimeter-wave radar data to two specific planes, namely the radar plane and the ground plane.

[0027] By using the homography transformation relationship between the image and the radar plane, the point cloud in the radar plane is projected onto the image plane, and the point group Q1 that falls within the image recognition selection area is selected.

[0028] By using the homography transformation relationship between the image and the ground plane, the point cloud in the ground plane is projected onto the image plane, and points Q2 that fall within a certain range above and below the lower border of the image recognition box are selected.

[0029] If the points in Q1 and Q2 are traced back, and Q1 and Q2 respectively contain the projection points of the original 4D millimeter-wave radar point cloud data in two defined planes, then the point cloud in the 4D millimeter-wave radar is considered to be the target point corresponding to the image target, and data fusion can be performed.

[0030] In step S4, the target information includes the target's spatial location and the target's spatial altitude.

[0031] The spatial position of a target includes its distance, speed, and horizontal angle.

[0032] Because the height of point cloud data in 4D millimeter-wave radar cannot accurately represent the target's height information, calculations are needed to determine the target's spatial height. The specific steps are as follows:

[0033] In the point cloud where the image target is finally successfully mapped, its height is recorded; the pixel height of the target bounding box in the image is recorded; the pixel height of the point cloud relative to the bottom border of the target bounding box in the image is recorded when the point cloud is dimensionality reduced to the radar plane and projected onto the image. These values ​​can be calculated as follows:

[0034]

[0035] The actual spatial height of the target is obtained.

[0036] The beneficial effects of this invention are: This invention calibrates, fuses, and analyzes image target data with 4D millimeter-wave radar data to determine the spatial position and height information of the image target. This not only enables the function of fusion detection but also avoids security problems caused by unclear target height. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the process of the present invention;

[0038] The following will describe in detail, with reference to the accompanying drawings, embodiments of the present invention. Detailed Implementation

[0039] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are for illustrative purposes only and are not intended to limit the scope of the invention. The invention is described more specifically in the following paragraphs by way of example with reference to the accompanying drawings. The advantages and features of the invention will become clearer from the following description. It should be noted that the drawings are in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the invention.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0042] Based on breakthroughs in 4D millimeter-wave radar technology, 4D millimeter-wave devices are gradually appearing in the market, capable of outputting target velocity, distance, horizontal angle, and altitude information. The fusion of 4D millimeter-wave radar and camera data solves both the high cost problem of LiDAR and the lack of altitude information in 3D millimeter-wave radar and camera data fusion. This fusion of camera and 4D millimeter-wave radar data will become an important fusion detection method in 3D target detection and will experience rapid development.

[0043] This invention specifically proposes a method for data fusion and target detection using a camera and 4D millimeter-wave radar. The 4D millimeter-wave radar can output data on four dimensions of the target: distance, velocity, horizontal angle, and altitude. Figure 1 As shown, the specific process is as follows:

[0044] The camera captures images of the area ahead, enabling the perception of targets such as people and vehicles in the environment ahead; 4D millimeter-wave radar illuminates the environment ahead.

[0045] The camera and 4D millimeter-wave radar are jointly calibrated, and the image data and the point cloud data collected by the 4D millimeter-wave radar are synchronized in time and space through a spatiotemporal synchronization algorithm.

[0046] After spatiotemporal synchronization, targets in the image are detected by neural networks, and information such as the spatial position and height of the targets are determined by a fusion decision algorithm.

[0047] The spatiotemporal synchronization algorithm used for joint calibration of the camera and 4D millimeter-wave radar synchronizes data time through timestamps of the camera and 4D millimeter-wave radar data. Spatial calibration of the camera and 4D millimeter-wave radar is performed using a small calibration board, which is a black, 10cm*10cm grid board.

[0048] The camera acquires image data of the calibration board, while the 4D millimeter-wave radar acquires spatial data (distance, velocity, horizontal angle, and height) of the calibration board. The position of the small calibration board is moved multiple times, acquiring multiple pairs of images (map1, map2, map3, ..., mapN) and 4D millimeter-wave radar data (point1, point2, point3, ..., pointN). The calibration board corresponds to each pair of images and 4D millimeter-wave radar data; that is, the pixel data of the calibration board in map1 corresponds to the radar data in point1, the pixel data of the calibration board in map2 corresponds to the radar data in point2, and so on. The 4D millimeter-wave radar data is dimensionality-reduced on two defined spatial planes, resulting in two sets of dimensionality-reduced radar data. Spatial calibration is then performed using the dimensionality-reduced radar data and the images.

[0049] The reduced-dimensional radar data is coupled with two defined spatial planes, namely:

[0050] The first set of dimensionality-reduced radar data uses the horizontal angle dimension in the data to calculate the distance from all radar target points to the center line of the radar transmission plane (i.e., the Z-axis). Then, the 4D millimeter-wave radar data is projected onto the X / Y axis plane in the radar spatial coordinate system. The X / Y axis plane in this radar spatial coordinate system is one of two defined spatial planes, referred to as the radar plane.

[0051] The second set of dimensionality-reduced radar data removes the height dimension from the data, which is equivalent to projecting the 4D millimeter-wave radar data onto the Earth's surface. This Earth's surface is one of two defined spatial planes, referred to as the Earth's surface.

[0052] Spatial calibration is performed using the reduced-dimensional radar data and images, following these steps:

[0053] The following operations were performed on the acquired multiple pairs of images and 4D millimeter-wave radar data:

[0054] Locate the small calibration board in the camera images and record the pixel coordinates of the center of the small calibration board. The pixel coordinates of the small calibration board in multiple images form a pixel matrix M[pixel(u1,v1),pixel(u2,v2),pixel(u3,v3),...,pixel(uN,vN)];

[0055] Point cloud information of small calibration boards is found in the 4D millimeter-wave radar, and the information of this point is reduced in dimensionality in two defined planes to obtain its information in the two defined planes. The point cloud coordinates of multiple small calibration boards, after dimensionality reduction, form a point cloud matrix in each of the two planes: the point cloud matrix Prader[Prader1,Prader2,Prader3,...,PraderN] = Prader[point(x1,y1), point(x2,y2), point(x3,y3),...,point(xN,yN)] in the radar plane, and the point cloud matrix Pland[Pland1,Pland2,Pland3,...,PlandN] = Pland[point(x1,z1), point(x2,z2), point(x3,z3),...,point(xN,zN)] in the ground plane.

[0056] By calibrating the camera's intrinsic parameters in advance using Zhang Zhengyou's calibration method, the correspondence between image pixel coordinates and camera spatial coordinates can be obtained. Then, the homography transformation relationship between the image and the radar plane is calculated using the pixel matrix and the point cloud matrix in the radar plane, and the homography transformation relationship between the image and the ground plane is calculated using the pixel matrix and the point cloud matrix in the ground plane, thus completing the calibration process.

[0057] The fusion decision algorithm is performed according to the following steps:

[0058] The camera and 4D millimeter-wave radar have completed spatiotemporal calibration, identified targets in the images and bounded them, and reduced the dimensionality of the 4D millimeter-wave radar data to two defined planes (radar plane and ground plane).

[0059] By using the homography transformation relationship between the image and the radar plane, the point cloud in the radar plane is projected onto the image plane, and the point group Q1 that falls within the image recognition selection range is selected, assuming Q1 = {Prader1, Prader3, Prader6}.

[0060] By using the homography transformation relationship between the image and the ground plane, the point cloud in the ground plane is projected onto the image plane, and the points Q2={Pland2,Pland3} that fall within a small range above and below the lower border of the image recognition box are selected.

[0061] If we backtrack the points in Q1 and Q2, and Q1 and Q2 respectively contain the projection points of the original 4D millimeter-wave radar point cloud data in two defined planes, then we consider that the point cloud in the 4D millimeter-wave radar is the target point corresponding to the image target. That is, Prader3 in Q1 and Pland3 in Q2 both come from the same point cloud in the 4D millimeter-wave radar, and we consider that the point cloud is the point cloud corresponding to the image target.

[0062] Information such as the spatial location of the target and its spatial altitude includes:

[0063] The target's positional relationships include its distance, speed, and horizontal angle;

[0064] Because the height of point cloud data in 4D millimeter-wave radar cannot accurately represent the target's height information, it is necessary to calculate and determine the target's spatial height.

[0065] In the point cloud where the image target is finally successfully mapped, its height is recorded; the pixel height of the target bounding box in the image is recorded; the pixel height of the point cloud relative to the bottom border of the target bounding box in the image is recorded when the point cloud is dimensionality reduced to the radar plane and projected onto the image. These values ​​can be calculated as follows:

[0066]

[0067] The actual spatial height of the target is obtained.

[0068] The camera can be either a telephoto camera or a short-focus camera, which is fused with 4D millimeter-wave radar data to obtain two sets of fused results. These two sets of results are then combined. This method can greatly increase the target detection range and distance of the camera and 4D millimeter-wave radar.

[0069] This invention calibrates, fuses, and analyzes image target data with 4D millimeter-wave radar data to determine the spatial location and height information of the image target. This not only enables fusion detection but also avoids security problems caused by unclear target height.

[0070] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any improvements made using the inventive concept and technical solution of the present invention, or direct application to other occasions without modification, are all within the protection scope of the present invention.

Claims

1. A method for camera and 4D millimeter-wave radar data fusion and target detection, characterized in that, Includes the following steps: S1. The camera captures images of the area in front and performs target perception of the environment in front; The S2 and 4D millimeter-wave radars illuminate the environment ahead. The 4D millimeter-wave radar outputs data on the target's range, speed, horizontal angle, and altitude in four dimensions. The S3, camera, and 4D millimeter-wave radar are jointly calibrated, and the image data and point cloud data collected by the 4D millimeter-wave radar are synchronized in time and space through a spatiotemporal synchronization algorithm. The spatiotemporal synchronization algorithm used for joint calibration of the camera and 4D millimeter-wave radar synchronizes data time using timestamps from both the camera and 4D millimeter-wave radar. Specifically, the calibration board performs spatial calibration on both the camera and 4D millimeter-wave radar. A camera spatial coordinate system is constructed at the camera end, with the optical center of the camera lens as its origin. The X and Y axes are parallel to the horizontal and vertical axes of the image, respectively, and the Z axis extends outward from the optical center through the lens. Similarly, a radar spatial coordinate system is constructed at the 4D millimeter-wave radar end, with the center of the radar's transmission plane as its origin. The X and Y axes are parallel to the X and Y axes of the camera coordinate system, respectively, and the Z axis is perpendicular to the radar's transmission plane. The camera acquires image data of the calibration board, and the 4D millimeter-wave radar acquires spatial data of the calibration board, including distance, velocity, horizontal angle, and height. The position of the calibration board is moved multiple times to acquire multiple pairs of images and 4D millimeter-wave radar data. The relationship between the calibration board and each pair of images and 4D millimeter-wave radar data is corresponding. The 4D millimeter-wave radar data is dimensionality reduced on two defined spatial planes to obtain two sets of dimensionality-reduced radar data. Spatial calibration is performed using the dimensionality-reduced radar data and images. The dimensionality-reduced radar data is coupled to two defined spatial planes: The first set of dimensionality-reduced radar data uses the horizontal angle dimension to calculate the distance from all radar target points to the centerline of the radar transmission plane. This 4D millimeter-wave radar data is then projected onto the X / Y axis plane of the radar spatial coordinate system. This X / Y axis plane is a defined spatial plane, referred to as the radar plane. The second set of dimensionality-reduced radar data removes the height dimension and projects the 4D millimeter-wave radar data onto the Earth's surface. This Earth's surface is also a defined spatial plane, referred to as the Earth's surface. Spatial calibration is performed using the reduced-dimensional radar data and images, following these steps: P1. In the acquired multiple pairs of images and 4D millimeter-wave radar data, perform the following operations: P11. Locate the calibration board in the camera image and record the pixel coordinates of the center of the calibration board. The pixel coordinates of the calibration board in multiple images form a pixel matrix. P12. Locate the point cloud information of the calibration board in the 4D millimeter-wave radar, and reduce the dimensionality of this point in two defined planes to obtain its information in the two defined planes. After the point cloud coordinates of multiple calibration boards are reduced in dimensionality, they form a point cloud matrix in the two planes respectively. P2. By calibrating the camera's intrinsic parameters in advance using Zhang Zhengyou's calibration method, the correspondence between image pixel coordinates and camera spatial coordinates can be obtained. Then, the homography transformation relationship between the image and the radar plane is calculated using the pixel matrix and the point cloud matrix in the radar plane. The homography transformation relationship between the image and the ground plane is also calculated using the pixel matrix and the point cloud matrix in the ground plane, thus completing the calibration work. S4. After spatiotemporal synchronization, targets in the image are detected through a neural network, and the target information is determined through a fusion decision algorithm. The target information includes the target's spatial position and spatial height. Because the height of the point cloud data in the 4D millimeter-wave radar cannot accurately represent the target's height information, calculation is required to determine the target's spatial height. The specific steps are as follows: In the point cloud where the image target is finally successfully mapped, its height is recorded; the pixel height of the target bounding box in the image is recorded; the pixel height of the point cloud relative to the bottom border of the target bounding box in the image is recorded when the point cloud is dimensionality reduced to the radar plane and projected onto the image. These values ​​can be calculated as follows: ； The actual spatial height of the target is obtained.

2. The method for camera and 4D millimeter-wave radar data fusion and target detection according to claim 1, characterized in that, In step S4, the fusion decision algorithm is performed according to the following steps: The camera and 4D millimeter-wave radar have completed spatiotemporal calibration, identified targets in the images and bounded them, and reduced the 4D millimeter-wave radar data to two specific planes, namely the radar plane and the ground plane. By using the homography transformation relationship between the image and the radar plane, the point cloud in the radar plane is projected onto the image plane, and the point group Q1 that falls within the image recognition selection area is selected. By using the homography transformation relationship between the image and the ground plane, the point cloud in the ground plane is projected onto the image plane, and points Q2 that fall within a certain range above and below the lower border of the image recognition box are selected. If the points in Q1 and Q2 are traced back, and Q1 and Q2 respectively contain the projection points of the original 4D millimeter-wave radar point cloud data in two defined planes, then the point cloud in the 4D millimeter-wave radar is considered to be the target point corresponding to the image target, and data fusion can be performed.

3. The method for camera and 4D millimeter-wave radar data fusion and target detection according to claim 2, characterized in that, The spatial position of a target includes its distance, speed, and horizontal angle.

4. The method for camera and 4D millimeter-wave radar data fusion and target detection according to claim 3, characterized in that, In step S1, the targets being sensed include people and vehicles.

Images