Millimeter wave radar detection method, detection device and storage medium

Abstract

The invention discloses a millimeter wave radar detection method and device and a storage medium, and relates to the technical field of radars, and the method comprises the steps: collecting the point cloud data of a to-be-detected millimeter wave radar and the video data of a camera; joint calibration is carried out on the to-be-detected millimeter wave radar and the camera, and a conversion matrix between the to-be-detected millimeter wave radar and the camera is determined; detecting and marking a target object in the video data to obtain truth value data; projecting the point cloud data through the conversion matrix to obtain a corresponding relation between the point cloud data and the truth value data; and adopting a maximum weight matching algorithm to match the truth value data and the point cloud data with the corresponding relationship, and determining the detection result of the millimeter-wave radar to be detected according to the obtained matching result, thereby improving the reliability of the detection result of the millimeter-wave radar. And the problem of low detection efficiency of manual operation is avoided.

Description

Technical Field

[0001] The present application relates to the field of radar technology, and in particular to a detection method, detection equipment and storage medium of a millimeter wave radar. Background Art

[0002] As an important sensor technology, millimeter wave radar is widely used in fields such as autonomous driving, drone navigation, industrial detection and environmental monitoring. Because millimeter wave radar can work normally in various adverse weather conditions (such as rain, fog, snow, etc.) and can accurately detect the distance, speed and angle information of objects, it has become an indispensable core sensing device for realizing advanced driver assistance systems (ADAS) and autonomous driving systems.

[0003] Conventional millimeter-wave radar detection rate tests are mostly based on manual operations. This leads to inconsistent statistical data results in complex outdoor scenes due to different statistical analysis standards of different testers, resulting in low reliability and low test efficiency of millimeter-wave radar detection results. Summary of the invention

[0004] The main purpose of this application is to provide a millimeter-wave radar detection method, detection equipment and storage medium, aiming to solve the technical problem of how to effectively improve the test reliability and test efficiency of millimeter-wave radar.

[0005] To achieve the above object, the present application proposes a millimeter wave radar detection method, which includes:

[0006] Collect the point cloud data of the millimeter-wave radar to be detected and the video data of the camera;

[0007] The millimeter-wave radar and camera to be detected are jointly calibrated to determine the conversion matrix between the millimeter-wave radar and the camera to be detected;

[0008] Detect and mark the target objects in the video data to obtain true value data;

[0009] Project the point cloud data through the transformation matrix to obtain the corresponding relationship between the point cloud data and the true value data;

[0010] The maximum weight matching algorithm is used to match the true value data and point cloud data with corresponding relationships, and the detection result of the millimeter-wave radar to be detected is determined based on the obtained matching result.

[0011] In one embodiment, the step of collecting point cloud data of the millimeter wave radar to be detected and video data of the camera includes:

[0012] Acquire in real time the point cloud data of several millimeter-wave radars to be detected, and acquire in real time the video data corresponding to the scene where the point cloud data is captured by the camera;

[0013] Among them, point cloud data and video data are stored according to timestamps.

[0014] In one embodiment, the step of jointly calibrating the millimeter-wave radar to be detected and the camera to determine the conversion matrix between the millimeter-wave radar to be detected and the camera includes:

[0015] Calibrate the internal parameters of the camera to obtain the camera's internal parameter matrix;

[0016] The external parameters of the millimeter-wave radar and camera to be detected are calibrated, and the rotation matrix and translation matrix between the millimeter-wave radar and camera to be detected are obtained by combining the internal parameter matrix;

[0017] Among them, the transformation matrix includes the intrinsic parameter matrix, the rotation matrix and the translation matrix.

[0018] In one embodiment, the step of detecting and marking the target object in the video data to obtain true value data includes:

[0019] Perform frame extraction on the video data to obtain several target scene graphs;

[0020] Use the preset deep learning target detection model to detect each target scene graph and mark the target object in each target scene graph;

[0021] Generate a bounding box containing each target object using a preset algorithm, and record the location, category, and confidence score of each target object to obtain detection data;

[0022] Associate the detection data with the timestamp to obtain the true value data.

[0023] In one embodiment, the step of projecting the point cloud data through the transformation matrix to obtain the corresponding relationship between the point cloud data and the true value data includes:

[0024] According to the scene graph timestamp corresponding to each target scene graph, the point cloud data corresponding to the scene graph timestamp is searched at the timestamp to obtain the target point cloud data;

[0025] Based on the intrinsic parameter matrix, rotation matrix and translation matrix, the target point cloud data is projected to obtain the point cloud coordinate position of the target point cloud data in the image coordinate system of the camera, and then the correspondence between the target point cloud data and the true value data is determined according to the point cloud coordinate position.

[0026] In one embodiment, the maximum weight matching algorithm includes a first maximum weight matching algorithm and a second maximum weight matching algorithm. The maximum weight matching algorithm is used to match the true value data and the point cloud data having a corresponding relationship, including:

[0027] An undirected graph is created by using a first maximum weight matching algorithm, and a corresponding relationship between the target point cloud data and a first true value bounding box of the true value data is obtained by using the undirected graph;

[0028] Traversing each target point cloud data, and determining whether each target point cloud data is within the first true value bounding box corresponding to the target point cloud data;

[0029] If the target point cloud data is within the first true value bounding box corresponding to the target point cloud data, the target point cloud data within the first true value bounding box corresponding to the target point cloud data is determined as the first point cloud data, a first distance between the first point cloud data and the center of the bounding box of the first true value bounding box corresponding to the first point cloud data is calculated, and a matching relationship between the first point cloud data and the true value data having a corresponding relationship is determined based on the first distance;

[0030] If the target point cloud data is not within the first true value bounding box corresponding to the target point cloud data, determining whether the target point cloud data is within a second true value bounding box established based on the first true value bounding box corresponding to the target point cloud data;

[0031] If the target point cloud data is within the second true value bounding box corresponding to the target point cloud data, the target point cloud data within the second true value bounding box corresponding to the target point cloud data is determined as the second point cloud data, and a second distance between the second point cloud data and the center of the bounding box of the second true value bounding box corresponding to the second point cloud data is calculated, and a matching relationship between the second point cloud data and the true value data having a corresponding relationship is determined based on the second distance.

[0032] In one embodiment, before the step of determining the detection result of the to-be-detected millimeter-wave radar according to the obtained matching result, the step further includes:

[0033] A test scenario is determined, and lane division and distance interval division are performed on the test scenario to obtain multiple lane partitions and distance partitions.

[0034] In one embodiment, the step of determining the detection result of the to-be-detected millimeter-wave radar according to the obtained matching result includes:

[0035] Determine the bounding box center of the first truth bounding box, the bounding box center of the second truth bounding box, the lane partition and the distance partition where each first point cloud data and each second point cloud data are located at each time stamp, and determine the total number of truth bounding boxes and the total number of point cloud data in each lane partition and each distance partition at each time stamp;

[0036] Divide the total number of point cloud data by the total number of true bounding boxes to obtain the detection rate of each lane partition and each distance partition at each timestamp, and determine the detection result based on the detection rate.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a detection device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the millimeter wave radar detection method as described above.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the millimeter wave radar detection method described above are implemented.

[0039] One or more technical solutions proposed in this application have at least the following technical effects:

[0040] By collecting the point cloud data of the millimeter-wave radar to be detected and the video data of the camera; jointly calibrating the millimeter-wave radar to be detected and the camera, determining the transformation matrix between the millimeter-wave radar to be detected and the camera; detecting and marking the target objects in the video data to obtain the true value data; projecting the point cloud data through the transformation matrix to obtain the corresponding relationship between the point cloud data and the true value data; using the maximum weight matching algorithm to match the true value data and the point cloud data with the corresponding relationship, and determining the detection result of the millimeter-wave radar to be detected based on the matching result.

[0041] This application combines the joint calibration results of the camera and millimeter-wave radar, that is, the transformation matrix, in the data matching process, projects the point cloud data into the image coordinate system, obtains the correspondence between the point cloud data and the true value data, realizes spatial alignment, and then uses the maximum weight matching algorithm to match the true value data and point cloud data with the corresponding relationship, thereby realizing high-precision matching operations of the point cloud data and the true value data, thereby improving the reliability of the detection results of the millimeter-wave radar. The above operations are all performed based on the detection equipment, avoiding the problem of low detection efficiency caused by manual operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A schematic diagram of a flow chart of a first embodiment of a detection method for a millimeter-wave radar of the present application;

[0045] Figure 2 This is a schematic diagram of the point cloud data of different millimeter-wave radars to be detected that are continuously collected by the detection equipment of this application;

[0046] Figure 3 A schematic diagram of the lane division and distance division for this application;

[0047] Figure 4 A statistical diagram of the detection rate of the ground truth bounding box for each lane partition and each distance partition for this application;

[0048] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the millimeter wave radar detection method in the embodiment of the present application.

[0049] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0051] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0052] The main solution of the embodiment of the present application is: collecting the point cloud data of the millimeter-wave radar to be detected and the video data of the camera; jointly calibrating the millimeter-wave radar to be detected and the camera, and determining the transformation matrix between the millimeter-wave radar to be detected and the camera; detecting and marking the target objects in the video data to obtain true value data; projecting the point cloud data through the transformation matrix to obtain the corresponding relationship between the point cloud data and the true value data; using the maximum weight matching algorithm to match the true value data and the point cloud data with the corresponding relationship, and determining the detection result of the millimeter-wave radar to be detected based on the matching result.

[0053] Conventional millimeter-wave radar detection rate tests are mostly based on manual operations. This leads to inconsistent statistical data results due to different statistical analysis standards of different testers in complex outdoor scenes, resulting in low reliability and low test efficiency of millimeter-wave radar detection results.

[0054] The present application provides a solution. During the data matching process, the joint calibration results of the camera and the millimeter-wave radar, that is, the transformation matrix, are combined to project the point cloud data into the image coordinate system, and the corresponding relationship between the point cloud data and the true value data is obtained to achieve spatial alignment. Then, a maximum weight matching algorithm is used to match the true value data and the point cloud data with the corresponding relationship, so as to achieve high-precision matching operations of the point cloud data and the true value data, thereby improving the reliability of the detection results of the millimeter-wave radar. The above operations are all performed based on the detection equipment, avoiding the problem of low detection efficiency caused by manual operation.

[0055] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a detection device, etc. that can realize the above functions. The following takes the detection device as an example to illustrate this embodiment and the following embodiments.

[0056] Based on this, the present application embodiment provides a detection method of millimeter wave radar, referring to Figure 1 Combined with Figure 2 , Figure 1 This is a flow chart of the first embodiment of the detection method of the millimeter wave radar of the present application.

[0057] In this embodiment, the detection method of the millimeter wave radar includes steps S10 to S50:

[0058] Step S10, collecting point cloud data of the millimeter wave radar to be detected and video data of the camera.

[0059] After establishing the correct connection relationship between the detection device and the millimeter-wave radar and camera to be detected, the detection device can collect and obtain the point cloud data of the millimeter-wave radar to be detected and the video data of the camera in real time, as described below:

[0060] In a feasible implementation manner, step S10 includes step S11:

[0061] Step S11, acquiring in real time the point cloud data of several millimeter-wave radars to be detected, and acquiring in real time the video data corresponding to the occurrence scene of the point cloud data captured by the camera, wherein the point cloud data and the video data are stored according to timestamps.

[0062] When the detection device establishes correct connections with the millimeter-wave radar and camera to be detected, and both the millimeter-wave radar and camera to be detected can work normally and are turned on synchronously, the detection device continuously collects point cloud data of different millimeter-wave radars to be detected. For details, please refer to Figure 2 As shown, the green point cloud data and the red point cloud data can be regarded as point cloud data detected by different millimeter-wave radars to be detected, and the video data of the corresponding scene is recorded synchronously. All the collected point cloud data and video data are stored according to the corresponding timestamps.

[0063] Among them, the timestamp is used for time alignment of point cloud data and video data.

[0064] Step S20: jointly calibrate the millimeter-wave radar to be detected and the camera to determine the conversion matrix between the millimeter-wave radar to be detected and the camera.

[0065] After pre-calibrating the camera's intrinsic parameters, the external parameters are calibrated using the self-developed millimeter-wave radar and camera joint calibration tool to obtain the rotation matrix and translation matrix (i.e., conversion matrix), thereby achieving spatial alignment between the camera and millimeter-wave radar point clouds and providing a reliable basis for matching the scene's real point cloud data, as described below:

[0066] In a feasible implementation manner, step S20 includes steps S21 to S22:

[0067] Step S21, calibrating the internal parameters of the camera to obtain the internal parameter matrix of the camera.

[0068] Step S22, calibrate the external parameters of the millimeter wave radar to be detected and the camera, and combine the internal parameter matrix to obtain the rotation matrix and translation matrix between the millimeter wave radar to be detected and the camera. The conversion matrix includes the internal parameter matrix, the rotation matrix and the translation matrix.

[0069] Assume that the coordinate of the millimeter-wave radar to be detected in the camera coordinate system is X offset , Y offset , Z offset It should be noted that OX r Y r Z r Coordinate system (radar coordinate system) in OX c Y c Z c The posture in the coordinate system (camera coordinate system) is (α, β, γ), where α, β, and γ are pitch angle, yaw angle, and roll angle, respectively. c The rotation matrix of the axis is R x , in Y c The rotation matrix of the axis is R y , in Zc The rotation matrix of the axis is R z , we can get the right day before yesterday OX r Y r Z r Coordinate system turns right to front OX c Y c Z c Coordinate system, the specific internal parameter matrix is ​​shown in Formula 1:

[0070]

[0071] The specific rotation matrix is ​​shown in Formula 2:

[0072]

[0073] The specific translation matrix is ​​shown in Formula 3:

[0074]

[0075] Among them, X r , Y r and Z r is the point cloud position coordinates of the point cloud data in the radar coordinate system, K is the intrinsic parameter matrix of the camera, u and v are the point cloud position coordinates of the point cloud data in the image coordinate system, X c , Y c and Z c It is the point cloud position coordinate of the point cloud data in the camera coordinate system.

[0076] Step S30, detecting and marking the target object in the video data to obtain true value data.

[0077] It should be noted that this embodiment combines deep learning to detect and mark target objects in video data, so that the generated true value data is closer to the actual value, thereby improving the detection accuracy of the millimeter wave radar to be detected.

[0078] In a feasible implementation, step S30 includes steps S31 to S35:

[0079] Step S31, performing frame extraction processing on the video data to obtain a plurality of target scene graphs.

[0080] Step S32: Detect each target scene graph using a preset deep learning target detection model, and mark the target objects in each target scene graph.

[0081] Step S33, using a preset algorithm to generate a bounding box containing each target object, and recording the position, category and confidence score of each target object to obtain detection data.

[0082] Step S34, associating the detection data with the timestamp to obtain true value data.

[0083] First, the video data is frame-processed, and representative scene images (i.e., target scene images) are extracted from the real-time video stream of the video data, ensuring that the selected target scene images cover different time periods and that the scene targets are in motion. Subsequently, the pre-trained YOLOv8 deep learning target detection model (i.e., the preset deep learning target detection model) is used to detect and process these target scene images to be analyzed. The model automatically identifies and marks the target objects, including predetermined target categories such as cars, trucks, and buses, by analyzing the features of each target scene image. During the detection process, the YOLOv8 algorithm in the YOLOv8 deep learning target detection model generates a bounding box containing the target object, which is used to record the location, category, and confidence score of each target object, and uses it as the detection result, which can ensure the accuracy and reliability of the detection result.

[0084] In addition, all detection results will be associated with timestamp information to form a structured data set. The detection results will be used as true value data to match and statistically analyze the point cloud data in subsequent steps. This process not only improves the efficiency of scene analysis, but also lays the foundation for the precise alignment and statistical analysis of subsequent data.

[0085] Step S40, projecting the point cloud data through the transformation matrix to obtain the corresponding relationship between the point cloud data and the true value data.

[0086] In a feasible implementation manner, step S40 includes steps S41 to S42:

[0087] Step S41, based on the scene graph timestamp corresponding to each target scene graph, search for point cloud data corresponding to the scene graph timestamp at the timestamp to obtain the target point cloud data.

[0088] Step S42, based on the intrinsic parameter matrix, the rotation matrix and the translation matrix, the target point cloud data is projected to obtain the point cloud coordinate position of the target point cloud data in the image coordinate system of the camera, and then the correspondence between the target point cloud data and the true value data is determined according to the point cloud coordinate position.

[0089] First, according to the scene graph timestamp corresponding to the target scene graph, find the timestamp corresponding to the point cloud data corresponding to or closest to the scene graph timestamp, ensure that the two timestamps are as consistent as possible in time, and determine the found point cloud data as the target point cloud data.

[0090] The determined target point cloud data will be converted to the camera's image coordinate system to achieve spatial alignment of the data. Therefore, it is necessary to project the target point cloud data based on the intrinsic matrix, rotation matrix, and translation matrix, and perform projection calculation on each point cloud data to calculate the point cloud coordinate position of each target point cloud data in the image coordinate system, so as to determine the correspondence between the target point cloud data and the true value data.

[0091] Step S50, using a maximum weight matching algorithm, matches the true value data and the point cloud data having a corresponding relationship, and determines the detection result of the millimeter wave radar to be detected based on the obtained matching result.

[0092] In this embodiment, a maximum weight matching algorithm is used to match the true value data calculated based on step S40 with the target point cloud data. The maximum weight matching algorithm can effectively reduce the mismatch between the true value data and the target point cloud data, improve the accuracy and precision of the matching, and thus improve the detection accuracy of the detection result of the millimeter wave radar to be detected.

[0093] The entire matching process is divided into the first maximum weight matching process and the second maximum weight matching process. The difference between the two maximum weight matching processes is that the rules for adding the true value bounding box of the true value data are different, as described below:

[0094] In a feasible implementation manner, step S50 includes steps S51 to S55:

[0095] Step S51, creating an undirected graph through a first maximum weight matching algorithm, and obtaining a corresponding relationship between the target point cloud data and a first true value bounding box of the true value data through the undirected graph;

[0096] Step S52, traversing each target point cloud data, and determining whether each target point cloud data is within a first true value boundary box corresponding to the target point cloud data;

[0097] Step S53: If the target point cloud data is within the first true value bounding box corresponding to the target point cloud data, the target point cloud data within the first true value bounding box corresponding to the target point cloud data is determined as the first point cloud data, and a first distance between the first point cloud data and the center of the bounding box of the first true value bounding box corresponding to the first point cloud data is calculated, and a matching relationship between the first point cloud data and the true value data having a corresponding relationship is determined based on the first distance.

[0098] In the first maximum weight matching process, an undirected graph is first created to represent the relationship between the first truth bounding box of the truth data and the target point cloud data. Each first truth bounding box is traversed and a node is created for each truth bounding box. The node is named gt_x, where x is the index of the truth data.

[0099] All target point cloud data are traversed, and for each target point cloud data, its pixel point cloud data coordinates in the pixel coordinate system are checked to see whether they are within the first true value bounding box of a certain true value data.

[0100] If a certain target point cloud data is within a first true value bounding box of a certain true value data, the target point cloud data within the first true value bounding box is determined as the first point cloud data, a first distance between a point of the first point cloud data and the center of the bounding box of the first true value bounding box is calculated, and a first score is calculated according to the first distance. The calculation formula of the score is shown in Formula 4.

[0101]

[0102] It should be noted that score1 is the first score calculated based on the first distance, and distance1 is the first distance.

[0103] If a certain target point cloud data is not within the first true value bounding box corresponding to any true value data, the first score of the target point cloud data is directly determined to be none.

[0104] After all target point cloud data are calculated, for target point cloud data whose first score is not none, an edge is added to the undirected graph from the point of the first point cloud data to the center of the bounding box of the first true value bounding box, and the weight of the edge is the first score corresponding to the first point cloud data.

[0105] Step S54: if the target point cloud data is not within the first true value bounding box corresponding to the target point cloud data, it is determined whether the target point cloud data is within a second true value bounding box established based on the first true value bounding box corresponding to the target point cloud data.

[0106] Step S55: If the target point cloud data is within the second true value bounding box corresponding to the target point cloud data, the target point cloud data within the second true value bounding box corresponding to the target point cloud data is determined as the second point cloud data, and a second distance between the second point cloud data and the center of the bounding box of the second true value bounding box corresponding to the second point cloud data is calculated, and a matching relationship between the second point cloud data and the true value data having a corresponding relationship is determined based on the second distance.

[0107] Obtain the target point cloud data whose first score is none in step S52, that is, the target point cloud data is not in the first true value bounding box corresponding to the target point cloud data, and perform a second maximum weight matching calculation on the target point cloud data, specifically: use three quarters of the diagonal length of the first true value bounding box of the true value data as a radius, and use the first true value bounding box as the center of the circle to establish a second true value bounding box for each true value data, and determine whether the target point cloud data whose first score is none exists in the second true value bounding box corresponding to any true value data.

[0108] If the target point cloud data exists in any second true value bounding box, the target point cloud data in the second true value bounding box is determined as the second point cloud data, a second distance between the point of the second point cloud data and the center of the bounding box of the second true value bounding box is calculated, and a second score is calculated according to the second distance. The calculation formula of the score is shown in Formula 5.

[0109]

[0110] It should be noted that score2 is the second score calculated based on the second distance, and distance2 is the second distance.

[0111] If a certain target point cloud data is not within the second true value bounding box corresponding to any true value data, the second score of the target point cloud data is directly determined to be none.

[0112] After calculating all the target point cloud data, for the second point cloud data whose second score is not none, an edge is added to the undirected graph from the point of the second point cloud data to the center of the bounding box of the second true value bounding box, and the weight of the edge is the second score corresponding to the second point cloud data.

[0113] From the above, we can know that through the first maximum weight matching process and the second maximum weight matching process, the matching relationship between the true value data and the target point cloud data can be determined, ensuring high-quality pairing results in subsequent analysis and statistics. This process lays the foundation for the subsequent matching result statistics and analysis.

[0114] Furthermore, before step S50, step S60 is also included:

[0115] Step S60, determine a test scene, and divide the test scene into lanes and distance intervals to obtain a plurality of lane partitions and distance partitions.

[0116] When performing statistics and analysis on the matching results, we first use the semi-automatic statistical analysis tool to divide the test scene in detail. Specifically, we divide the test scene into multiple lane partitions according to the test scene captured by the camera, and set two horizontal dividing lines to divide the test scene into three distance partitions: short distance, medium distance and long distance. Figure 3 As shown in the figure, the green quadrilateral represents the lane partition range, and the yellow straight line represents the distance partition dividing line. By setting the lane partition and distance partition, the true value bounding box at each timestamp can be accurately classified.

[0117] Furthermore, step S50 also includes steps S56 to S57:

[0118] Step S56, determining the bounding box center of the first true value bounding box, the bounding box center of the second true value bounding box, the lane partition and the distance partition where each first point cloud data and each second point cloud data are located at each timestamp, and determining the total number of true value bounding boxes and the total number of point cloud data for each lane partition and each distance partition at each timestamp.

[0119] Step S57, divide the total number of point cloud data by the total number of true value bounding boxes to obtain the detection rate of each lane partition and each distance partition at each timestamp, and determine the detection result based on the detection rate.

[0120] When statistically analyzing data at a certain timestamp, firstly, all first truth bounding boxes and second truth bounding boxes at the timestamp are traversed, and classification analysis is performed based on their bounding box centers.

[0121] The classification process is divided into two steps: the first classification is based on lane partitions. The detection device determines which lane partition the bounding box belongs to based on the position of the bounding box center of the first true bounding box and the bounding box center of the second true bounding box. Each lane partition has a corresponding quadrilateral box, which is defined by four vertex coordinates in the image coordinate system. The ray method is used to check whether the center of each true bounding box falls within these lane partition boxes. The detection device divides the first point cloud data that has a matching relationship with the true bounding box and the second point cloud data into the lane partition where the true bounding box is located. This classification process helps distinguish the lane position of the target object in the scene image and provides support for further statistics.

[0122] The second classification is based on distance partitioning. The detection device determines whether the bounding box belongs to the short-range, medium-range or long-range area according to the position of the bounding box center of the first truth bounding box and the bounding box center of the second truth bounding box relative to the two set horizontal dividing lines. Each distance interval is defined according to the actual road scene. For example, the short distance may correspond to the closer area in front of the camera's field of view, while the long distance covers the distant area. This step can provide statistical information about the distance distribution of the target object, which helps to evaluate the detection accuracy of millimeter-wave radar and camera at different distances.

[0123] After the classification is completed, the detection rate statistics of the true value bounding box of each lane partition and each distance partition are performed. Taking the detection rate statistics of the first lane partition as an example: the total number N of the true value bounding boxes of lane partition 1 is obtained according to the classification result, and the first point cloud data and the second point cloud data matched by each true value bounding box on lane partition 1 are obtained according to the matching result of step S40. Let the total number of matched point cloud data (i.e. the sum of the first point cloud data and the second point cloud data) be C, then the detection rate Similarly, statistics are collected for each lane partition and each distance partition at each time stamp. The statistical results are saved on the detection device for subsequent analysis of the test results of the millimeter-wave radar to be detected. Figure 4 shown.

[0124] Figure 4 The frames in represent timestamps. It can be seen from the figure that when the timestamp is 4166, there are 6 target point cloud data in lane 1 (i.e. lane partition 1), 5 target point cloud data in lane 2 (i.e. lane partition 2), 2 target point cloud data in lane 3 (i.e. lane partition 3), 2 target point cloud data in lane 4 (i.e. lane partition 4), 1 target point cloud data in lane 5 (i.e. lane partition 5), 5 target point cloud data in lane 6 (i.e. lane partition 6), 15 target point cloud data in close range, 4 target point cloud data in medium range, and 2 target point cloud data in long range. The detection rate on consecutive frames is calculated according to the number of true values. The detection rate of lane 1 is 0.92, the detection rate of lane 2 is 1, the detection rate of lane 3 is 1, the detection rate of lane 4 is 1, the detection rate of lane 5 is 0.97, the detection rate of lane 6 is 1, the detection rate of close distance is 0.99, the detection rate of medium distance is 0.97, and the detection rate of long distance is 0.97.

[0125] In this embodiment, during the data matching process, the point cloud data is projected into the image coordinate system by combining the joint calibration results of the camera and the millimeter-wave radar, that is, the transformation matrix, to obtain the correspondence between the point cloud data and the true value data, thereby achieving spatial alignment. Then, the maximum weight matching algorithm is used to match the true value data and the point cloud data with the corresponding relationship, thereby achieving high-precision matching of the point cloud data and the true value data, thereby improving the reliability of the detection results of the millimeter-wave radar. The above operations are all performed based on the detection equipment, thus avoiding the problem of low detection efficiency caused by manual operation.

[0126] The present application provides a detection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the detection method of the millimeter wave radar in the above-mentioned embodiment one.

[0127] Reference below Figure 5 , which shows a schematic diagram of the structure of a detection device suitable for implementing the embodiment of the present application. The detection device in the embodiment of the present application may include but is not limited to mobile terminals such as laptop computers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The detection device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0128] like Figure 5As shown, the detection device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the detection device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the detection device to communicate with other devices wirelessly or wired to exchange data. Although the detection device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0129] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0130] The detection device provided by the present application adopts the detection method of the millimeter wave radar in the above embodiment, which can solve the technical problem of how to effectively improve the test reliability and test efficiency of the millimeter wave radar. Compared with the prior art, the beneficial effects of the detection device provided by the present application are the same as the beneficial effects of the detection method of the millimeter wave radar provided in the above embodiment, and the other technical features in the detection device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0131] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0132] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0133] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the detection method of the millimeter-wave radar in the above-mentioned embodiment.

[0134] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0135] The computer-readable storage medium may be included in the detection device; or may exist independently without being assembled into the detection device.

[0136] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the detection device, the detection device: collects point cloud data of the millimeter-wave radar to be detected and video data of the camera; jointly calibrates the millimeter-wave radar to be detected and the camera to be detected, and determines the transformation matrix between the millimeter-wave radar to be detected and the camera; detects and marks the target objects in the video data to obtain true value data; projects the point cloud data through the transformation matrix to obtain the corresponding relationship between the point cloud data and the true value data; uses the maximum weight matching algorithm to match the true value data and the point cloud data with the corresponding relationship, and determines the detection result of the millimeter-wave radar to be detected based on the matching result.

[0137] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0138] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0139] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0140] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned millimeter-wave radar detection method, and can solve the technical problem of how to effectively improve the test reliability and test efficiency of the millimeter-wave radar. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the millimeter-wave radar detection method provided in the above-mentioned embodiment, and will not be repeated here.

[0141] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A millimeter wave radar detection method, characterized in that: The detection method of the millimeter wave radar includes: Collect the point cloud data of the millimeter-wave radar to be detected and the video data of the camera; Jointly calibrating the millimeter-wave radar to be detected and the camera to determine a conversion matrix between the millimeter-wave radar to be detected and the camera; Detecting and marking target objects in the video data to obtain true value data; Projecting the point cloud data through the transformation matrix to obtain a corresponding relationship between the point cloud data and the true value data; The maximum weight matching algorithm is used to match the true value data and the point cloud data having a corresponding relationship, and the detection result of the millimeter wave radar to be detected is determined according to the obtained matching result.

2. The detection method of millimeter wave radar according to claim 1, characterized in that: The step of collecting the point cloud data of the millimeter wave radar to be detected and the video data of the camera includes: Acquire in real time the point cloud data of a plurality of the millimeter-wave radars to be detected, and acquire in real time the video data corresponding to the occurrence scene of the point cloud data captured by the camera; Wherein, the point cloud data and the video data are stored according to timestamps.

3. The detection method of millimeter wave radar as claimed in claim 2, characterized in that: The step of jointly calibrating the millimeter-wave radar to be detected and the camera to determine the conversion matrix between the millimeter-wave radar to be detected and the camera comprises: Calibrate the internal parameters of the camera to obtain the internal parameter matrix of the camera; Calibrate the external parameters of the millimeter-wave radar to be detected and the camera, and obtain the rotation matrix and translation matrix between the millimeter-wave radar to be detected and the camera in combination with the internal parameter matrix; Wherein, the transformation matrix includes the intrinsic parameter matrix, the rotation matrix and the translation matrix.

4. The detection method of millimeter wave radar as claimed in claim 3, characterized in that: The step of detecting and marking the target object in the video data to obtain true value data comprises: Performing frame extraction processing on the video data to obtain a plurality of target scene graphs; Detect each of the target scene graphs using a preset deep learning target detection model, and mark the target objects in each of the target scene graphs; Generate a bounding box containing each of the target objects using a preset algorithm, and record the position, category and confidence score of each of the target objects to obtain detection data; The detection data is associated with the timestamp to obtain the true value data.

5. The detection method of millimeter wave radar as claimed in claim 4, characterized in that: The step of projecting the point cloud data by means of the transformation matrix to obtain a corresponding relationship between the point cloud data and the true value data comprises: According to the scene graph timestamp corresponding to each of the target scene graphs, searching for the point cloud data corresponding to the scene graph timestamp at the timestamp to obtain the target point cloud data; Based on the intrinsic parameter matrix, the rotation matrix and the translation matrix, the target point cloud data is projected to obtain the point cloud coordinate position of the target point cloud data in the image coordinate system of the camera, and then the correspondence between the target point cloud data and the true value data is determined according to the point cloud coordinate position.

6. The detection method of millimeter wave radar as claimed in claim 5, characterized in that: The maximum weight matching algorithm includes a first maximum weight matching algorithm and a second maximum weight matching algorithm. The step of matching the true value data and the point cloud data having a corresponding relationship by using the maximum weight matching algorithm includes: Creating an undirected graph by using the first maximum weight matching algorithm, and obtaining a corresponding relationship between the target point cloud data and a first true value bounding box of the true value data by using the undirected graph; Traversing each of the target point cloud data, and determining whether each of the target point cloud data is within a first true value boundary box corresponding to the target point cloud data; If the target point cloud data is within a first true value bounding box corresponding to the target point cloud data, the target point cloud data within the first true value bounding box corresponding to the target point cloud data is determined as the first point cloud data, a first distance between the first point cloud data and a center of a bounding box of the first true value bounding box corresponding to the first point cloud data is calculated, and a matching relationship between the first point cloud data and the true value data having a corresponding relationship is determined based on the first distance; If the target point cloud data is not within the first true value bounding box corresponding to the target point cloud data, determining whether the target point cloud data is within a second true value bounding box established based on the first true value bounding box corresponding to the target point cloud data; If the target point cloud data is within a second true value bounding box corresponding to the target point cloud data, the target point cloud data within the second true value bounding box corresponding to the target point cloud data is determined as second point cloud data, and a second distance between the second point cloud data and the center of the bounding box of the second true value bounding box corresponding to the second point cloud data is calculated, and a matching relationship between the second point cloud data and the true value data having a corresponding relationship is determined based on the second distance.

7. The detection method of millimeter wave radar as claimed in claim 6, characterized in that: Before the step of determining the detection result of the to-be-detected millimeter-wave radar according to the obtained matching result, the step further includes: A test scenario is determined, and lane division and distance interval division are performed on the test scenario to obtain a plurality of lane partitions and distance partitions.

8. The detection method of millimeter wave radar as claimed in claim 7, characterized in that: The step of determining the detection result of the to-be-detected millimeter-wave radar according to the obtained matching result comprises: Determine, at each of the timestamps, a bounding box center of the first true value bounding box, a bounding box center of the second true value bounding box, a lane partition and a distance partition in which each of the first point cloud data and each of the second point cloud data are located, and determine the total number of true value bounding boxes and the total number of point cloud data in each of the lane partitions and each of the distance partitions at each timestamp; The total number of the point cloud data is divided by the total number of the true value bounding boxes to obtain the detection rate of each lane partition and each distance partition at each timestamp, and the detection result is determined based on the detection rate.

9. A detection device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the detection method of the millimeter-wave radar according to any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the detection method of the millimeter-wave radar according to any one of claims 1 to 8 are implemented.

Images