An automatic joint calibration method for roadside sensors based on grid maps

Abstract

The present invention relates to the technical field of roadside sensors, and specifically discloses an automatic joint calibration method for roadside sensors based on a grid map. The roadside sensor includes a millimeter wave radar and a camera. The millimeter wave radar is used to receive radar points of a target, and the camera is used to receive an image. By establishing a millimeter wave radar grid map A and an image grid map B, and by associating the millimeter wave radar grid map A with the image grid map B, the joint calibration of the roadside sensors is achieved. The present invention can automatically complete the calibration without manual work, and has high efficiency; the method of the present invention does not require characteristic markers, and can automatically perform road test sensor calibration by detecting vehicle targets traveling normally on the road, and has low cost.

Description

Technical Field

[0001] The present invention relates to the technical field of roadside sensors, and in particular to an automatic joint calibration method for roadside sensors based on a grid map. Background Art

[0002] In today's rapidly developing urbanization and transportation fields, roadside sensors, as one of the core components of intelligent traffic management, are installed at fixed locations beside the road to detect and record data from passing vehicles, pedestrians, or other traffic participants. These sensors can collect data such as vehicle speed, vehicle density, lane occupancy, pedestrian traffic conditions, etc., providing valuable real-time information for traffic management departments. However, in order to ensure the accuracy and reliability of these data, roadside sensors must be accurately calibrated. Accurate joint calibration of roadside sensors can enhance the data correlation between different roadside sensors and provide more accurate information for traffic management departments. In view of this, an automatic joint calibration method for roadside sensors based on raster maps is proposed. Summary of the invention

[0003] The purpose of the present invention is to provide a method for automatic joint calibration of roadside sensors based on a grid map, aiming to solve the problems of high cost, long time consumption and low efficiency of existing roadside sensor calibration methods.

[0004] To achieve the above object, the present invention provides the following technical solution: a method for automatic joint calibration of roadside sensors based on a grid map, characterized in that the roadside sensor includes a millimeter wave radar and a camera, the millimeter wave radar is used to receive radar points of the target, and the camera is used to receive an image, by establishing a millimeter wave radar grid map A and an image grid map B, and by associating the millimeter wave radar grid map A with the image grid map B, the joint calibration of the roadside sensors is achieved, specifically comprising the following steps:

[0005] S1. Construct a millimeter-wave radar grid map A, where the width of each grid is set to Sr meters, and the number of grids in the horizontal and vertical directions of the grid map is consistent with the visual range of the millimeter-wave radar used;

[0006] S2. Construct an image grid map B. Divide the image received by the camera into grids of Si pixels. The grid size Si should be divided according to the image field size. The size of a single grid Si should be smaller than the size of the vehicle at 80 meters from the camera in the image. The number of horizontal grids Wr of the image grid map B is the image width / Si, and the number of vertical grids Hr is the image height / Si.

[0007] S3, associating radar grid map A with image grid map B;

[0008] S4, collecting millimeter wave radar and camera image data at different times, and updating the image grid id and probability corresponding to each grid in the radar grid map A;

[0009] S5. Design a probability threshold T, count the radar grid IDs and their corresponding image grid IDs whose probability is greater than the threshold T, and convert them into radar point and pixel point coordinates;

[0010] S6. Calculate the calibration matrix H based on the matching point pairs.

[0011] Preferably, the specific operations in S1 are as follows:

[0012] S11, the grids in the grid map A are numbered from left to right and from bottom to top, and each grid is assigned an ID, that is, the grid ID in the lower left corner is 0, and the grid IDs are incremented by 1 as they go to the right;

[0013] S12, obtaining the target radar point data received by the millimeter wave radar, and calculating the grid id of each radar point p(x, y) in the millimeter wave radar grid map A. The algorithm is: ; Where y / Sr and x / Sr are the rows and columns corresponding to the radar points in the grid map respectively;

[0014] S13. The grid IDs corresponding to all radar points are combined into a radar grid ID set of the current frame, recorded as ids1.

[0015] Preferably, the specific operations in S2 are as follows:

[0016] S21, refer to the radar grid map numbering method, number the image grid map, and assign each grid id;

[0017] S22, selecting an existing vehicle detection method, performing vehicle detection on the camera image, and obtaining pixel points p(u,v) of the vehicle in the image;

[0018] S23, calculating the grid ID corresponding to the detected vehicle pixel point in the image grid map B;

[0019] S24. The grid IDs corresponding to all vehicle pixels are combined into a current frame image grid ID set, which is recorded as ids2.

[0020] Preferably, the specific operations in S3 are as follows:

[0021] S31, expand each grid in the radar grid map A, each grid contains an integer subsequence, the number of the subsequences is equal to the total number of grids of the image grid B Wr×Hr, and the number of the subsequence is consistent with the image grid id number, and the initial value of each value in the subsequence is 0;

[0022] S32, traverse the statistical radar grid id set ids1, find the subsequence corresponding to each radar grid in ids1 in the radar grid map A; find the values ​​in these subsequences that have the same number as the id number in the statistical image grid id set ids2, and add 1 to these corresponding values;

[0023] S33, calculating the corresponding image grid id and probability of each grid in the radar grid map A; the corresponding image grid id is the sequence number corresponding to the maximum value in the radar grid subsequence, and the probability is the maximum value of the subsequence / the sum of the grid subsequence.

[0024] Preferably, the specific operations in S5 are as follows:

[0025] S51, traverse each grid in the radar grid map A, and if the probability of the grid is greater than T, calculate the radar point coordinates pr(xr, yr) corresponding to the grid id1, and calculate the pixel point coordinates pi(u, v) of the image grid id2 corresponding to the grid;

[0026] For the radar grid id1 that meets the conditions, calculate the corresponding radar point coordinates pr(xr,yr), the formula is:

[0027] ;

[0028] ;

[0029] Among them, Sr is the side length of the grid, Wr is the number of horizontal grids in the radar grid map, and % represents the co-operation;

[0030] Calculate pixel coordinates: Find the image grid id2 corresponding to the radar grid; calculate the corresponding image pixel coordinates pi (u, v), the formula is:

[0031] ;

[0032] ;

[0033] Where Si is the size of the image grid, Wr is the number of horizontal grids in the image grid map, and % represents the residual operation;

[0034] S52, the calculated radar point coordinates pr and pixel point coordinates pi are taken as a pair of matching point pairs, and the statistical point pairs are combined into a series of matching point pairs.

[0035] Preferably, the specific operations in S6 are as follows:

[0036] S61. The function of the calibration matrix H is to project the radar points to the image pixel coordinates. The projection formula is as follows: ; pi is the pixel in the image coordinate system, pr is the point in the radar coordinate system;

[0037] S62, construct an error function based on a series of statistical matching points, solve H using the least squares method, and use matching point pairs to construct an error function to measure the difference between the transformed point and the actual image point. The formula is: ; Where e is the error function; and are the image point and radar point in the mth matching point pair respectively;

[0038] S63. Select a general least squares method and iteratively solve the calibration matrix H according to multiple groups of matching points.

[0039] The present invention proposes a method for automatic joint calibration of roadside sensors based on a grid map, which has the following beneficial effects:

[0040] 1. The present invention establishes two grid maps (millimeter wave radar grid map A and image grid map B) and utilizes the association between the two to achieve automatic joint calibration of roadside sensors. The entire process does not require manual setting of calibration points or reliance on specific markers, which significantly improves the degree of automation of the calibration process and reduces manual operation costs. Since the method does not require manual intervention and automatically associates radar and camera data through an algorithm, the calibration time is significantly shortened. Compared with traditional methods, the present invention has significantly improved efficiency and can complete the calibration of roadside sensors more quickly, thereby being put into practical application more quickly.

[0041] 2. The present invention does not require additional hardware auxiliary equipment (such as self-positioning robots, combined navigation, etc.), nor does it require specific markers, thus greatly reducing the calibration cost; this has important economic significance for the large-scale promotion and application of roadside sensors. Through automatic joint calibration, the data correlation between different roadside sensors can be strengthened, thereby providing more accurate and reliable traffic data; this is of great significance for traffic management departments to make scientific decisions and optimize traffic management; through the association of raster maps and probability statistics, efficient calibration between sensors is achieved.

[0042] In summary, the prior art usually requires manual facilities to calibrate calibration points, while the present invention can automatically complete the calibration without manual work, and is highly efficient; the prior art usually requires specific markers for calibration, while the method of the present invention does not require specific markers, and can automatically calibrate road test sensors by detecting vehicle targets traveling normally on the road, and is low-cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a process step diagram of the present invention;

[0044] Figure 2A schematic diagram of a millimeter wave radar grid map A of the present invention;

[0045] Figure 3 This is a schematic diagram of an example of a grid ID of the present invention;

[0046] Figure 4 This is a schematic diagram of an example of an image grid map B of the present invention;

[0047] Figure 5 Schematic diagram of radar grid subsequence of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] For examples, see Figure 1-Figure 5 The present invention provides a technical solution: a method for automatic joint calibration of roadside sensors based on a grid map, wherein the roadside sensor comprises a millimeter wave radar and a camera, wherein the millimeter wave radar is used to receive radar points of a target, and the camera is used to receive an image, and the joint calibration of the roadside sensors is achieved by establishing a millimeter wave radar grid map A and an image grid map B, and by associating the millimeter wave radar grid map A with the image grid map B, and specifically comprising the following steps:

[0050] S1. Construct a millimeter wave radar grid map A (such as Figure 2 As shown in the figure), the width of each grid is set to Sr meters, and the number of grids in the horizontal and vertical directions of the grid map is consistent with the visual range of the millimeter-wave radar used; if the vertical visual range of the millimeter-wave radar is 100 meters and the horizontal range is 50 meters, and the grid width Sr is set to 1 meter, then the horizontal number of grids Wr is 50, and the vertical number Hr is 100;

[0051] S11, such as Figure 3 As shown, the grids in grid map A are numbered from left to right and from bottom to top, and each grid is assigned an id, that is, the id of the lower left grid is 0, and the ids of the grids to the right are incremented by 1;

[0052] S12, obtain the target radar point data received by the millimeter wave radar, for each radar point p(x, y), where x and y are the positions of the point in the radar coordinate system, calculate the grid id in the millimeter wave radar grid map A, the algorithm is: ; Where y / Sr and x / Sr are the rows and columns corresponding to the radar points in the grid map respectively;

[0053] S13, the grid IDs corresponding to all radar points are combined into a radar grid ID set of the current frame, recorded as ids1;

[0054] The millimeter-wave radar grid map A is constructed and numbered, and the radar point data is mapped to the corresponding position in the grid map; this method converts the radar point data into discrete grid IDs, which is helpful for subsequent roadside sensor data fusion and joint calibration; in this way, continuous spatial data can be effectively converted into discrete grid data for easy analysis and processing.

[0055] S2, construct image grid map B (such as Figure 4 As shown in the figure, the image received by the camera is divided into grids with a grid size of Si pixels. The grid size Si should be divided according to the size of the image field of view. The size of a single grid Si should be smaller than the size of the vehicle at 80 meters from the camera in the image. Then the number of horizontal grids Wr of the image grid map B is the image width / Si, and the number of vertical grids Hr is the image height / Si.

[0056] S21. Referring to the radar grid map numbering method, the image grid map is numbered and each grid is assigned an ID; the grids are numbered from left to right and from bottom to top. For example, the grid in the lower left corner is numbered 0, the first row is numbered from 0 to Wr-1 from left to right, the second row is numbered from Wr to 2Wr-1 from left to right, and so on;

[0057] S22. Select an existing vehicle detection method (such as YOLO, Faster R-CNN, etc.), perform vehicle detection on the camera image, and obtain the pixel point p(u,v) of the vehicle in the image;

[0058] S23, calculating the grid ID corresponding to the detected vehicle pixel point in the image grid map B;

[0059] S24, forming a grid ID set of the current frame image corresponding to all the vehicle pixel points, recorded as ids2;

[0060] The image grid map B is constructed and numbered, and the vehicle detection results in the image are mapped to the corresponding positions in the grid map; the purpose of this is to jointly analyze and calibrate the detection results in the image with the radar data; by discretizing the image data into raster data, it can be more easily matched and fused with the data of other sensors, thereby improving the overall perception capability of the system.

[0061] S3, associating radar grid map A with image grid map B;

[0062] S31, expand each grid in the radar grid map A, each grid contains an integer subsequence, the number of the subsequences is equal to the total number of grids of the image grid B Wr×Hr, and the number of the subsequence is consistent with the image grid id number, and the initial value of each value in the subsequence is 0;

[0063] S32, traverse the statistical radar grid id set ids1, find the subsequence corresponding to each radar grid in ids1 in the radar grid map A; find the values ​​in these subsequences that have the same number as the id number in the statistical image grid id set ids2, and add 1 to these corresponding values;

[0064] S33, such as Figure 5 As shown, the corresponding image grid id and probability of each grid in the radar grid map A are calculated; the corresponding image grid id is the sequence number corresponding to the maximum value in the radar grid subsequence, and the probability is the maximum value of the subsequence / the sum of the grid subsequence;

[0065] Through the above steps, the association between radar grid map A and image grid map B is established; each radar grid is assigned a corresponding image grid ID and an association probability; this association can help match and fuse data from different sensors, thereby achieving more accurate sensor joint calibration.

[0066] S4. Collect millimeter-wave radar and camera image data at different times, and update the image grid id and probability corresponding to each grid in the radar grid map A.

[0067] S5. Design a probability threshold T, count the radar grid IDs and their corresponding image grid IDs whose probability is greater than the threshold T, and convert them into radar point and pixel point coordinates;

[0068] S51, traverse each grid in the radar grid map A, and if the probability of the grid is greater than T, calculate the radar point coordinates pr(xr, yr) corresponding to the grid id1, and calculate the pixel point coordinates pi(u, v) of the image grid id2 corresponding to the grid;

[0069] For the radar grid id1 that meets the conditions, calculate the corresponding radar point coordinates pr(xr,yr), the formula is:

[0070] ;

[0071] ;

[0072] Where Sr is the side length of the grid, Wr is the number of horizontal grids in the radar grid map, and % represents the remainder operation, which is used to determine in which grid a point is located when calculating grid coordinates;

[0073] Calculate pixel coordinates: Find the image grid id2 corresponding to the radar grid; calculate the corresponding image pixel coordinates pi (u, v), the formula is:

[0074] ;

[0075] ;

[0076] Where Si is the size of the image grid, Wr is the number of horizontal grids in the image grid map, and % represents the remainder operation, which is used to determine in which grid a point is located when calculating grid coordinates;

[0077] S52, the calculated radar point coordinates pr and pixel point coordinates pi are taken as a pair of matching point pairs, and the statistical point pairs are formed into a series of matching point pairs, which reflect the spatial correspondence between the radar and image coordinate systems, and all matching point pairs that meet the conditions are collected to form a matching point pair set, which can be used for subsequent sensor calibration, data fusion and other tasks;

[0078] Through the above steps, matching point pairs with high correlation can be extracted from the radar and image data. These point pairs can be used for further sensor calibration or data fusion to help improve the accuracy and consistency of the multi-sensor system. This method utilizes the complementary characteristics of radar and camera data, allowing the system to maintain good performance in different environments and conditions.

[0079] S6. Calculate the calibration matrix H based on the matching point pairs;

[0080] S61. The function of the calibration matrix H is to project the radar points to the image pixel coordinates. The projection formula is as follows: ; pi is the pixel in the image coordinate system, pr is the point in the radar coordinate system;

[0081] S62, construct an error function based on a series of statistical matching points, solve H using the least squares method, and use matching point pairs to construct an error function to measure the difference between the transformed point and the actual image point. The formula is: ; Where e is the error function; and are the image point and radar point in the mth matching point pair respectively;

[0082] S63, selecting a general least square method, and iteratively solving a calibration matrix H according to multiple sets of matching points;

[0083] A calibration matrix H is calculated, which accurately projects points in the radar coordinate system into the image coordinate system; this is very important for the integration of multi-sensor systems because it allows the data from different sensors to be fused into a unified coordinate system, thereby improving the overall perception and accuracy of the system; in practical applications, an accurate calibration matrix can significantly improve the performance of systems such as autonomous vehicles and robot navigation.

[0084] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for automatic joint calibration of roadside sensors based on a grid map, characterized in that: The roadside sensor includes a millimeter-wave radar and a camera. The millimeter-wave radar is used to receive radar points of the target, and the camera is used to receive images. By establishing a millimeter-wave radar grid map A and an image grid map B, and by associating the millimeter-wave radar grid map A and the image grid map B, the joint calibration of the roadside sensor is realized. Specifically, the following steps are included: S1. Construct a millimeter-wave radar grid map A, where the width of each grid is set to Sr meters, and the number of grids in the horizontal and vertical directions of the grid map is consistent with the visual range of the millimeter-wave radar used; S2. Construct an image grid map B. Divide the image received by the camera into grids of Si pixels. The grid size Si should be divided according to the image field size. The size of a single grid Si should be smaller than the size of the vehicle at 80 meters from the camera in the image. The number of horizontal grids Wr of the image grid map B is the image width / Si, and the number of vertical grids Hr is the image height / Si. S3, associating radar grid map A with image grid map B; S4, collecting millimeter wave radar and camera image data at different times, and updating the image grid id and probability corresponding to each grid in the radar grid map A; S5. Design a probability threshold T, count the radar grid IDs and their corresponding image grid IDs whose probability is greater than the threshold T, and convert them into radar point and pixel point coordinates; S6. Calculate the calibration matrix H based on the matching point pairs; The specific operations in S1 are as follows: S11, the grids in the grid map A are numbered from left to right and from bottom to top, and each grid is assigned an ID, that is, the grid ID in the lower left corner is 0, and the grid IDs are incremented by 1 as they go to the right; S12, obtaining the target radar point data received by the millimeter wave radar, and calculating the grid id of each radar point p(x, y) in the millimeter wave radar grid map A. The algorithm is: ; Where y / Sr and x / Sr are the rows and columns corresponding to the radar points in the grid map respectively; S13, the grid IDs corresponding to all radar points are combined into a radar grid ID set of the current frame, recorded as ids1; The specific operations in S2 are as follows: S21, refer to the radar grid map numbering method, number the image grid map, and assign each grid id; S22, selecting an existing vehicle detection method, performing vehicle detection on the camera image, and obtaining pixel points p(u,v) of the vehicle in the image; S23, calculating the grid ID corresponding to the detected vehicle pixel point in the image grid map B; S24, forming a grid ID set of the current frame image corresponding to all the vehicle pixel points, recorded as ids2; The specific operations in S3 are as follows: S31, expand each grid in the radar grid map A, each grid contains an integer subsequence, the number of the subsequences is equal to the total number of grids of the image grid B Wr×Hr, and the number of the subsequence is consistent with the image grid id number, and the initial value of each value in the subsequence is 0; S32, traverse the statistical radar grid id set ids1, find the subsequence corresponding to each radar grid in ids1 in the radar grid map A; find the values ​​in these subsequences that have the same number as the id number in the statistical image grid id set ids2, and add 1 to these corresponding values; S33, calculating the corresponding image grid id and probability of each grid in the radar grid map A; the corresponding image grid id is the sequence number corresponding to the maximum value in the radar grid subsequence, and the probability is the maximum value of the subsequence / the sum of the grid subsequence; The specific operations in S5 are as follows: S51, traverse each grid in the radar grid map A, and if the probability of the grid is greater than T, calculate the radar point coordinates pr(xr, yr) corresponding to the grid id1, and calculate the pixel point coordinates pi(u, v) of the image grid id2 corresponding to the grid; For the radar grid id1 that meets the conditions, calculate the corresponding radar point coordinates pr(xr,yr), the formula is: ； ； Among them, Sr is the side length of the grid, Wr is the number of horizontal grids in the radar grid map, and % represents the co-operation; Calculate pixel coordinates: Find the image grid id2 corresponding to the radar grid; calculate the corresponding image pixel coordinates pi (u, v), the formula is: ； ； Among them, Si is the size of the image grid, Wi is the number of horizontal grids in the image grid map, and % represents the residual operation; S52, the calculated radar point coordinates pr and pixel point coordinates pi are used as a pair of matching point pairs, and the statistical point pairs are combined into a series of matching point pairs; The specific operations in S6 are as follows: S61. The function of the calibration matrix H is to project the radar points to the image pixel coordinates. The projection formula is as follows: ; pi is the pixel in the image coordinate system, pr is the point in the radar coordinate system; S62, construct an error function based on a series of statistical matching points, solve H using the least squares method, and use matching point pairs to construct an error function to measure the difference between the transformed point and the actual image point. The formula is: ; Where e is the error function; and are the image point and radar point in the mth matching point pair respectively; S63. Select a general least squares method and iteratively solve the calibration matrix H according to multiple groups of matching points.

Images