Method for implementing automatic road end camera calibration based on key point capture
By using a key point capture-based method and RTK UAVs and network models for automatic camera calibration, the problems of accuracy and safety in roadside camera calibration are solved, and efficient and accurate camera calibration and detection are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI GEOMETRICAL PERCEPTION & LEARNING CO LTD
- Filing Date
- 2023-08-09
- Publication Date
- 2026-05-15
AI Technical Summary
Existing camera calibration technologies suffer from low accuracy, complex operation, high cost, and unsuitability for harsh weather conditions in roadside applications. In particular, the calibration parameters of intersection cameras are inaccurate due to shaking, and traditional methods require the use of calibration objects, which affects application and safety.
A keypoint-based capture method is adopted, which uses an RTK UAV to measure camera points and ground keypoints, obtains the coordinates of ground keypoints, manually calibrates the homography matrix, uses a network model for target detection and self-calibration, and combines grayscale processing and contour detection to automatically adjust the homography matrix to achieve automatic camera calibration.
It enables safe, efficient, and accurate calibration of cameras at the roadside, reduces inaccurate calibration parameters caused by shaking, avoids the safety risks of setting up calibration tools at intersections, and improves detection accuracy and real-time performance.
Smart Images
Figure CN116977447B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and more particularly to the field of automatic camera calibration technology. Specifically, it relates to a method, apparatus, processor, and computer-readable storage medium for automatic calibration processing of roadside cameras based on key point capture. Background Technology
[0002] Camera calibration technology is a key module in existing intelligent road condition systems such as video traffic incident detection and video visibility detection. Its ability to convert images to world coordinates and calculate the absolute distance from road points to the camera forms the foundation for subsequent intelligent detection.
[0003] Traditional camera calibration methods require a calibration object of known size. By establishing a correspondence between points on the calibration object with known coordinates and their corresponding image points, specific algorithms are used to obtain the intrinsic and extrinsic parameters of the camera model. Calibration objects can be categorized into three-dimensional (3D) and planar (2D) calibration objects. 3D calibration objects can be calibrated using a single image, offering high accuracy, but the fabrication and maintenance of high-precision 3D calibration objects are difficult. Planar calibration objects are simpler to fabricate and easier to ensure accuracy than 3D calibration objects, but require two or more images for calibration. Traditional camera calibration methods always require a calibration object during the calibration process, and the fabrication accuracy of the calibration object affects the calibration results. Furthermore, the unsuitability of placing a calibration object in certain situations limits the application of traditional camera calibration methods.
[0004] Roadside cameras experience significant shaking due to inclement weather and the passage of large vehicles, causing a sudden and large deviation between the detected target position and the actual value. Furthermore, prolonged shaking can lead to changes in the camera's position and angle, making it impossible to accurately obtain target location information using the original calibration parameters. In addition, traditional camera calibration methods are complex and costly, failing to meet the needs of real-time roadside monitoring.
[0005] Scene-constrained self-calibration primarily utilizes parallel or orthogonal information within the scene. The intersection of parallel lines in space on the camera image plane is called the vanishing point, a crucial feature in projective geometry. Therefore, many researchers have explored camera self-calibration methods based on vanishing points. However, because these methods are based on absolute quadratic curves or surfaces, their robustness is poor.
[0006] Intersection images contain rich feature information, but the large number of vehicles can easily cause obstruction, and some signs are difficult to identify in low light, leading to problems in the calculation of self-calibration parameters. Many intersections also have uneven ground surfaces, so self-calibration based on lane lines and pedestrian crossings may not yield accurate calibration parameters.
[0007] Therefore, a new camera calibration scheme is needed that can safely, efficiently, and accurately calibrate cameras and easily and quickly calculate the three-dimensional spatial position information of each target vehicle within the camera based on the calibration results. Summary of the Invention
[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, apparatus, processor and computer-readable storage medium for automatic calibration processing of roadside cameras based on key point capture, which is simple, accurate and more convenient and safe.
[0009] To achieve the above objectives, the present invention provides a method, apparatus, processor, and computer-readable storage medium for automatic calibration processing of roadside cameras based on key point capture, as follows:
[0010] The method for automatic calibration of roadside cameras based on key point capture is characterized by the following steps:
[0011] (1) Install a camera, use an RTK drone to measure the camera point, azimuth point and ground key point RTK information for target detection, and obtain the coordinates of the ground key points accordingly;
[0012] (2) Based on the determined key point coordinates, manually and accurately calibrate the homography matrix parameters of the target detection area. By obtaining the coordinates of the road arrow mark pixels on the camera image and the ground points corresponding to the world coordinate system, the homography matrix between the camera image and the ground key points is obtained.
[0013] (3) Use a network model to perform target detection on camera images and perform self-calibration processing on the set key point regions;
[0014] (4) Perform corresponding grayscale processing, edge detection, and contour detection on the images acquired by the camera;
[0015] (5) Set the target contour length threshold according to the set key point area of the acquired image, and extract the vertices of the road surface arrows;
[0016] (6) Determine whether the distance between the manually calibrated key points and the key points detected in the image is within the preset range. If it is, update the homography matrix parameters; otherwise, prompt that the camera position angle is too large and needs to be reinstalled.
[0017] Preferably, step (1) specifically includes the following steps:
[0018] (1.1) Install cameras and use industrial RTK drones to measure the coordinates of camera points, direction points, and key ground points at intersections;
[0019] (1.2) Set the azimuth angle (angle) of the azimuth point B relative to camera A, and calculate the corresponding azimuth angle (angle) according to the following formula:
[0020] ;
[0021] in, for Longitude of a point for Point latitude, for Longitude of a point for Point latitude;
[0022] (1.3) Calculate the ground key points according to the following formula. Distance d between the camera point and the camera point:
[0023] ;
[0024] in, for Longitude of a point for Point latitude, The radius of the Earth;
[0025] Furthermore, following the above calculation method, eight key points were selected, and the distances d1 to d8 between the eight key points and the camera point were calculated.
[0026] (1.4) Set the coordinates of the 8 key points as follows: (d1×cos(angle), d1×sin(angle)), (d2×cos(angle), d2×sin(angle)), (d3×cos(angle), d3×sin(angle)), (d4×cos(angle), d4×sin(angle)), (d5×cos(angle), d5×sin(angle)), (d6×cos(angle), d6×sin(angle)), (d7×cos(angle), d7×sin(angle)), (d8×cos(angle), d8×sin(angle)).
[0027] Preferably, step (2) specifically involves calculating the homography matrix in the following manner. :
[0028] According to key points The coordinates are set by manually selecting eight coordinate points of the vertex pixels of the road arrow marker on the camera image and eight ground points corresponding to them in the world coordinate system. These ground points are the key points selected above. , , , , , , , The relationship between the world coordinate system and the pixel coordinate system is as follows:
[0029] ;
[0030] Where K is the camera intrinsic parameter matrix, s is the scale of arbitrary values, R is the rotation matrix, T is the translation matrix, u and v are pixel coordinates, and x, y, and z are world coordinates. , , is the column vector of matrix R;
[0031] Then the homography matrix between the camera image and the ground for: .
[0032] Preferably, in the world coordinate system, the homography matrix is... The eight ground points set in the model have eight degrees of freedom, which are solved using four pairs of points, as follows:
[0033] Suppose there is a keypoint P with homogeneous coordinates (x, y, 1) and the corresponding pixel with homogeneous coordinates (x', y', 1). Then we have:
[0034] ;
[0035] in, , , , , , , , , All are homography matrices The value in;
[0036] homography matrix Expanding further, we can see:
[0037] ;
[0038] set up =1, from , , , , , , , In each iteration, four different pairs of points and their corresponding pixel coordinates are randomly selected, and the corresponding homography matrix is calculated according to the above formula. Until the homography matrix corresponding to the selected 4 pairs of points is obtained. Highest accuracy.
[0039] Preferably, the homography matrix It also includes checking for bidirectional projection errors in the following manner:
[0040] ;
[0041] Where N is the number of selected points, The average value of the selected points. Standard deviation, This represents the mapping matrix from the pixel coordinate system to the world coordinate system. This represents the mapping matrix from the world coordinate system to the pixel coordinate system. Represents pixel coordinates, Represents world coordinates, Error in pixel coordinate system Error in the world coordinate system;
[0042] After calculating the bidirectional projection error, the accumulated score value is used to represent the homography matrix in the following manner. Precision:
[0043] ;
[0044] in, Indicates the projection error threshold;
[0045] The latest homography matrix is obtained through continuous iteration. And the corresponding score, until the maximum number of iterations is met, and the homography matrix corresponding to the highest score is used. This is the final result of the self-calibration process.
[0046] Preferably, step (3) specifically includes the following steps:
[0047] (3.1) Acquire camera image data and extract key points from the image data. The bounding rectangle of the road surface arrow is set as the key point area;
[0048] (3.2) Use the official YOLOv6 model to detect targets in the current image, and use this to obtain the center pixel coordinates and length and width of the target 2D detection box;
[0049] (3.3) The top, bottom, left and right boundary values of the detection box are obtained by adding or subtracting half of the length and width from the center point of the detection box. If the key point area is included in the detection box, it indicates that there is a target in the current key point area and the process returns to step (3.1). Otherwise, if the key point area is not included in the detection box, it indicates that there is no target in the key point area and the process proceeds to step (3.4).
[0050] (3.4) Obtain network time data and determine whether the time status of the currently obtained key point area without target is within the daytime period. If so, proceed to step (4); otherwise, continue to obtain network time until the time range is met.
[0051] Preferably, step (4) includes:
[0052] Perform image grayscale processing as follows:
[0053] Based on the relationship between the RGB and YUV color spaces, the correspondence between the luminance Y and the three color components R, G, and B is established as follows: Y = 0.3R + 0.59G + 0.11B, and this luminance value represents the grayscale value of the current image; and
[0054] The acquired grayscale image is then used to perform edge detection and contour detection using OpenCV functions.
[0055] Preferably, step (5) specifically includes:
[0056] (5.1) Based on the set key point area, the outer rectangle is divided into regions for road surface arrow vertex detection according to the arrow pointing upwards and downwards;
[0057] (5.2) Set the outline length threshold of the bounding rectangle according to the distance of the target detected in the current camera image, wherein the outline length threshold of the bounding rectangle closer to the camera is greater than the outline length threshold of the bounding rectangle farther from the camera.
[0058] (5.3) For a downward arrow, by traversing and comparing all the contour points of the bounding rectangle, the bottom contour point is taken as the vertex; for an upward arrow, by traversing and comparing all the contour points of the bounding rectangle, the top contour point is taken as the vertex, thereby extracting the vertex of the road surface arrow in the current camera image.
[0059] Preferably, step (6) specifically includes the following steps:
[0060] (6.1) Match the 8 manually calibrated pixels with the points detected in the current camera image according to their distance from the camera;
[0061] (6.2) Determine whether there are fewer than 6 matched key points. If so, proceed to step (6.3); otherwise, proceed to step (6.4).
[0062] (6.3) Further determine whether the current image has failed to match for multiple consecutive frames. If so, prompt to reinstall the camera; otherwise, continue to use the original calibration parameters.
[0063] (6.4) Calculate the homography matrix in the current state. It determines whether the current image has been successfully matched for multiple consecutive frames. If so, it directly updates the calibration parameters; otherwise, the current self-calibration parameters are only used for the current frame.
[0064] The device for automatic calibration processing of roadside cameras based on key point capture is characterized in that the device includes:
[0065] A processor is configured to execute computer-executable instructions;
[0066] The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the method for automatic calibration processing of roadside cameras based on key point capture described above.
[0067] The processor for automatic calibration of roadside cameras based on key point capture is characterized in that the processor is configured to execute computer-executable instructions, and when the computer-executable instructions are executed by the processor, the steps of the method for automatic calibration of roadside cameras based on key point capture described above are implemented.
[0068] The main feature of this computer-readable storage medium is that it stores a computer program thereon, which can be executed by a processor to implement the steps of the method for automatic calibration processing of roadside cameras based on key point capture described above.
[0069] The method, apparatus, processor, and computer-readable storage medium for automatic calibration processing of roadside cameras based on key point capture, as described in this invention, have the following significant advantages:
[0070] (1) Since the lane line is a rectangle with small width at both ends, it is difficult to accurately locate the corner points. In this regard, the technical solution proposes to use the method of matching the pixel coordinates of the vertex of the road arrow mark with the vertex coordinates of the arrow in the world coordinate system to automatically adjust the homography matrix. Compared with the lane line self-calibration, the pixel at the top of the arrow is more accurate. In addition, the technical solution is set to trigger the self-calibration program during the day and under no target occlusion, which results in better detection effect.
[0071] (2) In response to the problem that heavy vehicles, strong winds, heavy rain and other weather conditions can cause the poles at intersections to shake significantly, resulting in inaccurate calibration parameters, the self-calibration method of this technical solution only uses the self-calibration parameters in the current frame when the camera shakes. If the camera position and angle change due to long-term shaking, the calibration parameters are updated to reduce the problem of excessive deviation in the detected target position caused by inaccurate calibration parameters.
[0072] (3) Using calibration boards and other tools at intersections will occupy the road, and not only is it necessary to apply to the relevant departments, but there are also safety risks. The self-calibration method of this technical solution does not require setting up calibration tools at intersections, does not affect traffic order, and is more convenient and safer. Attached Figure Description
[0073] Figure 1 This is a flowchart of the method for automatic calibration processing of roadside cameras based on key point capture according to the present invention.
[0074] Figure 2 This is a schematic diagram illustrating the key point self-calibration process in a specific embodiment of the present invention. Detailed Implementation
[0075] To more clearly describe the technical content of the present invention, the following description is provided in conjunction with specific embodiments.
[0076] Before describing the embodiments of the present invention in detail, it should be noted that, in the following, the terms “comprising,” “including,” or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.
[0077] Please see Figure 1 As shown, this method for automatic calibration processing of roadside cameras based on key point capture includes the following steps:
[0078] (1) Install a camera, use an RTK drone to measure the camera point, azimuth point and ground key point RTK information for target detection, and obtain the coordinates of the ground key points accordingly;
[0079] (2) Based on the determined key point coordinates, manually and accurately calibrate the homography matrix parameters of the target detection area. By obtaining the coordinates of the road arrow mark pixels on the camera image and the ground points corresponding to the world coordinate system, the homography matrix between the camera image and the ground key points is obtained.
[0080] (3) Use a network model to perform target detection on camera images and perform self-calibration processing on the set key point regions;
[0081] (4) Perform corresponding grayscale processing, edge detection, and contour detection on the images acquired by the camera;
[0082] (5) Set the target contour length threshold according to the set key point area of the acquired image, and extract the vertices of the road surface arrows;
[0083] (6) Determine whether the distance between the manually calibrated key points and the key points detected in the image is within the preset range. If it is, update the homography matrix parameters; otherwise, prompt that the camera position angle is too large and needs to be reinstalled.
[0084] In a preferred embodiment of the present invention, step (1) specifically includes the following steps:
[0085] (1.1) Install cameras and use industrial RTK drones to measure the coordinates of camera points, direction points, and key ground points at intersections;
[0086] (1.2) Set the azimuth angle (angle) of the azimuth point B relative to camera A, and calculate the corresponding azimuth angle (angle) according to the following formula:
[0087] ;
[0088] in, for Longitude of a point for Point latitude, for Longitude of a point for Point latitude;
[0089] (1.3) Calculate the ground key points according to the following formula. Distance d between the camera point and the camera point:
[0090] ;
[0091] in, for Longitude of a point for Point latitude, The radius of the Earth;
[0092] Furthermore, following the above calculation method, eight key points were selected, and the distances d1 to d8 between the eight key points and the camera point were calculated.
[0093] (1.4) Set the coordinates of the 8 key points as follows: (d1×cos(angle), d1×sin(angle)), (d2×cos(angle), d2×sin(angle)), (d3×cos(angle), d3×sin(angle)), (d4×cos(angle), d4×sin(angle)), (d5×cos(angle), d5×sin(angle)), (d6×cos(angle), d6×sin(angle)), (d7×cos(angle), d7×sin(angle)), (d8×cos(angle), d8×sin(angle)).
[0094] In a preferred embodiment of the present invention, step (2) specifically involves calculating the homography matrix in the following manner. :
[0095] According to key points The coordinates are set by manually selecting eight coordinate points of the vertex pixels of the road arrow marker on the camera image and eight ground points corresponding to them in the world coordinate system. These ground points are the key points selected above. , , , , , , , The relationship between the world coordinate system and the pixel coordinate system is as follows:
[0096] ;
[0097] Where K is the camera intrinsic parameter matrix, s is the scale of arbitrary values, R is the rotation matrix, T is the translation matrix, u and v are pixel coordinates, and x, y, and z are world coordinates. , , is the column vector of matrix R;
[0098] Then the homography matrix between the camera image and the ground for: .
[0099] In a preferred embodiment of the present invention, the homography matrix is described in the world coordinate system. The eight ground points set in the model have eight degrees of freedom, which are solved using four pairs of points, as follows:
[0100] Suppose there is a keypoint P with homogeneous coordinates (x, y, 1) and the corresponding pixel with homogeneous coordinates (x', y', 1). Then we have:
[0101] ;
[0102] in, , , , , , , , , All are homography matrices The value in;
[0103] homography matrix Expanding further, we can see:
[0104] ;
[0105] set up =1, from , , , , , , , In each iteration, four different pairs of points and their corresponding pixel coordinates are randomly selected, and the corresponding homography matrix is calculated according to the above formula. Until the homography matrix corresponding to the selected 4 pairs of points is obtained. This method offers the highest accuracy. Here, we select 4 pairs of points from 8 pairs each time and substitute them into the system. There are 70 different ways to do this, and we need to substitute them into the system 70 times in total.
[0106] As a preferred embodiment of the present invention, the homography matrix It also includes checking for bidirectional projection errors in the following manner:
[0107] ;
[0108] Where N is the number of selected points, The average value of the selected points. Standard deviation, This represents the mapping matrix from the pixel coordinate system to the world coordinate system. This represents the mapping matrix from the world coordinate system to the pixel coordinate system. Represents pixel coordinates, Represents world coordinates, Error in pixel coordinate system Error in the world coordinate system;
[0109] After calculating the bidirectional projection error, the accumulated score value is used to represent the homography matrix in the following manner. Precision:
[0110] ;
[0111] in, Indicates the projection error threshold;
[0112] The latest homography matrix is obtained through continuous iteration. And the corresponding score, until the maximum number of iterations is met, and the homography matrix corresponding to the highest score is used. This is the final result of the self-calibration process.
[0113] In a preferred embodiment of the present invention, step (3) specifically includes the following steps:
[0114] (3.1) Acquire camera image data and extract key points from the image data. The bounding rectangle of the road surface arrow is set as the key point area;
[0115] (3.2) Use the official YOLOv6 model to detect targets in the current image, and use this to obtain the center pixel coordinates and length and width of the target 2D detection box;
[0116] (3.3) The top, bottom, left and right boundary values of the detection box are obtained by adding or subtracting half of the length and width from the center point of the detection box. If the key point area is included in the detection box, it indicates that there is a target in the current key point area and the process returns to step (3.1). Otherwise, if the key point area is not included in the detection box, it indicates that there is no target in the key point area and the process proceeds to step (3.4).
[0117] (3.4) Obtain network time data and determine whether the time status of the currently obtained key point area without target is within the daytime period. If so, proceed to step (4); otherwise, continue to obtain network time until the time range is met.
[0118] In a preferred embodiment of the present invention, step (4) includes:
[0119] Perform image grayscale processing as follows:
[0120] Based on the relationship between the RGB and YUV color spaces, the correspondence between the luminance Y and the three color components R, G, and B is established as follows: Y = 0.3R + 0.59G + 0.11B, and this luminance value represents the grayscale value of the current image; and
[0121] The acquired grayscale image is then used to perform edge detection and contour detection using OpenCV functions.
[0122] In a preferred embodiment of the present invention, step (5) specifically comprises:
[0123] (5.1) Based on the set key point area, the outer rectangle is divided into regions for road surface arrow vertex detection according to the arrow pointing upwards and downwards;
[0124] (5.2) Set the outline length threshold of the bounding rectangle according to the distance of the target detected in the current camera image, wherein the outline length threshold of the bounding rectangle closer to the camera is greater than the outline length threshold of the bounding rectangle farther from the camera.
[0125] (5.3) For a downward arrow, by traversing and comparing all the contour points of the bounding rectangle, the bottom contour point is taken as the vertex; for an upward arrow, by traversing and comparing all the contour points of the bounding rectangle, the top contour point is taken as the vertex, thereby extracting the vertex of the road surface arrow in the current camera image.
[0126] In a preferred embodiment of the present invention, step (6) specifically includes the following steps:
[0127] (6.1) Match the 8 manually calibrated pixels with the points detected in the current camera image according to their distance from the camera;
[0128] (6.2) Determine whether there are fewer than 6 matched key points. If so, proceed to step (6.3); otherwise, proceed to step (6.4).
[0129] (6.3) Further determine whether the current image has failed to match for multiple consecutive frames. If so, prompt to reinstall the camera; otherwise, continue to use the original calibration parameters.
[0130] (6.4) Calculate the homography matrix in the current state. It determines whether the current image has been successfully matched for multiple consecutive frames. If so, it directly updates the calibration parameters; otherwise, the current self-calibration parameters are only used for the current frame.
[0131] like Figure 1As shown, the main processing steps of this method for automatic calibration of roadside cameras based on key point capture include:
[0132] (1) Install a camera and use an RTK drone to measure the RTK information of the camera point, azimuth point, and ground key points. The coordinates of the ground key points can be calculated using the camera point and azimuth point.
[0133] (2) Manually and accurately calibrate the homography matrix parameters, find the coordinates of the road arrow mark pixels on the image and the corresponding ground points in the world coordinate system, and calculate the homography matrix between the image and the ground.
[0134] (3) Use the network model to detect targets. When there are no targets in the key point area and it is daytime, self-calibration begins.
[0135] (4) Perform grayscale processing, edge detection, and contour detection on the images obtained by the camera.
[0136] (5) Set the contour length threshold for image regions and extract the arrow vertices.
[0137] (6) Determine the distance between the manually set key points and the key points detected in the image. If the distance is within the range, use the self-calibration parameters in the current frame. If the matching is successful for multiple consecutive frames, update the homography matrix parameters. If the distance is outside the range for multiple consecutive frames, the camera position angle is too large and needs to be reinstalled.
[0138] In one specific embodiment, the actual processing procedure of the method for automatic calibration of roadside cameras based on key point capture is as follows:
[0139] 1. Install a camera and use an industrial RTK drone to measure the coordinates of the camera point, directional point, and key points on the road surface at the intersection. By determining the coordinate system direction and origin position using the camera point and directional point, the coordinates of the key points on the road surface can be calculated.
[0140] (1) Here we set the azimuth angle of the azimuth point B relative to the camera A, that is, A is the current position and B is the camera direction point.
[0141] (2) The formula for calculating the azimuth angle is:
[0142] ;
[0143] in, for Longitude of a point for Point latitude, for Longitude of a point for Point latitude;
[0144] (3) The distance between the key point and the camera point is:
[0145] ;
[0146] in, for Longitude of a point for Point latitude, The radius of the Earth;
[0147] (4) Key points The coordinates are (d×cos(angle), d×sin(angle)).
[0148] 2. Manually calibrate the homography matrix parameters. Theoretically, selecting four key points is sufficient to calculate the homography matrix. However, due to pixel selection errors and UAV measurement errors of key point positions, the calculation results are inaccurate. Therefore, multiple points are generally selected, and the RANSAC method is used to reduce the impact of individual point errors on the calculation results. This technical solution selects eight coordinate points of the road arrow marker pixels on the image and eight ground points corresponding to them in the world coordinate system. The RANSAC method is used to calculate the homography matrix between the image and the ground. It should be noted that the eight selected coordinate points should be the arrow vertices. While ensuring the arrow is clear in the image, the greater the distance between the eight points, the better. Specifically:
[0149] (2.1) Homography matrix It can be understood as describing the positional mapping relationship between an object in the world coordinate system and the pixel coordinate system.
[0150] Taking chessboard-based relative pose estimation as an example, the world coordinate system is defined on the chessboard, i.e., z=0. Therefore, the relationship between the world coordinate system and the pixel coordinate system is as follows:
[0151] ;
[0152] therefore, This is the homography matrix between the imaging plane and the calibration plane. Where K is the camera intrinsic parameter matrix, s is the scale of any arbitrary size, R is the rotation matrix, T is the translation matrix, u and v are pixel coordinates, and x, y, and z are world coordinates. , , is the column vector of matrix R;
[0153] The homography matrix has 8 degrees of freedom and requires solving for 4 pairs of points. Let's assume the aligned coordinates of corresponding points in two images are (x', y', 1) and (x, y, 1), then we have:
[0154] ;
[0155] , , , , , , , , Homography matrix The value of .
[0156] The matrix expansion yields three equations:
[0157] (Equation 1)
[0158] (Equation 2)
[0159] (Equation 3)
[0160] Substituting (Equation 3) into (Equation 1) and (Equation 2) respectively, we get:
[0161] ;
[0162] In other words, one pair of points corresponds to two equations. Since a homogeneous coordinate system is used here, it means that arbitrary scaling is possible. For example, if we... Multiplying by any non-zero constant k does not change the result of the equation, so the homography matrix is actually... It has only 8 degrees of freedom.
[0163] Direct settings =1. Next, eight equations are listed based on the pixels of four corresponding points, and the homography matrix is obtained by solving the equations. Here, the homography matrix is calculated by randomly selecting four pairs of points. The subsequent step is to calculate the precision of this homography matrix. This process is repeated until the homography matrix obtained from the four pairs of points with the highest precision is found.
[0164] (2.2) The homography matrix check uses bidirectional projection error, and the formula for bidirectional projection error is as follows:
[0165] ;
[0166] N is the number of points to be selected. The average value of the points. Standard deviation, This represents the mapping matrix from the pixel coordinate system to the world coordinate system. This represents the mapping matrix from the world coordinate system to the pixel coordinate system. Represents pixel coordinates, Represents world coordinates, Error in pixel coordinate system This represents the error in the world coordinate system.
[0167] After calculating the bidirectional projection error, the accuracy of the homography matrix is represented by the cumulative score value. The score calculation formula is as follows:
[0168] ;
[0169] In the formula, This represents the projection error threshold, and N is the number of selected points.
[0170] By calculating and checking the homography matrix, we have completed the estimation of the homography matrix for a random feature point pair. To improve the quality of the estimated homography matrix, we repeat these two steps iteratively (i.e., RANSAC iteration), obtaining the latest homography matrix in each iteration. The corresponding score is used until the maximum number of iterations, and the homography matrix corresponding to the highest score is taken as the final result.
[0171] 3. Acquire camera image data, define the bounding rectangle of each arrow as the keypoint region, and use the official YOLOv6 model to detect targets in the image. Obtain the center pixel coordinates and length and width of the target's 2D detection box. The top, bottom, left, and right boundary values of the detection box can be obtained by adding or subtracting half of the length and width from the center point of the detection box. If the keypoint region is not contained within the detection box boundaries, it means there is no target in the keypoint region. Obtain the network time, setting the daytime range to 08:00-17:00 (the latest sunrise to earliest sunset time in China). If there is no target in the keypoint region and it is daytime, proceed to the next step; otherwise, wait.
[0172] 4. Grayscale processing of the image can visually increase contrast and highlight the target area. The principle of grayscale processing is based on the fact that in the YUV color space, the physical meaning of the Y component is the brightness of a point. This value reflects the brightness level. According to the relationship between the RGB and YUV color spaces, the correspondence between the brightness Y and the three color components R, G, and B can be established: Y = 0.3R + 0.59G + 0.11B. This brightness value is used to express the grayscale value of the image.
[0173] 5. Next, edge detection is performed. The basic idea of the algorithm is to find the location with the strongest grayscale intensity change in an image. Here, the OpenCV function cv::Canny(cv::InputArray src, cv::OutputArray dst, double threshold1, double threshold2, int aperture_size=3) is used, where src is the grayscale image obtained in step 4, dst is the edge detection output image, threshold1 is 50 by default, threshold2 is 150 by default, and aperture_size is 3 by default.
[0174] 6. Next, contour detection is performed. A contour can be simply considered as a curve connecting consecutive points, having the same color or grayscale. Here, the OpenCV function `cv::findContours(edgeImage, contours, cv::noArray(), cv::RETR_LIST, cv::CHAIN_APPROX_SIMPLE)` is used, where `edgeImage` is the edge detection image obtained in step 5, `contours` are the extracted contours, `cv::noArray()` is the default value, `cv::RETR_LIST` indicates that all contours are detected, and `cv::CHAIN_APPROX_SIMPLE` indicates that only the inflection point information of the contours is saved.
[0175] 7. Set key point regions and detect arrow vertices separately, that is, detect the regions where the arrows point upwards and the regions where the arrows point downwards separately. For example... Figure 2 As shown, the upper left region is a downward arrow. The contour length threshold is set to 30, and the contour point coordinates are traversed to obtain the coordinates of the bottom vertex. The upper right region is traversed to obtain the top vertex. The lower region contour has a threshold of 400, and the top vertex is obtained by traversing.
[0176] Furthermore, in one specific embodiment, since the bounding rectangle of each road arrow is a key point region, the contour length threshold is divided according to distance (the side where the camera is located is the near point, and the opposite side is the far point), with the far point set to 30 and the near point set to 400. For downward arrows, the contour point coordinates are compared, and the bottommost point is taken as the vertex; for upward arrows, the contour point coordinates are compared, and the topmost point is taken as the vertex.
[0177] 8. Set a certain range of the 8 manually calibrated pixels and match it with the points detected in the image. The larger the range, the better, but it is necessary to ensure that the range includes this arrow and does not include other markers. This range will be set according to the distance from the camera (the side where the camera is located is the near point, and the opposite side is the far point). In practical applications, set the distant pixels to 15 and the near pixels to 30.
[0178] like Figure 2 As shown, in one specific embodiment, the distance between the far-end arrow and other markers in the image is small, so a range of 15 pixels is set; the distance between the near-end arrow and other markers is large, so a range of 30 pixels is set. The red circle represents the matching range, the green dots represent key points detected in the image, and the red dots represent the pixel coordinates of manually calibrated points. Homography matrix calculation begins when at least 6 points match. If 60 consecutive frames match successfully, the calibration parameters are updated; otherwise, the self-calibration parameters are only used in the current frame. If fewer than 6 points match and 60 consecutive frames fail to match, it indicates that the camera position and angle have changed too much and cannot meet the requirements for roadside target detection, requiring the camera to be reinstalled.
[0179] The device for automatic calibration processing of roadside cameras based on key point capture includes:
[0180] A processor is configured to execute computer-executable instructions;
[0181] The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the method for automatic calibration processing of roadside cameras based on key point capture described above.
[0182] The processor for automatic calibration processing of roadside cameras based on key point capture is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the method for automatic calibration processing of roadside cameras based on key point capture described above.
[0183] The computer-readable storage medium contains a computer program that can be executed by a processor to implement the steps of the method for automatic calibration processing of roadside cameras based on key point capture described above.
[0184] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.
[0185] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution device.
[0186] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.
[0187] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.
[0188] In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "embodiment," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.
[0189] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
[0190] The method, apparatus, processor, and computer-readable storage medium for automatic calibration processing of roadside cameras based on key point capture, as described in this invention, have the following significant advantages:
[0191] (1) Since the lane line is a rectangle with small width at both ends, it is difficult to accurately locate the corner points. In this regard, the technical solution proposes to use the method of matching the pixel coordinates of the vertex of the road arrow mark with the vertex coordinates of the arrow in the world coordinate system to automatically adjust the homography matrix. Compared with the lane line self-calibration, the pixel at the top of the arrow is more accurate. In addition, the technical solution is set to trigger the self-calibration program during the day and under no target occlusion, which results in better detection effect.
[0192] (2) In response to the problem that heavy vehicles, strong winds, heavy rain and other weather conditions can cause the poles at intersections to shake significantly, resulting in inaccurate calibration parameters, the self-calibration method of this technical solution only uses the self-calibration parameters in the current frame when the camera shakes. If the camera position and angle change due to long-term shaking, the calibration parameters are updated to reduce the problem of excessive deviation in the detected target position caused by inaccurate calibration parameters.
[0193] (3) Using calibration boards and other tools at intersections will occupy the road, and not only is it necessary to apply to the relevant departments, but there are also safety risks. The self-calibration method of this technical solution does not require setting up calibration tools at intersections, does not affect traffic order, and is more convenient and safer.
[0194] In this specification, the invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and variations can be made without departing from the spirit and scope of the invention. Therefore, the specification and drawings should be considered illustrative rather than restrictive.
Claims
1. A method for automatic calibration processing of roadside cameras based on key point capture, characterized in that, The method includes the following steps: (1) Install a camera, use an RTK drone to measure the camera point, azimuth point and ground key point RTK information for target detection, and obtain the coordinates of the ground key points accordingly; (2) Based on the determined key point coordinates, manually and accurately calibrate the homography matrix parameters of the target detection area. By obtaining the coordinates of the road arrow mark pixels on the camera image and the ground points corresponding to the world coordinate system, the homography matrix between the camera image and the ground key points is obtained. (3) Use a network model to perform target detection on camera images and perform self-calibration processing on the set key point regions; (4) Perform corresponding grayscale processing, edge detection, and contour detection on the images acquired by the camera; (5) Set the target contour length threshold according to the set key point area of the acquired image, and extract the vertices of the road surface arrows; (6) Determine whether the distance between the manually calibrated key points and the key points detected in the image is within the preset range. If it is, update the homography matrix parameters; otherwise, prompt that the camera position angle is too large and needs to be reinstalled.
2. The method for automatic calibration processing of roadside cameras based on key point capture according to claim 1, characterized in that, Step (1) specifically includes the following steps: (1.1) Install cameras and use industrial RTK drones to measure the coordinates of camera points, direction points, and key ground points at intersections; (1.2) Set the azimuth angle (angle) of the azimuth point B relative to camera A, and calculate the corresponding azimuth angle (angle) according to the following formula: in, for Longitude of a point for Point latitude, for Longitude of a point for Point latitude; (1.3) Calculate the ground key points according to the following formula. Distance d between the camera point and the camera point: ； in, for Longitude of a point for Point latitude, The radius of the Earth; Furthermore, select 8 key points according to the above formula, and calculate the distances d1 to d8 between the 8 key points and the camera point; (1.4) Set the coordinates of the 8 key points as follows: (d1×cos(angle), d1×sin(angle)), (d2×cos(angle), d2×sin(angle)), (d3×cos(angle), d3×sin(angle)), (d4×cos(angle), d4×sin(angle)), (d5×cos(angle), d5×sin(angle)), (d6×cos(angle), d6×sin(angle)), (d7×cos(angle), d7×sin(angle)), (d8×cos(angle), d8×sin(angle)).
3. The method for automatic calibration processing of roadside cameras based on key point capture according to claim 2, characterized in that, Step (2) specifically involves calculating the homography matrix in the following manner. : According to key points The coordinates are set by manually selecting eight coordinate points of the vertex pixels of the road arrow marker on the camera image and eight ground points corresponding to them in the world coordinate system. These ground points are the key points selected above. , , , , , , , The relationship between the world coordinate system and the pixel coordinate system is as follows: Where K is the camera intrinsic parameter matrix, s is the scale of any value, R is the rotation matrix, and T is the translation matrix. , x, y, z are pixel coordinates, and x, y, z are world coordinates. , , is the column vector of matrix R; Then the homography matrix between the camera image and the ground for: .
4. The method for automatic calibration processing of roadside cameras based on key point capture according to claim 3, characterized in that, In the world coordinate system, the homography matrix is described. The eight ground points set in the model have eight degrees of freedom, which are solved using four pairs of points, as follows: Suppose there is a keypoint P with homogeneous coordinates (x, y, 1) and the corresponding pixel with homogeneous coordinates (x', y', 1). Then we have: in, , , , , , , , , All are homography matrices The value in; homography matrix Expanding further, we can see: set up =1, from , , , , , , , In each iteration, four different pairs of points and their corresponding pixel coordinates are randomly selected, and the corresponding homography matrix is calculated according to the above formula. Until the homography matrix corresponding to the selected 4 pairs of points is obtained. Highest accuracy.
5. The method for automatic calibration processing of roadside cameras based on key point capture according to claim 4, characterized in that, The homography matrix It also includes checking for bidirectional projection errors in the following manner: Where N is the number of selected points, The average value of the selected points. Standard deviation, This represents the mapping matrix from the pixel coordinate system to the world coordinate system. This represents the mapping matrix from the world coordinate system to the pixel coordinate system. Represents pixel coordinates, Represents world coordinates, Error in pixel coordinate system Error in the world coordinate system; After calculating the bidirectional projection error, the accumulated score value is used to represent the homography matrix in the following manner. Precision: in, Indicates the projection error threshold; The latest homography matrix is obtained through continuous iteration. And the corresponding score, until the maximum number of iterations is met, and the homography matrix corresponding to the highest score is used. This is the final result of the self-calibration process.
6. The method for automatic calibration processing of roadside cameras based on key point capture according to claim 3, characterized in that, Step (3) specifically includes the following steps: (3.1) Acquire camera image data and extract key points from the image data. The bounding rectangle of the road surface arrow is set as the key point area; (3.2) Use the official YOLOv6 model to detect targets in the current image, and use this to obtain the center pixel coordinates and length and width of the target 2D detection box; (3.3) The top, bottom, left and right boundary values of the detection box are obtained by adding or subtracting half of the length and width from the center point of the detection box. If the key point area is included in the detection box, it indicates that there is a target in the current key point area and the process returns to step (3.1). Otherwise, if the key point area is not included in the detection box, it indicates that there is no target in the key point area and the process proceeds to step (3.4). (3.4) Obtain network time data and determine whether the time status of the currently obtained key point area without target is within the daytime period. If so, proceed to step (4); otherwise, continue to obtain network time until the time range is met.
7. The method for automatic calibration processing of roadside cameras based on key point capture according to claim 1, characterized in that, Step (4) includes: Perform image grayscale processing as follows: Based on the relationship between the RGB and YUV color spaces, the correspondence between the luminance value Y and the three color components R, G, and B is established as follows: Y = 0.3R + 0.59G + 0.11B, and this luminance value represents the grayscale value of the current image; and The acquired grayscale image is then used to perform edge detection and contour detection using OpenCV functions.
8. The method for automatic calibration processing of roadside cameras based on key point capture according to claim 6, characterized in that, The specific steps (5) are as follows: (5.1) Based on the set key point areas, the outer rectangle is divided into regions for road surface arrow vertex detection according to the arrow pointing upwards and downwards; (5.2) Set the outline length threshold of the bounding rectangle according to the distance of the target detected in the current camera image, wherein the outline length threshold of the bounding rectangle closer to the camera is greater than the outline length threshold of the bounding rectangle farther from the camera. (5.3) For a downward arrow, by traversing and comparing all the contour points of the bounding rectangle, the bottom contour point is taken as the vertex; for an upward arrow, by traversing and comparing all the contour points of the bounding rectangle, the top contour point is taken as the vertex, thereby extracting the vertex of the road surface arrow in the current camera image.
9. The method for automatic calibration processing of roadside cameras based on key point capture according to claim 8, characterized in that, Step (6) specifically includes the following steps: (6.1) Match the 8 manually calibrated pixels with the points detected in the current camera image according to their distance from the camera; (6.2) Determine whether there are fewer than 6 matched key points. If so, proceed to step (6.3); otherwise, proceed to step (6.4). (6.3) Further determine whether the current image has failed to match for multiple consecutive frames. If so, prompt to reinstall the camera; otherwise, continue to use the original calibration parameters. (6.4) Calculate the homography matrix in the current state. It determines whether the current image has been successfully matched for multiple consecutive frames. If so, it directly updates the calibration parameters; otherwise, the current self-calibration parameters are only used for the current frame.
10. A device for automatic calibration processing of roadside cameras based on key point capture, characterized in that, The device includes: A processor is configured to execute computer-executable instructions; The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the method for automatic calibration processing of roadside cameras based on key point capture as described in any one of claims 1 to 9.
11. A processor for automatic calibration processing of roadside cameras based on key point capture, characterized in that, The processor is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the method for automatic calibration processing of roadside cameras based on key point capture as described in any one of claims 1 to 9.
12. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method for automatic calibration processing of roadside cameras based on key point capture as described in any one of claims 1 to 9.