A method for calibrating a homography camera based on road traffic signs and a storage medium

Abstract

This invention discloses a homography camera calibration method and storage medium based on road traffic signs, belonging to the interdisciplinary field of intelligent transportation and computer vision. The method includes: acquiring roadside image data and inputting it into a pre-trained road traffic sign recognition model to obtain the type and geometric representation data of the road traffic signs; extracting a set of pixel calibration points from the geometric representation data according to the type of road traffic sign, and obtaining the construction standard parameters of the corresponding type of road traffic sign, thereby generating a set of local world calibration points corresponding one-to-one with the pixel calibration points, obtaining multiple calibration point pairs; solving the homography matrix from the image coordinate system to the local world coordinate system based on the multiple calibration point pairs; judging whether the homography matrix meets the calibration accuracy requirements through reprojection, and taking the homography matrix that meets the calibration accuracy as the final homography matrix. This invention can overcome the dependence on manual intervention in the calibration process, reduce error accumulation in dynamic environments, and improve calibration accuracy.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation and computer vision interdisciplinary technology, and in particular to a homography camera calibration method and storage medium based on road traffic signs. Background Technology

[0002] In the field of surveillance technology, accurate environmental perception and positioning are crucial for ensuring system security. Especially in complex surveillance environments, such as those requiring efficient monitoring systems, precise image calibration information, and accurate environmental perception and location, accurately acquiring and analyzing target object information to successfully optimize monitoring effectiveness has become one of the core issues in improving surveillance system performance.

[0003] Roadside surveillance video is widely used for traffic operation monitoring and safety analysis due to its low deployment cost and wide coverage. However, the pixel coordinates output by the camera are two-dimensional imaging information. Without reliable calibration and coordinate transformation, it is difficult to obtain the position and speed of traffic participants at a real scale, thus affecting the credibility of traffic parameter statistics and safety evaluation.

[0004] Currently, most calibration methods rely on manual intervention or adjustment, requiring manual selection of calibration points on the screen and input of real-world coordinates. In practical applications, manual operation is not only time-consuming but also prone to error accumulation, resulting in high costs, poor repeatability, uncontrollable point-to-point errors, and the potential for outlier point pairs under conditions of occlusion, lighting changes, and camera shake, affecting calibration accuracy and system stability. Furthermore, traditional methods often ignore dynamic changes during the calibration process, especially in environments with multiple road signs and complex conditions, where manual adjustment struggles to achieve optimal calibration accuracy and lacks flexible adaptive capabilities.

[0005] Therefore, there is a need for a technical solution that can automatically identify road traffic signs and generate point pairs without human interaction, thereby automatically calculating the homography transformation matrix and performing quantifiable quality assessment and adaptive correction on the results. Summary of the Invention

[0006] The purpose of this invention is to provide a homography camera calibration method and storage medium based on road traffic signs. By identifying road traffic signs and markings in roadside images or videos, and using the construction standard parameters of road traffic signs as geometric constraints, pixel-world coordinate corresponding point pairs are generated, thereby realizing the homography calibration of cameras in road traffic environments. This breaks through the dependence on manual intervention in the calibration process, reduces the accumulation of errors in dynamic environments, and improves calibration accuracy.

[0007] The technical solution adopted in this invention will be described in detail below.

[0008] On one hand, the present invention provides a homography camera calibration method based on road traffic signs, comprising:

[0009] Acquire roadside image data;

[0010] Input roadside image data into a pre-trained road traffic sign recognition model to obtain the type and geometric representation data of the road traffic signs output by the road traffic sign recognition model.

[0011] Based on the type of road traffic sign, extract the set of pixel calibration points from the geometric representation data and obtain the construction standard parameters of the corresponding type of road traffic sign;

[0012] Based on the construction standard parameters, a set of local world calibration points corresponding one-to-one with the pixel calibration points is generated, resulting in multiple calibration point pairs;

[0013] Based on the multiple calibration point pairs, the homography matrix from the image coordinate system to the local world coordinate system is solved;

[0014] Based on the homography matrix, the reprojection error index is calculated for the multiple calibration point pairs to determine whether the homography matrix meets the calibration accuracy requirements. If the homography matrix does not meet the calibration accuracy requirements, the calibration point pairs are optimized, and the homography matrix is ​​resolved based on the optimized calibration point pairs.

[0015] The homography matrix that meets the calibration accuracy for reprojection error is taken as the final homography matrix.

[0016] The above technical solution uses the standard parameters for road traffic sign construction as a reference to achieve the fusion of calibration point pairs of the homography matrix, which has high calibration accuracy. Furthermore, by calculating the reprojection error, it realizes the dynamic adaptive adjustment of the homography matrix, making the calibration process more efficient and accurate.

[0017] It should be noted that the roadside image data includes video frame data. The surveillance cameras used to capture roadside image data can be installed on the roadside, at intersections, or on highways, etc., to collect images and videos of road traffic conditions in real time. To ensure the quality of the video data, high-resolution cameras are preferred to ensure that details of traffic participants, such as vehicles and pedestrians, are clearly captured.

[0018] The road traffic sign recognition model can be pre-trained based on a large number of labeled image samples containing various types of road traffic signs, enabling it to output geometric representation data such as key point or line segment parameters for calibration, as well as the type of road traffic sign. Specific details can be found in existing technologies and will not be elaborated further.

[0019] Optionally, the types of road traffic signs include at least one or more of the following: zebra crossings, lane lines, and stop lines;

[0020] The geometric representation data includes at least one or more of the following: key point coordinates, line segment endpoints and direction parameters, polylines, boundary polygons, and pixel-level segmentation masks.

[0021] The construction standard parameters for the road traffic signs shall include at least one or more of the following: zebra stripe width and spacing, lane line width, lane width, and stop line width.

[0022] Based on the construction standard parameters of road traffic signs, a local world coordinate system can be constructed with any point of the road traffic sign itself as the origin, thereby obtaining the set of local world calibration points of multiple key points of the road traffic sign in the local world coordinate system.

[0023] Optionally, the step of extracting a set of pixel calibration points from the geometric representation data according to the type of road traffic sign includes:

[0024] The pixel coordinates of key points corresponding to each road traffic sign are calculated based on the geometric representation data.

[0025] Determine whether the number of key points corresponding to each road traffic sign has reached the preset lower limit m: if not, select at least m key points from all the currently obtained key points to obtain a set of pixel calibration points; if it has reached the lower limit, select the key points of a single road traffic sign whose number of key points has reached the preset lower limit m to obtain a set of pixel calibration points.

[0026] The above technical solution selects road traffic signs with a sufficient number of pixel calibration points during pixel calibration point generation. If the number of pixel calibration points that can be extracted for a single road traffic sign is insufficient, it triggers joint point extraction from multiple road traffic signs. This ensures that the number of calibration points in the generated pixel calibration point set meets the requirements for solving the homography matrix, guaranteeing accuracy. m is preferably 6. In some possible embodiments, the initial point extraction may be based solely on the value of m.

[0027] Optionally, when extracting the set of pixel calibration points from the geometric representation data according to the type of road traffic sign:

[0028] If the road traffic sign type is a zebra crossing, then extract multiple key points from the four outer corners of the zebra crossing, the endpoints of the strip boundary line, and the intersection of the strip boundary and the curb boundary as pixel calibration points to obtain the pixel calibration point set.

[0029] If the road traffic sign type is a lane line or a stop line, then extract the endpoints of the lane line or stop line segments, the intersections of the line segments, and / or multiple key points from the equidistant sampling points along the preset direction as pixel calibration points to obtain the pixel calibration point set.

[0030] Optionally, the method of the present invention further includes: performing distribution constraint verification on the set of pixel calibration points or the plurality of calibration point pairs, including:

[0031] Calculate one or more distribution indicators, such as the convex hull area of ​​multiple calibration point pairs or pixel calibration point sets, and the minimum distance between different pixel calibration points or calibration point pairs, and compare them with the corresponding distribution indicator thresholds. If the distribution indicators are qualified, the current multiple calibration point pairs or pixel calibration point sets are maintained. If the distribution indicators are not qualified, multiple calibration point pairs or pixel calibration point sets with qualified distribution indicators are obtained by supplementing pixel calibration points or calibration point pairs or reselecting roadside images.

[0032] The threshold for the distribution index corresponding to the convex hull area is set to be no less than 30% of the pixels in the image.

[0033] The threshold for the distributed index corresponding to the minimum distance is set to be no less than 5% of the width of the image.

[0034] It should be noted that, as described in the above technical solution, distribution constraint verification can be performed after the pixel calibration point set is generated, or after the calibration point pair is generated. Since the local world calibration points are calculated based on the road traffic sign type and construction standard parameters, once the pixel calibration points are determined, the corresponding local world calibration points can be calculated. Therefore, performing distribution constraint verification after obtaining the pixel calibration point set is also feasible.

[0035] The supplementary pixel calibration points or calibration point pairs refer to the following: If the number of key points corresponding to a single road traffic sign is greater than the preset lower limit m, but the initial point selection only selected m key points corresponding to that road traffic sign, then other key points of the same road traffic sign can be added; if the original pixel calibration points or calibration point pairs correspond only to a single road traffic sign, then pixel calibration points of other single or multiple road traffic signs, or local world calibration points corresponding to the pixel calibration points, can be selected from the pixel calibration points extracted from the geometric representation data of the same image and added to the original pixel calibration point set or calibration point pair set. Then, the distribution constraint verification is performed on the supplemented pixel calibration points or calibration point pairs until the distribution constraint is satisfied.

[0036] Optionally, generating a set of local world calibration points corresponding one-to-one with the pixel calibration points based on the construction standard parameters includes:

[0037] Select any key point of the road traffic sign that corresponds to the pixel calibration point as the origin to construct a local world coordinate system;

[0038] Based on the construction standard parameters of road traffic signs, calculate the coordinates of the local world calibration points that correspond one-to-one with the pixel calibration points in the local world coordinate system.

[0039] In some possible embodiments, the road traffic sign type is a zebra crossing, and its standard construction parameters include a strip width of [missing information]. The strip length is Strip spacing If a pixel calibration point includes at least four pixel corner points of at least one strip, then based on the construction standard parameters of the road traffic sign, calculate the coordinates of the local world calibration point corresponding to the pixel corner point in the local world coordinate system, including:

[0040] If we take one of the pixel corner points of one of the strips as the origin of the local world coordinate system, then the coordinates of the local world calibration points corresponding to its four pixel corner points are:

[0041] ,

[0042] Among them, superscript Indicates transpose. That is, the origin of the selected local world coordinate system. (Based on pixel corner points) coordinates For example, it represents the corner point of a pixel. The X-axis and Y-axis coordinates are respectively equal to the strip length in the construction standard parameters. and strip width is 1 represents the w component of homogeneous coordinates, which is an auxiliary component that enables projective transformation of points in a two-dimensional plane.

[0043] Taking the strip containing the origin as the first strip, along the Y-axis of the local world coordinate system, the... The local world calibration point coordinates corresponding to the four pixel corner points of each strip are:

[0044] ,

[0045] Among them, the pixel corners of each strip and The Y-axis coordinate is: .

[0046] Based on the above technical solutions, for the case where the road traffic sign type is a zebra crossing, if the entire set of pixel calibration points is obtained from the zebra crossing key points, but there are some unclear stripes in the roadside image, then the key points of multiple non-adjacent stripes can be selected to form a set of pixel calibration points, and then the corresponding local world calibration point coordinates can be calculated by the above formula.

[0047] Optionally, the step of solving the homography matrix from the image coordinate system to the local world coordinate system based on the plurality of calibration point pairs includes:

[0048] For each calibration point pair, based on the homography projection relationship, two linear constraint equations with the elements in the homography matrix as unknowns are constructed using the DLT algorithm.

[0049] Construct a homogeneous linear equation system based on the linear constraint equations of multiple calibration point pairs;

[0050] Based on the homogeneous linear equation system, the homography matrix is ​​obtained through singular value decomposition.

[0051] Optionally, based on the homography matrix, the reprojection error index is calculated for the plurality of calibration point pairs, including:

[0052] For each calibration point pair, calculate the projected pixel in the pixel coordinate system by projecting the local world calibration point onto the pixel coordinate system through the inverse of the homography matrix; calculate the Euclidean distance between the projected pixel and the pixel calibration point as the reprojection error of the calibration point pair;

[0053] Alternatively, for each calibration point pair, calculate the reprojected coordinate point of the pixel calibration point projected onto the local world coordinate system through the homography matrix; calculate the Euclidean distance between the reprojected coordinate point and the local world calibration point as the reprojection error of the calibration point pair;

[0054] Based on the reprojection error of each calibration point pair, the reprojection error index of multiple calibration point pairs is calculated, and the reprojection error index includes the average error and the maximum error.

[0055] Optionally, determining whether the homography matrix meets the calibration accuracy requirements includes determining whether the following conditions are met: a. the average error is not greater than a first error threshold, b. the maximum error is not greater than a second error threshold; if conditions a and b are met simultaneously, then the homography matrix is ​​determined to meet the calibration accuracy requirements.

[0056] The first error threshold corresponding to the average error is set to 2% of the image pixel width; the second error threshold corresponding to the maximum error is set to 2.5% of the image pixel width.

[0057] Optionally, the optimization of calibration point pairs includes: determining outlier calibration point pairs based on the magnitude of the reprojection error of each calibration point pair and the average error, and deleting outlier calibration point pairs.

[0058] If the number of calibration point pairs after deleting outlier calibration point pairs is not less than the preset m pairs, then the set of calibration point pairs after deleting outlier calibration point pairs will be used as the optimized set of calibration point pairs.

[0059] If the number of calibration point pairs after deleting outlier calibration point pairs is less than the preset m pairs, then a new set of calibration point pairs is obtained by re-identifying road traffic sign types and geometric representation data, extracting pixel calibration point sets, obtaining construction standard parameters, and generating local world calibration point sets based on real-time roadside image data.

[0060] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the homography camera calibration method based on road traffic signs as described in the first aspect.

[0061] Beneficial effects

[0062] The homography camera calibration method of this invention uses a neural network model to identify the type and geometric representation data of road traffic signs. Based on this, and constrained by the construction parameters of the road traffic signs, it achieves homography calibration of cameras in the road traffic environment. This not only improves calibration accuracy but also effectively addresses error accumulation in dynamic environments. More importantly, this invention breaks through the reliance on manual intervention in traditional calibration techniques, achieving automatic calculation and optimization of the homography matrix without manual intervention. This highly automates the calibration process, avoiding the complexity and potential errors of manual operation.

[0063] In the implementation of the method of this invention, the evaluation of calibration accuracy and the optimization of calibration point pairs realize an adaptive homography matrix adjustment strategy. This allows the homography matrix to be optimized and dynamically adjusted in real time based on feedback from sensor data, rather than relying on preset manual calibration parameters, thus making the calibration process more efficient and accurate. This method can not only perform high-precision calibration in various road environments, but also effectively improve the stability and accuracy of the system.

[0064] Furthermore, this invention utilizes the joint constraint relationships between road traffic signs, combined with point-to-point fusion technology, to achieve comprehensive optimization of the calibration process. The automatic calculation and adjustment of the homography matrix can eliminate error propagation in real time, ensuring continuous stability of positioning accuracy, reducing the need for manual intervention, improving the automation and adaptability of the calibration process, and significantly enhancing the robustness and reliability of the road environment perception system, thus meeting the perception needs of complex and dynamic road traffic environments. Attached Figure Description

[0065] Figure 1 The diagram shown is a flowchart illustrating the homography camera calibration method based on road traffic signs according to the present invention.

[0066] Figure 2 The diagram shown is a schematic representation of the specific implementation process of the homography camera calibration method based on road traffic signs in an embodiment of the present invention. Detailed Implementation

[0067] The following description, in conjunction with the accompanying drawings and specific embodiments, provides further details.

[0068] The technical concept of this invention is as follows: Considering the prevalence of road traffic signs and markings that meet construction standards in road scenarios, such as zebra crossings, lane lines, and stop lines, these elements have stable shapes and standard dimensional parameters, possessing a natural advantage as calibration references. Therefore, this invention utilizes the construction standards of road traffic signs as constraints to generate calibration point pairs and solve the homography matrix, breaking through the dependence on manual intervention in the traditional calibration process. Simultaneously, it combines an adaptive adjustment strategy for calibration point pairs to improve calibration accuracy.

[0069] Example 1

[0070] This embodiment introduces a homography camera calibration method based on road traffic signs, referencing... Figure 1 It includes the following processes:

[0071] Acquire roadside image data;

[0072] Input roadside image data into a pre-trained road traffic sign recognition model to obtain the type and geometric representation data of the road traffic signs output by the road traffic sign recognition model.

[0073] Based on the type of road traffic sign, extract the set of pixel calibration points from the geometric representation data and obtain the construction standard parameters of the corresponding type of road traffic sign;

[0074] Based on the construction standard parameters, a set of local world calibration points corresponding one-to-one with the pixel calibration points is generated, resulting in multiple calibration point pairs;

[0075] Based on the multiple calibration point pairs, the homography matrix from the image coordinate system to the local world coordinate system is solved;

[0076] Based on the homography matrix, the reprojection error index is calculated for the multiple calibration point pairs to determine whether the homography matrix meets the calibration accuracy requirements. If the homography matrix does not meet the calibration accuracy requirements, the calibration point pairs are optimized, and the homography matrix is ​​resolved based on the optimized calibration point pairs.

[0077] The homography matrix that meets the calibration accuracy for reprojection error is taken as the final homography matrix.

[0078] refer to Figure 2 The specific implementation of this embodiment includes the following aspects.

[0079] I. Acquisition of Roadside Image Data

[0080] The hardware system running the road traffic sign point generation and homography calibration method of this embodiment can communicate with traffic monitoring equipment to acquire traffic environment video in real time. A series of consecutive frames can be selected as the calibration input frame set, or key frames with stable images can be selected as the calibration input frames.

[0081] Surveillance cameras used to collect roadside images can be installed on the roadside, at intersections, or on highways. The video content can be transmitted over the network and uploaded to a server for storage, providing raw data for subsequent operations such as trajectory extraction.

[0082] It should be noted that, in order to ensure the quality of video data, the system should be equipped with high-resolution cameras to ensure that details of traffic participants, such as vehicles and pedestrians, can be clearly captured.

[0083] II. Recognition of Road Traffic Signs

[0084] This invention employs a pre-trained road traffic sign recognition model, which identifies the type of road traffic sign and the geometric representation data of the road traffic sign in the image based on the input roadside image data.

[0085] The road traffic sign recognition model can utilize existing image classification neural network models, trained on a large number of labeled image samples containing various categories of road traffic signs. The labeled content includes the key points of the road traffic signs, enabling the trained model to output geometric representation data such as key points or line segment parameters for calibration, as well as the type of road traffic sign. For details, please refer to existing technologies, which will not be elaborated further.

[0086] The types of road traffic signs applicable to this embodiment include, but are not limited to, zebra crossings, lane lines, and stop lines. The geometric representation data of road traffic signs can be key point coordinates, line segment endpoints and direction parameters, polylines, boundary polygons, or pixel-level segmentation masks, etc.

[0087] III. Extraction of pixel calibration points

[0088] In order to automatically extract pixel calibration point sets from geometric representation data according to the type of road traffic signs, the method of this embodiment pre-sets corresponding key point extraction rules for various types of road traffic signs. These key points also correspond to the key points in the actual road traffic sign construction parameters, laying the foundation for obtaining calibration point pairs.

[0089] Specifically, the step of extracting a set of pixel calibration points from the geometric representation data according to the type of road traffic sign includes:

[0090] According to the preset key point extraction rules, the preset key points corresponding to the road traffic sign type are selected, and then the pixel coordinates of the preset key points are calculated based on the geometric representation data.

[0091] Considering that the recognition results of roadside images may involve multiple road traffic signs, it is also necessary to determine whether the number of key points corresponding to each road traffic sign reaches a preset lower limit m. If not, at least m key points are selected from all currently obtained key points to obtain a set of pixel calibration points. If the number of key points reaches the preset lower limit, the key points of a single road traffic sign with a number of key points are selected to obtain a set of pixel calibration points. This embodiment selects road traffic signs with a sufficient number of pixel calibration points during pixel calibration point generation. If the number of pixel calibration points that can be extracted for a single road traffic sign is insufficient, multiple road traffic signs are jointly selected, which ensures that the number of calibration points in the generated set of pixel calibration points meets the requirements for solving the homography matrix and guarantees accuracy. m is preferably 6. In some possible embodiments, the initial point selection can be based solely on the value of m.

[0092] For example, when the identified road traffic sign is a zebra crossing, the four outer corners of the zebra crossing, the endpoints of the strip boundary line, and the intersection of the strip boundary and the curb boundary can be extracted as pixel calibration points according to preset key point extraction rules. When lane lines or stop lines are identified, the endpoints of the line segments, the intersections of the line segments, or equidistant sampling points along a preset direction can be extracted as pixel calibration points. If there are insufficient available points for a single road traffic sign type or the point distribution is too concentrated, a road traffic sign switching or fusion strategy is triggered to jointly extract pixel calibration points from multiple types of road traffic signs to obtain a more sufficient and more evenly distributed point set.

[0093] In some possible embodiments, the triggering conditions for the road traffic sign switching or fusion strategy for key point extraction may also include: under the premise that the number of pixels in the pixel point set meets the minimum limit m, determining whether the pixel distribution of the currently extracted pixel calibration point set meets the preset distribution constraints, that is: after each extraction of the pixel calibration point set, the pixel calibration point set also needs to meet the distribution constraint verification before it can be used as the pixel calibration point set for constructing calibration point pairs.

[0094] Specifically, the distribution constraint verification of the pixel calibration point set includes: calculating the convex hull area of ​​multiple pixel calibration point sets, the minimum distance between different pixel calibration points, or the proportion of the convex hull area region as distribution indicators, and comparing them with the corresponding distribution indicator thresholds. If any distribution indicator is qualified, the current pixel calibration point set is maintained; if the distribution indicator is unqualified, a pixel calibration point set with qualified distribution indicators is obtained by supplementing pixel calibration points or reselecting roadside images.

[0095] In some possible embodiments, the distribution constraint verification may set only one of the above distribution indicators, or multiple indicators may be set at the same time. As long as one of the distribution indicators is satisfied, the current set of pixel calibration points can be regarded as meeting the distribution constraint verification.

[0096] The distribution index threshold corresponding to the above convex hull area This setting can be adjusted based on the pixel size of the image, such as setting it to 30% of the image pixels, provided the convex hull area of ​​the pixel calibration point set is not lower than this threshold. When the minimum distance is reached, the constraint condition is considered satisfied. (Distributed index threshold) The value is set based on the width of the image, for example, 5% of the image width, when the distance between any two pixel calibration points is not less than this threshold. When the convex hull area is calculated, it is considered to satisfy the constraint condition. Then, based on the pixel area of ​​the image... Further calculation of the convex hull area ratio The proportion of the convex hull area region is compared with the corresponding distributed index threshold. In comparison, if the area ratio of the convex hull satisfies If so, it is considered to satisfy the coverage area ratio constraint condition of the pixel calibration point set. Based on the aforementioned concept of setting the threshold for the convex hull area distribution index, it can be set to 30%. Taking a roadside image with a pixel size of 1920x1080 as an example, the convex hull area of ​​the selected pixel calibration point should be no less than 622,080 pixels, and the distance between any two pixel calibration points should be no less than 96 pixels.

[0097] The aforementioned distribution constraint verification can be performed after the pixel calibration point set is generated, because the local world calibration points are calculated based on the road traffic sign type and construction standard parameters. Therefore, once the pixel calibration points are determined, the corresponding local world calibration points can be calculated. Alternatively, it can be performed after the calibration point pairs are generated.

[0098] In some possible embodiments, the aforementioned supplementary pixel calibration points refer to: if the number of key points corresponding to a single road traffic sign is greater than a preset lower limit m, but the initial point selection only selected m key points corresponding to that road traffic sign, then other key points of the same road traffic sign can be added; if the original pixel calibration points correspond only to a single road traffic sign, then pixel calibration points of other single or multiple road traffic signs, or local world calibration points corresponding to the pixel calibration points, can be selected from the pixels extracted from the geometric representation data of the same image and added to the original pixel calibration point set. Then, the distribution constraint verification is performed on the pixel calibration point set after the points are added, until the distribution constraint verification is satisfied.

[0099] IV. Generation of Calibration Point Pairs

[0100] After obtaining the set of pixel calibration points, this embodiment acquires the corresponding construction standard parameters for the road traffic sign type, and then generates a set of local world calibration points that correspond one-to-one with the pixel calibration points based on the construction standard parameters, thus obtaining multiple calibration point pairs.

[0101] Specifically, the method in this embodiment also pre-sets construction standard parameters corresponding to various possible road traffic sign types. These parameters are set according to established industry standards. After determining the road traffic sign type, it can be obtained through mapping query. Further, based on the construction standard parameters, a set of local world calibration points corresponding one-to-one with pixel calibration points is generated, including:

[0102] Select any key point of the road traffic sign that corresponds to the pixel calibration point as the origin to construct a local world coordinate system;

[0103] Based on the construction standard parameters of road traffic signs, calculate the coordinates of the local world calibration points corresponding one-to-one with each pixel calibration point in the local world coordinate system.

[0104] In some possible embodiments, the road traffic sign type is a zebra crossing, and its standard construction parameters include a strip width of [missing information]. The strip length is Strip spacing If a pixel calibration point includes at least four pixel corner points of at least one strip, then based on the construction standard parameters of the road traffic sign, calculate the coordinates of the local world calibration point corresponding to the pixel corner point in the local world coordinate system, including:

[0105] If we take one of the pixel corner points of one of the strips as the origin of the local world coordinate system, then the coordinates of the local world calibration points corresponding to its four pixel corner points are:

[0106] ,

[0107] Among them, superscript Indicates transpose. That is, the origin of the selected local world coordinate system. (Based on pixel corner points) coordinates For example, it represents the corner point of a pixel. The X-axis and Y-axis coordinates are respectively equal to the strip length in the construction standard parameters. and strip width is 1 represents the w component of homogeneous coordinates, which is an auxiliary component that enables projective transformation of points in a two-dimensional plane.

[0108] Taking the strip containing the origin as the first strip, along the Y-axis of the local world coordinate system, the... The local world calibration point coordinates corresponding to the four pixel corner points of each strip are:

[0109] ,

[0110] Among them, the pixel corners of each strip and The Y-axis coordinate is: .

[0111] Based on the above technical solutions, for the case where the road traffic sign type is a zebra crossing, if the entire set of pixel calibration points is obtained from the zebra crossing key points, but there are some unclear stripes in the roadside image, then the key points of multiple non-adjacent stripes can be selected to form a set of pixel calibration points, and then the corresponding local world calibration point coordinates can be calculated by the above formula.

[0112] Referring to the introduction in Part III above, after the calibration point pairs are generated, distribution constraint verification of the calibration point pairs can also be performed. Specifically, this involves calculating one or more distribution indicators, such as the convex hull area of ​​multiple calibration point pairs, the minimum distance between different calibration point pairs, and the proportion of the preset area of ​​the image covered by all calibration point pairs, and comparing them with the corresponding distribution indicator thresholds. If the distribution indicators are qualified, the current set of multiple calibration points or pixel calibration points is maintained. If the distribution indicators are unqualified, multiple calibration point pairs or pixel calibration point sets with qualified distribution indicators are obtained by supplementing pixel calibration points or calibration point pairs or reselecting roadside images. The threshold settings for each distribution indicator refer to the threshold setting method in the aforementioned pixel calibration point set distribution constraint verification.

[0113] In addition to the distribution constraint verification mentioned above, a construction standard consistency verification can be performed to further achieve point-to-point screening. This can further ensure that the road traffic signs used comply with road construction standards and avoid the impact of false detections or abnormal detection results on calibration accuracy. Of course, this can also be performed when selecting the pixel calibration point set.

[0114] Taking the verification of construction standard consistency by benchmarking point pairs as an example, specifically, assuming the generated benchmarking point pairs come from multiple road traffic signs, the quality of the corresponding benchmarking point pair can be calculated separately for each road traffic sign involved. The quality assessment criteria include detection accuracy, geometric consistency, and distribution coverage, and then a comprehensive score for construction standard consistency can be calculated.

[0115] The detection confidence level is determined by the confidence value output by the road traffic sign recognition model. Geometric consistency is evaluated by the consistency between the geometry of point pairs and the construction standard parameters, and the similarity coefficient can be obtained by calculating the similarity of geometric figures. Distribution coverage is the proportion of the convex hull area of ​​the aforementioned set of calibration point pairs to the area of ​​image pixels.

[0116] The formula for calculating the comprehensive score for consistency of construction standards is as follows:

[0117]

[0118] in, For the first The overall score for the consistency of construction standards between each benchmark point. To test the confidence level, For geometric consistency, For point-to-point distribution coverage, These are preset adjustment factors that correspond to the importance of detection confidence, geometric consistency, and distribution coverage, respectively.

[0119] A threshold is set for the comprehensive score of construction standard consistency. If the comprehensive score of construction standard consistency for a certain point pair is less than the corresponding threshold, it is considered that the construction standard consistency verification is not met. The corresponding calibration point pair can be removed, and then a new calibration point pair can be added by the aforementioned point supplementation method or a new calibration point pair can be generated based on the newly acquired roadside image data.

[0120] V. Solving for the homography matrix

[0121] Based on the multiple calibration point pairs obtained above, the homography matrix from the image coordinate system to the local world coordinate system can be solved. This includes: for each calibration point pair, constructing two linear constraint equations with elements in the homography matrix as unknowns using the DLT algorithm based on the homography projection relationship; constructing a homogeneous linear equation system based on the linear constraint equations of multiple calibration point pairs; and obtaining the homography matrix through singular value decomposition based on the homogeneous linear equation system.

[0122] Specifically, solve the homography matrix under the condition of at least six calibration point pairs. This ensures that the pixel-to-world mapping relationship is satisfied. , As a scale factor, It is a homography matrix. and These are the set of local world coordinate points and the set of pixel coordinate points, respectively.

[0123] The above mapping relationship can be expanded as follows:

[0124]

[0125] in , They represent the first The local world coordinates and pixel coordinates of a calibration point pair.

[0126] The DLT algorithm is used, based on homography projection relationship, to construct two linear constraint equations for each calibration point pair, and stack all constraint equations to form a matrix. Construct a homogeneous system of linear equations, i.e., linear constraints. ,in for The elements are stacked vectors. The two-row constraints for each pair of points are represented as:

[0127]

[0128]

[0129] For matrix Perform SVD decomposition and take the column vector corresponding to the minimum singular value as... The solution, after reshaping, yields the homography matrix. .

[0130] VI. Calculation and Accuracy Assessment of Reprojection Error

[0131] In obtaining the homography matrix Then, the reprojection error is calculated between each pair of pixels in the calibration point pair set and the local world calibration points to evaluate the accuracy of the current homography matrix. If the homography matrix does not meet the calibration accuracy requirements, the calibration point pairs are optimized, and the homography matrix is ​​resolved based on the optimized calibration point pairs.

[0132] Specifically, for the first 1 pixel and its corresponding local world calibration point First, the local world calibration points are mapped back to the pixel coordinate system through the inverse transformation of the homography matrix to obtain the projected pixel points. ,satisfy: ;

[0133] in, Then the original pixel calibration points are calculated. With the return pixel The Euclidean distance between them is taken as the reprojection error of the point pair: .

[0134] To evaluate the overall calibration quality, a statistical analysis was performed on the reprojection errors of all N calibration point pairs, constructing error statistics, including the average error. Maximum error The calculation method is as follows:

[0135]

[0136] The calibration accuracy is judged based on a pre-set error threshold. When the average error... Not greater than the first error threshold And the maximum error Not greater than the second error threshold When the current homography matrix meets the calibration accuracy requirements, the homography matrix is ​​output as the final calibration result.

[0137] The aforementioned first error threshold The second error threshold is typically set to 2% of the screen width. The standard setting is 2.5% of the screen width. Taking the Sony IMX335 camera as an example of a monitoring device, its resolution is 1920x1080. Therefore, the average error of all calibration points should not exceed 39 pixels, meaning the worst calibration point reprojection error should not exceed 48 pixels.

[0138] In another alternative implementation, the reprojection error can also be calculated in the world coordinate domain, i.e., the pixel points. Obtained by mapping the homography matrix to the world coordinate system. Then calculate its relationship with real-world points. The Euclidean distance is used as an error metric to obtain the calibration error.

[0139] When the obtained error statistics do not meet the preset accuracy conditions, it indicates that there may be falsely detected point pairs or point pairs with poor geometric consistency in the current point pair set, thus affecting the stability of the homography matrix. In this case, the calibration point pair set can be further adaptively optimized through outlier removal and matrix iterative re-estimation mechanisms, specifically including:

[0140] Outlier calibration point pairs are identified based on the magnitude of the reprojection error of each calibration point pair and the average error, and these pairs are then deleted. Outlier point identification can be achieved by setting an outlier threshold. To achieve this, i.e., the reprojection error is greater than... Then it is considered an outlier;

[0141] If the number of calibration point pairs after deleting outlier calibration point pairs is not less than the preset m pairs, then the set of calibration point pairs after deleting outlier calibration point pairs will be used as the optimized set of calibration point pairs.

[0142] If the number of calibration point pairs after deleting outlier calibration point pairs is less than the preset m pairs, then a new set of calibration point pairs is obtained by re-identifying road traffic sign types and geometric representation data, extracting pixel calibration point sets, obtaining construction standard parameters, and generating local world calibration point sets based on real-time roadside image data.

[0143] The aforementioned process of "error assessment—outlier removal—matrix reestimation" can be repeated multiple times, forming an adaptive iterative optimization process. To prevent excessive iteration from increasing computational overhead, a maximum number of iterations can be set. The iteration terminates and the current homography matrix is ​​output when any of the following conditions are met: 1. The error statistic meets the preset accuracy condition; 2. The number of iterations reaches the maximum number of iterations.

[0144] When outputting the final results, calibration quality information can also be output simultaneously, such as average error, maximum error, number of effective point pairs, and spatial distribution index of point pairs, so that the system can perform quality assessment or dynamic update of the calibration results in the future.

[0145] In video input scenarios, this embodiment can also enhance the spatial distribution of point pairs by combining multi-frame information. For example, key points of road traffic signs can be extracted from multiple consecutive frames, and point pairs with different spatial locations appearing in different frames can be fused to obtain a set of point pairs with a larger coverage area, thereby further improving calibration stability.

[0146] Example 2

[0147] Based on the same inventive concept as Embodiment 1, this embodiment introduces a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the homography camera calibration method based on road traffic signs described in Embodiment 1.

[0148] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0150] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0151] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0152] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A homography camera calibration method based on road traffic signs, characterized in that, include: Acquire roadside image data; Input roadside image data into a pre-trained road traffic sign recognition model to obtain the type and geometric representation data of the road traffic signs output by the road traffic sign recognition model. Based on the type of road traffic sign, extract the set of pixel calibration points from the geometric representation data and obtain the construction standard parameters of the corresponding type of road traffic sign; Based on the construction standard parameters, a set of local world calibration points corresponding one-to-one with the pixel calibration points is generated, resulting in multiple calibration point pairs; Based on the multiple calibration point pairs, the homography matrix from the image coordinate system to the local world coordinate system is solved; Based on the homography matrix, the reprojection error index is calculated for the multiple calibration point pairs to determine whether the homography matrix meets the calibration accuracy requirements. If the homography matrix does not meet the calibration accuracy requirements, the calibration point pairs are optimized, and the homography matrix is ​​resolved based on the optimized calibration point pairs. The homography matrix that meets the calibration accuracy for reprojection error is taken as the final homography matrix.

2. The homography camera calibration method based on road traffic signs according to claim 1, characterized in that, The types of road traffic signs include at least one or more of the following: zebra crossings, lane lines, and stop lines; The geometric representation data includes at least one or more of the following: key point coordinates, line segment endpoints and direction parameters, polylines, boundary polygons, and pixel-level segmentation masks. The construction standard parameters for the road traffic signs shall include at least one or more of the following: zebra stripe width and spacing, lane line width, lane width, and stop line width.

3. The homography camera calibration method based on road traffic signs according to claim 1, characterized in that, The step of extracting a set of pixel calibration points from the geometric representation data according to the type of road traffic signs includes: The pixel coordinates of key points corresponding to each road traffic sign are calculated based on the geometric representation data. Determine whether the number of key points corresponding to each road traffic sign has reached the preset lower limit m: if not, select at least m key points from all the currently obtained key points to obtain a set of pixel calibration points; if it has reached the lower limit, select the key points of a single road traffic sign whose number of key points has reached the preset lower limit m to obtain a set of pixel calibration points.

4. The homography camera calibration method based on road traffic signs according to claim 1, characterized in that, in When extracting the set of pixel calibration points from the geometric representation data according to the type of road traffic sign: If the road traffic sign type is a zebra crossing, then extract multiple key points from the four outer corners of the zebra crossing, the endpoints of the strip boundary line, and the intersection of the strip boundary and the curb boundary as pixel calibration points to obtain the pixel calibration point set. If the road traffic sign type is a lane line or a stop line, then extract the endpoints of the lane line or stop line segments, the intersections of the line segments, and / or multiple key points from the equidistant sampling points along the preset direction as pixel calibration points to obtain the pixel calibration point set.

5. The homography camera calibration method based on road traffic signs according to claim 1, characterized in that, Also includes: Perform distribution constraint verification on the set of pixel calibration points or the multiple calibration point pairs, including: Calculate one or more distribution indicators, such as the convex hull area of ​​multiple calibration point pairs or pixel calibration point sets, and the minimum distance between different pixel calibration points or calibration point pairs, and compare them with the corresponding distribution indicator thresholds. If the distribution indicators are qualified, the current multiple calibration point pairs or pixel calibration point sets are maintained. If the distribution indicators are not qualified, multiple calibration point pairs or pixel calibration point sets with qualified distribution indicators are obtained by supplementing pixel calibration points or calibration point pairs or reselecting roadside images. The threshold for the distribution index corresponding to the convex hull area is set to be no less than 30% of the pixels in the image. The threshold for the distributed index corresponding to the minimum distance is set to be no less than 5% of the width of the image.

6. The homography camera calibration method based on road traffic signs according to claim 1, characterized in that, The step of generating a set of local world calibration points that correspond one-to-one with the pixel calibration points based on the construction standard parameters includes: Select any key point of the road traffic sign that corresponds to the pixel calibration point as the origin to construct a local world coordinate system; Based on the construction standard parameters of road traffic signs, calculate the coordinates of the local world calibration points that correspond one-to-one with the pixel calibration points in the local world coordinate system.

7. The homography camera calibration method based on road traffic signs according to claim 6, characterized in that, The road traffic sign type is a zebra crossing, and its construction standard parameters include the strip width of... The strip length is Strip spacing If a pixel calibration point includes at least four pixel corner points of at least one strip, then based on the construction standard parameters of the road traffic sign, calculate the coordinates of the local world calibration point corresponding to the pixel corner point in the local world coordinate system, including: If we take one of the pixel corner points of one of the strips as the origin of the local world coordinate system, then the coordinates of the local world calibration points corresponding to its four pixel corner points are: ， Among them, superscript Indicates transpose; Taking the strip containing the origin as the first strip, along the Y-axis of the local world coordinate system, the... The local world calibration point coordinates corresponding to the four pixel corner points of each strip are: ， Among them, the pixel corners of each strip and The Y-axis coordinate is: .

8. The homography camera calibration method based on road traffic signs according to claim 1, characterized in that, Based on the homography matrix, the reprojection error index is calculated for the plurality of calibration point pairs, including: For each calibration point pair, calculate the projected pixel in the pixel coordinate system by projecting the local world calibration point onto the pixel coordinate system through the inverse of the homography matrix; calculate the Euclidean distance between the projected pixel and the pixel calibration point as the reprojection error of the calibration point pair; or, for each calibration point pair, calculate the reprojected coordinate point in the local world coordinate system by projecting the pixel calibration point onto the local world coordinate system through the homography matrix; calculate the Euclidean distance between the reprojected coordinate point and the local world calibration point as the reprojection error of the calibration point pair. Based on the reprojection error of each calibration point pair, the reprojection error index of multiple calibration point pairs is calculated, and the reprojection error index includes the average error and the maximum error. The determination of whether the homography matrix meets the calibration accuracy requirements includes determining whether the following conditions are met: a. the average error is not greater than a first error threshold, b. the maximum error is not greater than a second error threshold; if both conditions a and b are met, then the homography matrix is ​​determined to meet the calibration accuracy requirements.

9. The homography camera calibration method based on road traffic signs according to claim 8, characterized in that, The optimization of calibration point pairs includes: determining outlier calibration point pairs based on the magnitude of the reprojection error of each calibration point pair and the average error, and deleting outlier calibration point pairs. If the number of calibration point pairs after deleting outlier calibration point pairs is not less than the preset m pairs, then the set of calibration point pairs after deleting outlier calibration point pairs will be used as the optimized set of calibration point pairs. If the number of calibration point pairs after deleting outlier calibration point pairs is less than the preset m pairs, then a new set of calibration point pairs is obtained by re-identifying road traffic sign types and geometric representation data, extracting pixel calibration point sets, obtaining construction standard parameters, and generating local world calibration point sets based on real-time roadside image data.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the homography camera calibration method based on road traffic signs as described in any one of claims 1 to 9.

Images