Calibration method and device of roadside camera and electronic equipment
By using an automated calibration vehicle with a calibration board containing preset calibration marks and a combined navigation and positioning device, the external parameters of roadside cameras are automatically calculated, solving the problems of high cost and low efficiency of manual calibration in the prior art, and realizing efficient and safe roadside camera calibration.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- MUSHROOM CHELIAN INFORMATION TECH CO LTD
- Filing Date
- 2023-06-14
- Publication Date
- 2026-06-16
Smart Images
Figure CN116664697B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of roadside equipment calibration technology, and in particular to a calibration method, device and electronic equipment for a roadside camera. Background Technology
[0002] In applications such as vehicle-road cooperation, roadside equipment plays an important role. Roadside equipment needs to perceive road traffic participants, so cameras in roadside equipment are one of the important sensors. They can obtain rich target information and calculate the target's location information through road surface information.
[0003] Since roadside equipment needs to sense traffic participants on the road and multiple roadside equipment need to communicate with each other, and the camera coordinate system and lidar coordinate system often need to be unified to the UTM (Universal Transverse Mercator Grid System) coordinate system, and also need to be used in conjunction with high-precision maps, it is necessary to calibrate the camera coordinate system to the UTM coordinate system, that is, to calibrate the external parameter transformation relationship from the roadside camera coordinate system to the UTM coordinate system.
[0004] The commonly used calibration method is to manually collect image data by holding a calibration board and simultaneously collect the RTK (Real-time kinematic) points of the calibration board. On the collected image data, calibration pixels are manually selected according to the calibration board. Then, the RTK points and the selected pixels are matched one by one to form calibration point pairs. Finally, the 2D-3D point pair solution algorithm is used to solve the extrinsic parameters to obtain the transformation matrix from the roadside camera coordinate system to the UTM coordinate system.
[0005] However, the above calibration process requires manual collection of image data and manual selection of calibration points, which is costly, inefficient, and relatively dangerous to operate manually on the road. Summary of the Invention
[0006] This application provides a calibration method, apparatus, and electronic device for roadside cameras to achieve automated calibration of roadside cameras.
[0007] The embodiments of this application adopt the following technical solutions:
[0008] In a first aspect, embodiments of this application provide a calibration method for a roadside camera, wherein the method includes:
[0009] The system acquires raw image data collected by roadside cameras and raw combined navigation and positioning data collected by calibration vehicles. The raw image data includes preset calibration marks set in the calibration plate of the calibration vehicle.
[0010] Based on the preset calibration points, the original image data, and the original integrated navigation positioning data, determine the image data and integrated navigation positioning data corresponding to the preset calibration points;
[0011] The calibration point pair data corresponding to the calibration board is determined based on the image data and combined navigation positioning data corresponding to the preset calibration points.
[0012] Based on the calibration point pair data corresponding to the calibration board, the extrinsic parameters of the roadside camera are calibrated using a preset camera extrinsic parameter calibration algorithm to obtain the extrinsic parameters of the roadside camera.
[0013] Optionally, determining the calibration point pair data corresponding to the calibration board based on the image data and integrated navigation positioning data corresponding to the preset calibration point positions includes:
[0014] The image pixel calibration coordinates of the calibration board are determined based on the image data corresponding to the preset calibration points.
[0015] The UTM calibration coordinate point corresponding to the calibration board is determined based on the combined navigation positioning data corresponding to the preset calibration point.
[0016] Based on the image pixel calibration coordinates corresponding to the calibration board and the UTM calibration coordinates corresponding to the calibration board, determine the calibration point pair data corresponding to the calibration board.
[0017] Optionally, determining the image pixel calibration coordinates of the calibration board based on the image data corresponding to the preset calibration points includes:
[0018] The preset calibration markers in the image data corresponding to the preset calibration points are detected to obtain the detection results of the preset calibration markers;
[0019] The detection results of the preset calibration mark are filtered using a preset filtering strategy to obtain the filtered detection results of the preset calibration mark.
[0020] The image pixel calibration coordinates corresponding to the calibration board are determined based on the detection results of the preset calibration identifier after filtering.
[0021] Optionally, the detection result of the preset calibration mark includes pixel regions of multiple preset calibration marks, and the filtering process of the detection result of the preset calibration mark using a preset filtering strategy to obtain the filtered detection result of the preset calibration mark includes:
[0022] Determine the area information of the pixel region of each of the preset calibration identifiers;
[0023] Based on the preset area threshold and the area information of the pixel regions of each preset calibration mark, the pixel regions of multiple preset calibration marks are filtered to obtain the first filtering result.
[0024] Optionally, the detection result of the preset calibration mark includes pixel regions of multiple preset calibration marks, and the filtering process of the detection result of the preset calibration mark using a preset filtering strategy to obtain the filtered detection result of the preset calibration mark includes:
[0025] Based on the prior rectangle information of the calibration board, the diagonal length and opposite side length of the pixel region of each of the preset calibration marks are determined;
[0026] Based on the ratio of the diagonal lengths of the pixel regions of each preset calibration mark and the corresponding first preset ratio threshold, as well as the ratio of the opposite side lengths of the pixel regions of each preset calibration mark and the corresponding second preset ratio threshold, the pixel regions of multiple preset calibration marks are filtered to obtain a second filtering result.
[0027] Optionally, the detection result of the preset calibration mark includes pixel regions of multiple preset calibration marks, and the filtering process of the detection result of the preset calibration mark using a preset filtering strategy to obtain the filtered detection result of the preset calibration mark includes:
[0028] The diagonal angle information corresponding to the pixel area of each preset calibration mark is determined based on the corner point information corresponding to the pixel area of each preset calibration mark.
[0029] Based on the preset diagonal angle threshold and the diagonal angle information corresponding to the pixel regions of each preset calibration mark, the pixel regions of multiple preset calibration marks are filtered to obtain the third filtering result.
[0030] Optionally, the calibration point pair data corresponding to the calibration board includes the image pixel calibration coordinate points and UTM calibration coordinate points corresponding to the calibration board. The step of performing extrinsic parameter calibration using a preset camera extrinsic parameter calibration algorithm based on the calibration point pair data corresponding to the calibration board to obtain the extrinsic parameters of the roadside camera includes:
[0031] Obtain the intrinsic parameters and distortion coefficients of the roadside camera, and perform distortion correction processing on the image pixel calibration coordinates corresponding to the calibration board to obtain the image pixel calibration coordinates after distortion correction.
[0032] Based on the image pixel calibration coordinates after distortion correction and the UTM calibration coordinates, the extrinsic parameters of the roadside camera are calibrated using a preset camera extrinsic parameter calibration algorithm to obtain the extrinsic parameters of the roadside camera.
[0033] Secondly, embodiments of this application also provide a calibration device for a roadside camera, wherein the device includes:
[0034] The acquisition unit is used to acquire raw image data collected by the roadside camera and raw combined navigation positioning data collected by the calibration vehicle. The raw image data includes a preset calibration mark set in the calibration plate of the calibration vehicle.
[0035] The first determining unit is used to determine the image data and integrated navigation positioning data corresponding to the preset calibration point based on the preset calibration point, the original image data, and the original integrated navigation positioning data.
[0036] The second determining unit is used to determine the calibration point pair data corresponding to the calibration board based on the image data and combined navigation positioning data corresponding to the preset calibration point location;
[0037] The calibration unit is used to perform extrinsic parameter calibration based on the calibration point pair data corresponding to the calibration board and a preset camera extrinsic parameter calibration algorithm to obtain the extrinsic parameters of the roadside camera.
[0038] Thirdly, embodiments of this application also provide an electronic device, including:
[0039] Processor; and
[0040] A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform any of the methods described above.
[0041] Fourthly, embodiments of this application also provide a computer-readable storage medium that stores one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform any of the methods described above.
[0042] The at least one technical solution adopted in this application embodiment can achieve the following beneficial effects: The roadside camera calibration method of this application embodiment first acquires the original image data collected by the roadside camera and the original integrated navigation positioning data collected by the calibration vehicle. The original image data includes a preset calibration mark set in the calibration plate of the calibration vehicle. Then, based on the preset calibration point and the original image data and original integrated navigation positioning data, the image data and integrated navigation positioning data corresponding to the preset calibration point are determined. Then, the calibration point pair data corresponding to the calibration plate is determined according to the image data and integrated navigation positioning data corresponding to the preset calibration point. Finally, based on the calibration point pair data corresponding to the calibration plate, the extrinsic parameters are calibrated using a preset camera extrinsic parameter calibration algorithm to obtain the extrinsic parameters of the roadside camera. The roadside camera calibration method of this application embodiment realizes the automated calibration of the extrinsic parameters of the roadside camera through an automated calibration vehicle equipped with a calibration plate containing preset calibration marks and integrated navigation positioning equipment, eliminating the need for manual calibration, improving the calibration efficiency of the roadside camera, and reducing the calibration cost. Attached Figure Description
[0043] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0044] Figure 1 This is a flowchart illustrating a calibration method for a roadside camera according to an embodiment of this application.
[0045] Figure 2 This is a schematic diagram of the structure of a calibration device for a roadside camera according to an embodiment of this application;
[0046] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation
[0047] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0048] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.
[0049] This application provides a calibration method for roadside cameras, such as... Figure 1The diagram shows a flowchart of a roadside camera calibration method according to an embodiment of this application. The method includes at least the following steps S110 to S140:
[0050] Step S110: Obtain the raw image data collected by the roadside camera and the raw combined navigation positioning data collected by the calibration vehicle. The raw image data includes a preset calibration mark set in the calibration plate of the calibration vehicle.
[0051] In this embodiment of the application, when calibrating the roadside camera, it is necessary to first acquire the original image data collected by the roadside camera and the original integrated navigation and positioning data collected by the calibration vehicle. The calibration vehicle is pre-installed with a calibration board and integrated navigation and positioning equipment. Here, the calibration board refers to a calibration board with a preset calibration mark pasted on its surface, so that the roadside camera can perceive and detect the preset calibration mark on the calibration board. The preset calibration mark can be any identifiable code in the prior art, such as AprilTag code, etc., and is not specifically limited here.
[0052] The aforementioned calibration plate can be installed on the roof of the calibration vehicle. Because it is mounted on the roof, a larger calibration plate can be used, allowing for the attachment of larger pre-set calibration markings, thus facilitating the subsequent extraction of the image pixel area corresponding to the calibration plate. Furthermore, considering that the roadside camera will capture all objects within its perception range (i.e., the environment), to facilitate the differentiation of the calibration plate's image pixels, this embodiment can also install the calibration plate at a certain angle to distinguish it from other road markings such as roadside signs and streetlights. The specific tilt angle can be flexibly set according to actual needs; for example, a 45-degree tilt angle can be used for ease of installation.
[0053] Step S120: Based on the preset calibration point, the original image data, and the original combined navigation positioning data, determine the image data and combined navigation positioning data corresponding to the preset calibration point.
[0054] As the calibration vehicle moves within the sensing range of the roadside camera, it will acquire a series of raw image data and a series of raw integrated navigation and positioning data. Since the data acquisition frequency of the roadside camera and the data acquisition frequency of the integrated navigation and positioning device are different, it is necessary to determine the calibration point in advance. The main function of the calibration point is to serve as the reference point for calibration. The image data and integrated navigation and positioning data corresponding to the reference point are further determined by combining the raw image data and the raw integrated navigation and positioning data.
[0055] Step S130: Determine the calibration point pair data corresponding to the calibration board based on the image data and combined navigation positioning data corresponding to the preset calibration point.
[0056] Since the image data corresponding to the calibration point is the image data of all objects within the perception range of the roadside camera, while the integrated navigation positioning data corresponding to the calibration point usually refers to the positioning data of the integrated navigation positioning device at the center position of the rear axle of the vehicle, in order to achieve data alignment, the image pixels of the calibration board can be determined from the image data corresponding to the calibration point and converted to a fixed position on the calibration board. The positioning data of the integrated navigation positioning device at the center position of the rear axle of the vehicle can also be converted to the same position on the calibration board, thereby obtaining the calibration point pair data corresponding to the calibration board.
[0057] To achieve extrinsic parameter calibration, multiple preset calibration points are required. Therefore, the image data and integrated navigation positioning data corresponding to each preset calibration point can be converted into calibration point pair data corresponding to the calibration board, thereby obtaining multiple calibration point pair data corresponding to the calibration board.
[0058] Step S140: Based on the calibration point pair data corresponding to the calibration board, perform extrinsic parameter calibration using a preset camera extrinsic parameter calibration algorithm to obtain the extrinsic parameters of the roadside camera.
[0059] Based on the multiple calibration point pairs obtained from the above steps, the extrinsic transformation matrix from the camera coordinate system to the UTM coordinate system can be solved using preset camera extrinsic calibration algorithms such as the PnP (Perspective-n-Point) algorithm and other 3D to 2D point pair motion solving algorithms.
[0060] The calibration method for roadside cameras in this application embodiment achieves automated calibration of the external parameters of roadside cameras by using an automated calibration vehicle equipped with a calibration plate containing preset calibration marks and a combined navigation and positioning device. This eliminates the need for manual calibration, improves the calibration efficiency of roadside cameras, and reduces calibration costs.
[0061] In some embodiments of this application, the acquisition of image data collected by the roadside camera and combined navigation positioning data collected by the calibration vehicle includes: when the calibration vehicle is within the perception range of the roadside camera and within a preset distance range from the roadside camera, acquiring image data and combined navigation positioning data corresponding to the leftmost lane and image data and combined navigation positioning data corresponding to the rightmost lane respectively.
[0062] In this embodiment of the application, when collecting raw image data and raw integrated navigation positioning data, it is necessary to ensure that the calibration vehicle has entered the perception range of the roadside camera, that is, the roadside camera can perceive the calibration vehicle. On the other hand, image data and integrated navigation positioning data of the calibration vehicle can be acquired when it is within a certain distance range from the roadside camera, for example, when the calibration vehicle is approximately 10m to 70m away from the roadside camera. This is because if the distance is too far or too close, it may affect the roadside camera's perception of the calibration board on the calibration vehicle. Of course, those skilled in the art can flexibly adjust the above-mentioned distance range according to actual needs, and no specific limitation is made here.
[0063] On the other hand, considering the requirements for calibration points in the extrinsic calibration of roadside cameras, multiple preset calibration points cannot be on a straight line; that is, multiple calibration points need to form a plane. For example, 7 or 8 points can be selected in the rightmost lane at certain intervals, such as 5 meters, within a range of 10 to 70 meters. In the leftmost lane, 6 or 4 points can be selected at certain intervals, such as 8 meters, within a range of 10 to 70 meters. The final number of selected calibration points needs to be no less than a certain value, such as a minimum of 4 points. By using this method of selecting calibration points, the extrinsic parameter solution will not fail due to the symmetry of the points on both sides or the fact that multiple calibration points are all on a straight line when using extrinsic parameter calibration algorithms such as PnP.
[0064] Based on the aforementioned preset calibration points, the calibration vehicle can be controlled to drive in different lanes to collect raw image data and raw combined navigation positioning data for different lanes. For example, the raw image data and raw combined navigation positioning data of the calibration vehicle in the leftmost lane and the rightmost lane can be obtained respectively.
[0065] In some embodiments of this application, determining the image data and integrated navigation positioning data corresponding to the preset calibration point based on the preset calibration point, the original image data, and the original integrated navigation positioning data includes: determining the integrated navigation positioning data corresponding to the preset calibration point based on the preset calibration point and the original integrated navigation positioning data; and determining the image data corresponding to the preset calibration point based on the timestamp of the integrated navigation positioning data corresponding to the preset calibration point and the original image data.
[0066] Because roadside cameras and integrated navigation and positioning equipment collect data at different frequencies, time synchronization processing can be performed on image data and integrated navigation and positioning data based on preset calibration points to ensure the accuracy of data processing. Here, we first determine the integrated navigation and positioning data corresponding to each preset calibration point, and then, based on the timestamp of the integrated navigation and positioning data corresponding to each preset calibration point, find the image data corresponding to that timestamp and use it as the image data corresponding to that preset calibration point. This completes the time synchronization processing between the image data and integrated navigation and positioning data for each preset calibration point.
[0067] In some embodiments of this application, determining the calibration point pair data corresponding to the calibration board based on the image data and integrated navigation positioning data corresponding to the preset calibration point includes: determining the image pixel calibration coordinate point corresponding to the calibration board based on the image data corresponding to the preset calibration point; determining the UTM calibration coordinate point corresponding to the calibration board based on the integrated navigation positioning data corresponding to the preset calibration point; and determining the calibration point pair data corresponding to the calibration board based on the image pixel calibration coordinate point and the UTM calibration coordinate point corresponding to the calibration board.
[0068] When determining the calibration point pair data corresponding to the calibration board based on the image data and integrated navigation positioning data corresponding to the preset calibration points, the image pixel calibration coordinates of the calibration board can be determined based on the image data corresponding to the preset calibration points. These image pixel calibration coordinates can be understood as the image pixel pose information of the calibration board at a certain fixed position calculated based on the image data. Similarly, the UTM calibration coordinates of the calibration board can be determined based on the integrated navigation positioning data corresponding to the preset calibration points. These UTM calibration coordinates can be understood as the UTM pose information of the calibration board at the same fixed position calculated based on the integrated navigation positioning data.
[0069] The image pixel calibration coordinates and the UTM calibration coordinates corresponding to the calibration board can constitute the calibration point pair data corresponding to the calibration board. Each preset calibration point can be processed in the above way to obtain multiple calibration point pairs data corresponding to the calibration board.
[0070] In some embodiments of this application, determining the image pixel calibration coordinates of the calibration board based on the image data corresponding to the preset calibration points includes: detecting a preset calibration identifier in the image data corresponding to the preset calibration points to obtain a detection result of the preset calibration identifier; filtering the detection result of the preset calibration identifier using a preset filtering strategy to obtain a filtered detection result of the preset calibration identifier; and determining the image pixel calibration coordinates of the calibration board based on the filtered detection result of the preset calibration identifier.
[0071] In this embodiment, when determining the image pixel calibration coordinates of the calibration board based on image data, an image detection algorithm can first be used to detect preset calibration identifiers in the image. These preset calibration identifiers can be, for example, AprilTag codes. AprilTag can be considered a visual fiducial system, with applications including AR, robotics, and camera calibration. By recognizing AprilTag codes, the camera pose can be determined. In this embodiment, a roadside camera is used to detect AprilTag codes on the calibration vehicle, serving as the basis for determining the roadside camera's extrinsic parameters. The AprilTag code detection algorithm can be flexibly determined by combining existing technologies and is not specifically limited here.
[0072] To facilitate understanding of the above embodiments, an AprilTag code detection process is provided, which mainly includes three steps:
[0073] 1) Detect various edges in the image based on gradients;
[0074] 2) Find the required quadrilateral pattern in the edge image and filter it. AprilTag detects the detected edges as much as possible. First, it removes non-straight edges, and then searches for adjacent edges on the straight edges. Finally, if a closed loop is formed, a quadrilateral is detected.
[0075] 3) Perform QR code encoding and decoding. There are three encoding methods with black border color block lengths of 8, 7, and 6 respectively. For decoding content, a dot array is generated within the detected quadrilateral to calculate the value of each color block. Then, a simple classifier is constructed based on Local Binary Patterns to classify the color blocks within the quadrilateral, encoding positive color blocks as 1 and negative color blocks as 0. This yields the QR code encoding. After obtaining the encoding, it is matched with the encoding in the known library to determine if the decoded QR code is correct. If correct, the pose of the QR code can be further calculated.
[0076] Since the above detection algorithm may have false detections, the detection result of the preset calibration mark obtained in the above steps may detect multiple pixel regions of AprilTag code on the image. The embodiments of this application can further filter the detection result of the preset calibration mark to filter out pixel regions that do not belong to AprilTag code, thereby obtaining the final pixel region of AprilTag code. Finally, the image pixel calibration coordinate point corresponding to the calibration board is determined according to the pixel region of AprilTag code. For example, the pose information of the center point of the pixel region can be calculated according to the four corner point information of the pixel region of AprilTag code, and used as the pose information of the center point of the calibration board, that is, the image pixel calibration coordinate point corresponding to the calibration board.
[0077] In some embodiments of this application, the detection result of the preset calibration mark includes pixel regions of multiple preset calibration marks. The step of filtering the detection result of the preset calibration mark using a preset filtering strategy to obtain the filtered detection result of the preset calibration mark includes: determining the area information of the pixel region of each preset calibration mark; and filtering the pixel regions of multiple preset calibration marks based on a preset area threshold and the area information of the pixel region of each preset calibration mark to obtain a first filtering result.
[0078] On the one hand, considering that the distance between the calibration vehicle and the roadside camera affects the size of the pixel area of the preset calibration mark actually detected by the roadside camera—that is, there is a "nearer, smaller farther away" problem—the approximate range of the pixel area corresponding to the preset calibration mark can be determined within a certain distance range, provided the calibration plate size is fixed and known. On the other hand, considering that cameras generally have high accuracy in detecting nearby objects, while the possibility of false detection for distant objects is relatively high.
[0079] Based on this, the embodiments of this application can set a minimum area threshold as an area constraint to filter the pixel regions of multiple preset calibration marks. If the area of the detected pixel region of the preset calibration mark is less than the area threshold, it can be directly filtered out without further judgment in other dimensions, and it is considered that it cannot belong to the pixel region corresponding to the calibration board.
[0080] In some embodiments of this application, the detection result of the preset calibration mark includes pixel regions of multiple preset calibration marks. The step of filtering the detection result of the preset calibration mark using a preset filtering strategy to obtain the filtered detection result of the preset calibration mark includes: determining the diagonal length and opposite side length of the pixel region of each preset calibration mark based on the prior rectangle information of the calibration board; filtering the pixel regions of multiple preset calibration marks according to the ratio of the diagonal lengths of the pixel regions of each preset calibration mark and the corresponding first preset ratio threshold, and the ratio of the opposite side lengths of the pixel regions of each preset calibration mark and the corresponding second preset ratio threshold, to obtain a second filtering result.
[0081] To facilitate extrinsic parameter calibration and improve calibration accuracy, the calibration plate in this embodiment can be a relatively square and regular rectangular shape, such as a square or rectangle. For each pixel region detected in the image containing multiple preset calibration markers, only one pixel region corresponds to the actual preset calibration marker on the calibration plate. This pixel region can then be used for subsequent extrinsic parameter calibration. Since the calibration plate is fixed and possesses prior rectangular information, the pixel regions of the multiple preset calibration markers can be further filtered using this prior rectangular information to obtain the pixel region corresponding to the preset calibration marker on the calibration plate.
[0082] Based on this, we can first calculate the lengths of the two diagonals and the lengths of the two pairs of opposite sides of the pixel region of each preset calibration mark based on the detection results of the pixel region of each preset calibration mark, such as the information of the four corner points. Based on the prior information of the rectangle, the lengths of the two diagonals should be equal, and the lengths of the two pairs of opposite sides should also be equal. Therefore, we can further determine whether the pixel region of the preset calibration mark meets the requirements of the prior rectangle information by calculating the ratio of the lengths of the two diagonals and the ratio of the lengths of the two pairs of opposite sides.
[0083] Specifically, the ratio of the lengths of the two diagonals can be compared with the corresponding first preset ratio threshold, and the ratio of the lengths of the two sets of opposite sides can be compared with the corresponding second preset ratio threshold. If both are less than the corresponding threshold, then the pixel region of the preset calibration mark can be considered to meet the requirements of the prior rectangle information.
[0084] Pixel regions with preset calibration marks that do not meet the requirements of the prior rectangle information can be directly filtered out, thus obtaining the second filtering result.
[0085] In some embodiments of this application, the detection result of the preset calibration mark includes pixel regions of multiple preset calibration marks. The step of filtering the detection result of the preset calibration mark using a preset filtering strategy to obtain the filtered detection result of the preset calibration mark includes: determining the diagonal angle information corresponding to the pixel region of each preset calibration mark based on the corner information corresponding to the pixel region of each preset calibration mark; and filtering the pixel regions of multiple preset calibration marks based on a preset diagonal angle threshold and the diagonal angle information corresponding to the pixel region of each preset calibration mark to obtain a third filtering result.
[0086] For a square or rectangular calibration board, the diagonal angle in the image should be within a certain range. Therefore, this embodiment can also use the diagonal angle information to filter the pixel regions of multiple preset calibration marks. First, the diagonal angle can be calculated based on the corner information corresponding to the pixel regions of each preset calibration mark. Then, the diagonal angle corresponding to the pixel regions of each preset calibration mark is compared with the preset diagonal angle threshold. The preset diagonal angle threshold can be an angle range. If the calculated diagonal angle is within the angle range, the pixel region can be considered to meet the diagonal angle requirement; otherwise, the pixel region is discarded.
[0087] It should be noted that the filtering strategies described in the above embodiments can be used in combination according to actual needs. Since the filtering strategy based on area information is implemented through a minimum area threshold, it can serve as the most basic judgment condition. That is, based on the first filtering result obtained based on area information, further judgments are made on the diagonal length, opposite side length, and the included angle of the diagonals. Of course, the above filtering strategies can also be used individually, as long as the pixel region corresponding to the calibration board can be accurately determined in the end.
[0088] In some embodiments of this application, determining the UTM calibration coordinate point corresponding to the calibration board based on the combined navigation positioning data corresponding to the preset calibration point includes: converting the combined navigation positioning data corresponding to the preset calibration point to the center point of the calibration board based on the installation position of the calibration board on the calibration vehicle, and using it as the UTM calibration coordinate point corresponding to the calibration board.
[0089] The UTM calibration coordinate points corresponding to the calibration board in this embodiment are derived from the integrated navigation positioning data collected by the integrated navigation positioning device. Since the integrated navigation positioning device is usually installed at the center of the rear axle of the vehicle, it reflects the positioning information at the center of the rear axle of the vehicle. In the aforementioned embodiment, the pixel area of the preset calibration mark has been converted to the center of the calibration board. Correspondingly, this embodiment can also convert the integrated navigation positioning data at the center of the rear axle of the vehicle to the center of the calibration board, thereby achieving data alignment.
[0090] Since the installation positions of the calibration board and the integrated navigation and positioning equipment are known, the integrated navigation and positioning data at the installation position of the integrated navigation and positioning equipment can be converted to the center position of the calibration board based on the relative positional relationship between the calibration board and the integrated navigation and positioning equipment. The converted integrated navigation and positioning data can then be transformed into the UTM coordinate system to obtain the UTM coordinates corresponding to the center point of the calibration board, i.e., the UTM calibration coordinate point.
[0091] In some embodiments of this application, the calibration point pair data corresponding to the calibration board includes image pixel calibration coordinate points and UTM calibration coordinate points corresponding to the calibration board. The step of performing extrinsic parameter calibration using a preset camera extrinsic parameter calibration algorithm based on the calibration point pair data corresponding to the calibration board to obtain the extrinsic parameters of the roadside camera includes: obtaining the intrinsic parameters and distortion coefficients of the roadside camera; performing distortion correction processing on the image pixel calibration coordinate points corresponding to the calibration board to obtain distortion-corrected image pixel calibration coordinate points; and performing extrinsic parameter calibration using a preset camera extrinsic parameter calibration algorithm based on the distortion-corrected image pixel calibration coordinate points and the UTM calibration coordinate points to obtain the extrinsic parameters of the roadside camera.
[0092] Through the aforementioned embodiments, the image pixel calibration coordinates and UTM calibration coordinates corresponding to the center point of the calibration board can be obtained. When there are multiple preset calibration points, multiple calibration point pairs corresponding to the center point of the calibration board can be obtained.
[0093] Since the images captured by the roadside camera may have a certain degree of distortion, the image pixel calibration coordinates corresponding to the calibration board can be distorted first based on the known intrinsic parameters and distortion coefficients of the roadside camera. Then, based on the calibration point pair formed by the distorted image pixel calibration coordinates and the UTM calibration coordinates, the extrinsic transformation matrix from the roadside camera coordinate system to the UTM coordinate system can be solved using 3D to 2D point pair motion solving algorithms such as PnP. There are various specific methods for solving PnP. In this embodiment, the EPnP (Efficient Perspective-n-Point) algorithm can be selected to obtain a sufficiently accurate extrinsic transformation matrix. Of course, those skilled in the art can flexibly choose other solving algorithms according to actual needs, and no specific limitation is made here.
[0094] This application embodiment also provides a calibration device 200 for a roadside camera, such as... Figure 2 The diagram shows a schematic representation of a calibration device for a roadside camera according to an embodiment of this application. The device 200 includes: an acquisition unit 210, a first determination unit 220, a second determination unit 230, and a calibration unit 240, wherein:
[0095] The acquisition unit 210 is used to acquire the raw image data collected by the roadside camera and the raw combined navigation positioning data collected by the calibration vehicle. The raw image data includes a preset calibration mark set in the calibration plate of the calibration vehicle.
[0096] The first determining unit 220 is used to determine the image data and integrated navigation positioning data corresponding to the preset calibration point based on the preset calibration point, the original image data, and the original integrated navigation positioning data.
[0097] The second determining unit 230 is used to determine the calibration point pair data corresponding to the calibration board based on the image data and combined navigation positioning data corresponding to the preset calibration point;
[0098] The calibration unit 240 is used to perform extrinsic parameter calibration based on the calibration point pair data corresponding to the calibration plate and a preset camera extrinsic parameter calibration algorithm to obtain the extrinsic parameters of the roadside camera.
[0099] In some embodiments of this application, the second determining unit 230 is specifically used to: determine the image pixel calibration coordinate point corresponding to the calibration board based on the image data corresponding to the preset calibration point; determine the UTM calibration coordinate point corresponding to the calibration board based on the combined navigation positioning data corresponding to the preset calibration point; and determine the calibration point pair data corresponding to the calibration board based on the image pixel calibration coordinate point and the UTM calibration coordinate point corresponding to the calibration board.
[0100] In some embodiments of this application, the second determining unit 230 is specifically used to: detect the preset calibration mark in the image data corresponding to the preset calibration point, and obtain the detection result of the preset calibration mark; filter the detection result of the preset calibration mark using a preset filtering strategy, and obtain the detection result of the preset calibration mark after filtering; and determine the image pixel calibration coordinate point corresponding to the calibration board based on the detection result of the preset calibration mark after filtering.
[0101] In some embodiments of this application, the detection result of the preset calibration mark includes pixel regions of multiple preset calibration marks, and the second determining unit 230 is specifically used to: determine the size information of the pixel regions of each preset calibration mark; and perform filtering processing on the pixel regions of multiple preset calibration marks based on the size information of the calibration board and the size information of the pixel regions of each preset calibration mark to obtain a first filtering processing result.
[0102] In some embodiments of this application, the detection result of the preset calibration mark includes pixel regions of multiple preset calibration marks, and the second determining unit 230 is specifically used to: determine the area information of the pixel region of each preset calibration mark; and perform filtering processing on the pixel regions of multiple preset calibration marks based on a preset area threshold and the area information of the pixel region of each preset calibration mark to obtain a second filtering processing result.
[0103] In some embodiments of this application, the detection result of the preset calibration mark includes pixel regions of multiple preset calibration marks. The second determining unit 230 is specifically used to: determine the diagonal angle information corresponding to the pixel regions of each preset calibration mark according to the corner point information corresponding to the pixel regions of each preset calibration mark; and perform filtering processing on the pixel regions of multiple preset calibration marks based on the preset diagonal angle threshold and the diagonal angle information corresponding to the pixel regions of each preset calibration mark to obtain a third filtering processing result.
[0104] In some embodiments of this application, the calibration point pair data corresponding to the calibration board includes image pixel calibration coordinate points and UTM calibration coordinate points corresponding to the calibration board. The calibration unit 240 is specifically used to: obtain the intrinsic parameters and distortion coefficients of the roadside camera; perform distortion correction processing on the image pixel calibration coordinate points corresponding to the calibration board to obtain the distortion-corrected image pixel calibration coordinate points; and perform extrinsic parameter calibration using a preset camera extrinsic parameter calibration algorithm based on the distortion-corrected image pixel calibration coordinate points and the UTM calibration coordinate points to obtain the extrinsic parameters of the roadside camera.
[0105] It is understood that the above-mentioned roadside camera calibration device can realize each step of the roadside camera calibration method provided in the foregoing embodiments. The relevant explanations of the roadside camera calibration method are applicable to the roadside camera calibration device, and will not be repeated here.
[0106] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 3 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.
[0107] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.
[0108] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.
[0109] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming the roadside camera calibration device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:
[0110] The system acquires raw image data collected by roadside cameras and raw combined navigation and positioning data collected by calibration vehicles. The raw image data includes preset calibration marks set in the calibration plate of the calibration vehicle.
[0111] Based on the preset calibration points, the original image data, and the original integrated navigation positioning data, determine the image data and integrated navigation positioning data corresponding to the preset calibration points;
[0112] The calibration point pair data corresponding to the calibration board is determined based on the image data and combined navigation positioning data corresponding to the preset calibration points.
[0113] Based on the calibration point pair data corresponding to the calibration board, the extrinsic parameters of the roadside camera are calibrated using a preset camera extrinsic parameter calibration algorithm to obtain the extrinsic parameters of the roadside camera.
[0114] The above is as stated in this application. Figure 1 The method executed by the calibration device for a roadside camera disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.
[0115] The electronic device can also perform Figure 1 The method for executing the calibration device of the roadside camera, and realizing the calibration device of the roadside camera in Figure 1 The functions of the embodiments shown are not described in detail here.
[0116] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform... Figure 1The method executed by the calibration device of the roadside camera in the illustrated embodiment is specifically used to perform:
[0117] The system acquires raw image data collected by roadside cameras and raw combined navigation and positioning data collected by calibration vehicles. The raw image data includes preset calibration marks set in the calibration plate of the calibration vehicle.
[0118] Based on the preset calibration points, the original image data, and the original integrated navigation positioning data, determine the image data and integrated navigation positioning data corresponding to the preset calibration points;
[0119] The calibration point pair data corresponding to the calibration board is determined based on the image data and combined navigation positioning data corresponding to the preset calibration points.
[0120] Based on the calibration point pair data corresponding to the calibration board, the extrinsic parameters of the roadside camera are calibrated using a preset camera extrinsic parameter calibration algorithm to obtain the extrinsic parameters of the roadside camera.
[0121] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0122] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0123] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0124] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0125] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0126] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0127] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0128] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0129] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0130] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A calibration method for a roadside camera, wherein, The method includes: The system acquires raw image data collected by roadside cameras and raw combined navigation and positioning data collected by calibration vehicles. The raw image data includes preset calibration marks set in the calibration plate of the calibration vehicle. Based on the preset calibration points, the original image data, and the original integrated navigation positioning data, determine the image data and integrated navigation positioning data corresponding to the preset calibration points; The calibration point pair data corresponding to the calibration board is determined based on the image data and combined navigation positioning data corresponding to the preset calibration points. Based on the calibration point pair data corresponding to the calibration board, the external parameters of the roadside camera are calibrated using a preset camera external parameter calibration algorithm to obtain the external parameters of the roadside camera. The step of determining the calibration point pair data corresponding to the calibration board based on the image data and integrated navigation positioning data corresponding to the preset calibration point positions includes: The image pixel calibration coordinates of the calibration board are determined based on the image data corresponding to the preset calibration points. The UTM calibration coordinate point corresponding to the calibration board is determined based on the combined navigation positioning data corresponding to the preset calibration point. Based on the image pixel calibration coordinate points corresponding to the calibration board and the UTM calibration coordinate points corresponding to the calibration board, determine the calibration point pair data corresponding to the calibration board; The image pixel calibration coordinates are image pixel pose information of the calibration board at a fixed position, calculated based on image data. The UTM calibration coordinate points are UTM pose information of the calibration board at the same fixed position, calculated based on the integrated navigation positioning data. The calibration plate is installed at a preset tilt angle on the roof of the calibration vehicle; The step of determining the image pixel calibration coordinates of the calibration board based on the image data corresponding to the preset calibration points includes: The preset calibration markers in the image data corresponding to the preset calibration points are detected to obtain the detection results of the preset calibration markers; The detection results of the preset calibration mark are filtered using a preset filtering strategy to obtain the filtered detection results of the preset calibration mark. The image pixel calibration coordinates corresponding to the calibration board are determined based on the detection results of the preset calibration mark after filtering. The detection result of the preset calibration mark includes pixel regions of multiple preset calibration marks. The filtering process of the detection result of the preset calibration mark using a preset filtering strategy to obtain the filtered detection result of the preset calibration mark includes: Based on the prior rectangle information of the calibration board, the diagonal length and opposite side length of the pixel region of each of the preset calibration marks are determined; Based on the ratio of the diagonal lengths of the pixel regions of each preset calibration mark and the corresponding first preset ratio threshold, as well as the ratio of the opposite side lengths of the pixel regions of each preset calibration mark and the corresponding second preset ratio threshold, the pixel regions of multiple preset calibration marks are filtered to obtain a second filtering result.
2. The method as described in claim 1, wherein, The detection result of the preset calibration mark includes pixel regions of multiple preset calibration marks. The filtering process of the detection result of the preset calibration mark using a preset filtering strategy to obtain the filtered detection result of the preset calibration mark includes: Determine the area information of the pixel region of each of the preset calibration identifiers; Based on the preset area threshold and the area information of the pixel regions of each preset calibration mark, the pixel regions of multiple preset calibration marks are filtered to obtain the first filtering result.
3. The method as described in claim 1, wherein, The detection result of the preset calibration mark includes pixel regions of multiple preset calibration marks. The filtering process of the detection result of the preset calibration mark using a preset filtering strategy to obtain the filtered detection result of the preset calibration mark includes: The diagonal angle information corresponding to the pixel area of each preset calibration mark is determined based on the corner point information corresponding to the pixel area of each preset calibration mark. Based on the preset diagonal angle threshold and the diagonal angle information corresponding to the pixel regions of each preset calibration mark, the pixel regions of multiple preset calibration marks are filtered to obtain the third filtering result.
4. The method as described in claim 1, wherein, The calibration point pair data corresponding to the calibration board includes the image pixel calibration coordinate points and UTM calibration coordinate points corresponding to the calibration board. The extrinsic parameter calibration is performed using a preset camera extrinsic parameter calibration algorithm based on the calibration point pair data corresponding to the calibration board to obtain the extrinsic parameters of the roadside camera, including: Obtain the intrinsic parameters and distortion coefficients of the roadside camera, and perform distortion correction processing on the image pixel calibration coordinates corresponding to the calibration board to obtain the image pixel calibration coordinates after distortion correction. Based on the image pixel calibration coordinates after distortion correction and the UTM calibration coordinates, the extrinsic parameters of the roadside camera are calibrated using a preset camera extrinsic parameter calibration algorithm to obtain the extrinsic parameters of the roadside camera.
5. A calibration device for a roadside camera, wherein, The device includes: The acquisition unit is used to acquire raw image data collected by the roadside camera and raw combined navigation positioning data collected by the calibration vehicle. The raw image data includes a preset calibration mark set in the calibration plate of the calibration vehicle. The first determining unit is used to determine the image data and integrated navigation positioning data corresponding to the preset calibration point based on the preset calibration point, the original image data, and the original integrated navigation positioning data. The second determining unit is used to determine the calibration point pair data corresponding to the calibration board based on the image data and combined navigation positioning data corresponding to the preset calibration point location; The calibration unit is used to perform extrinsic parameter calibration based on the calibration point data corresponding to the calibration board and a preset camera extrinsic parameter calibration algorithm to obtain the extrinsic parameters of the roadside camera. The second determining unit is specifically used for: The image pixel calibration coordinates of the calibration board are determined based on the image data corresponding to the preset calibration points. The UTM calibration coordinate point corresponding to the calibration board is determined based on the combined navigation positioning data corresponding to the preset calibration point. Based on the image pixel calibration coordinate points corresponding to the calibration board and the UTM calibration coordinate points corresponding to the calibration board, determine the calibration point pair data corresponding to the calibration board; The image pixel calibration coordinates are image pixel pose information of the calibration board at a fixed position, calculated based on image data. The UTM calibration coordinate points are UTM pose information of the calibration board at the same fixed position, calculated based on the integrated navigation positioning data. The second determining unit is specifically used for: The preset calibration markers in the image data corresponding to the preset calibration points are detected to obtain the detection results of the preset calibration markers; The detection results of the preset calibration mark are filtered using a preset filtering strategy to obtain the filtered detection results of the preset calibration mark. The image pixel calibration coordinates corresponding to the calibration board are determined based on the detection results of the preset calibration mark after filtering. The detection result of the preset calibration mark includes pixel regions of multiple preset calibration marks, and the second determining unit is specifically used for: Based on the prior rectangle information of the calibration board, the diagonal length and opposite side length of the pixel region of each of the preset calibration marks are determined; Based on the ratio of the diagonal lengths of the pixel regions of each preset calibration mark and the corresponding first preset ratio threshold, as well as the ratio of the opposite side lengths of the pixel regions of each preset calibration mark and the corresponding second preset ratio threshold, the pixel regions of multiple preset calibration marks are filtered to obtain a second filtering result.
6. An electronic device, comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the method of any one of claims 1 to 4.
7. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of applications, cause the electronic device to perform the method of any one of claims 1 to 4.