A roadside multi-sensor calibration method and system

Abstract

The present application relates to the technical field of sensor calibration, and more particularly to a roadside multi-sensor calibration method and system. The roadside multi-sensor calibration method provided by the present application comprises the following steps: S1, collecting original data of various sensors of a roadside system; S2, calibrating the various sensors of the roadside system with a specified laser radar, and converting the coordinate systems of the various sensors into the coordinate system of the specified laser radar; and S3, calibrating the coordinate system of the specified laser radar with a GPS coordinate system, thereby realizing calibration of the roadside multi-sensor. The roadside multi-sensor calibration method and system provided by the present application first converts the coordinate systems of all sensors into the coordinate system of a laser radar, and then calibrates the coordinate system of the laser radar with the GPS coordinate system to realize multi-sensor calibration, which is good in engineering practicability and high in robustness. Meanwhile, the present application proposes a calibration trolley, which improves the efficiency and accuracy of sensor calibration of the roadside system.

Description

Technical Field

[0001] This invention relates to the field of sensor calibration technology, and more specifically, to a method and system for calibrating multiple roadside sensors. Background Technology

[0002] Roadside perception utilizes sensors such as cameras, lidar, and millimeter-wave radar, combined with roadside edge computing, with the ultimate goal of achieving instantaneous intelligent perception of traffic participants and road conditions on that road segment.

[0003] Commonly used sensors include cameras, millimeter-wave radar, and lidar.

[0004] Cameras can detect information such as the types of traffic participants (e.g., pedestrians, vehicles, cyclists), but their performance is greatly affected by weather and light intensity.

[0005] Millimeter-wave radar can accurately detect the position, speed and other information of targets and is not affected by weather conditions, but it is prone to missing stationary targets and the detected targets have a lot of noise.

[0006] LiDAR can accurately detect the position, speed, and size of stationary and moving targets, but it is highly sensitive to the environment and is easily affected by heavy snow and dust.

[0007] By fusing data from multiple sensors, we can combine the advantages of different sensors and achieve the effect of maximizing strengths and minimizing weaknesses. Multi-sensor calibration is the foundation of data fusion, which involves unifying the camera pixel coordinate system, the LiDAR coordinate system, and the millimeter-wave radar coordinate system.

[0008] There is no existing technology that can calibrate multiple sensors for roadside systems in a way that satisfies both the requirements for calibration efficiency and accuracy. Summary of the Invention

[0009] The purpose of this invention is to provide a roadside multi-sensor calibration method and system to solve the problems of low efficiency and poor accuracy in the calibration of roadside multi-sensor systems in the prior art.

[0010] To achieve the above objectives, the present invention provides a roadside multi-sensor calibration method, comprising the following steps:

[0011] Step S1: Collect raw data from various sensors of the roadside system;

[0012] Step S2: Calibrate the various sensors of the roadside system with the designated lidar, and transform the coordinate system of the various sensors to the coordinate system of the designated lidar.

[0013] Step S3: Calibrate the specified lidar coordinate system and GPS coordinate system to achieve the calibration of multiple roadside sensors.

[0014] In one embodiment, the roadside system includes various sensors such as lidar, millimeter-wave radar, and cameras;

[0015] LiDAR includes a specified LiDAR and other LiDARs;

[0016] Step S1 further includes:

[0017] A common field of view is formed by combining lidar, millimeter-wave radar, and camera sensors.

[0018] The calibration trolley is sequentially placed at multiple calibration locations within the public field of vision;

[0019] The GPS positioning sensor on the calibration vehicle records the GPS positioning information of each calibration location;

[0020] Data is collected simultaneously by lidar, millimeter-wave radar, and camera sensors.

[0021] In one embodiment, step S2 further includes:

[0022] Step S21: The camera is calibrated with the designated LiDAR, and the camera coordinate system is placed in the designated LiDAR coordinate system;

[0023] Step S22: Calibrate the millimeter-wave radar with the designated lidar, and place the millimeter-wave radar coordinate system one under the designated lidar coordinate system;

[0024] Step S23: Calibrate other lidars with the designated lidar, and place the coordinate systems of the other lidars under the coordinate system of the designated lidar.

[0025] In one embodiment, the calibration vehicle is equipped with a GPS positioning device and a special tablet:

[0026] The GPS positioning device is used to obtain GPS coordinate information at different calibration locations;

[0027] The special flat plate has several hollowed-out circular holes;

[0028] Each perforated circular hole has a marking pattern on one side.

[0029] In one embodiment, the marking pattern is an Aruco mark, consisting of a wide black border and an inner binary matrix;

[0030] The number of the hollowed-out circular holes is at least three.

[0031] In one embodiment, step S21 further includes extracting a set of 2D-3D corresponding points between the camera and the specified lidar based on calibration data, and calculating the rotation matrix and translation matrix of the camera coordinate system relative to the specified lidar coordinate system.

[0032] Step S22 further includes: using a measuring tool to measure the relative position and angle information between the millimeter-wave radar and the specified lidar to obtain initial values; adjusting the rotation matrix and translation matrix of the millimeter-wave radar coordinate system relative to the specified lidar coordinate system according to the calibration data; and continuously optimizing the initial values ​​until the calibration accuracy meets the requirements.

[0033] Step S23 further includes extracting a set of 3D-3D corresponding points of the point cloud based on the calibration data, and calculating the rotation matrix and translation matrix between other lidars and the specified lidar coordinate system.

[0034] In one embodiment, a camera intrinsic parameter calibration step is included before step S21:

[0035] The camera was calibrated using Zhang Zhengyou's checkerboard calibration method to obtain the camera's intrinsic parameters;

[0036] After calibration, the extrinsic parameters obtained from camera calibration are used to project the corresponding three-dimensional corner points onto the image plane, and then the error between the projected result and the original image corner points is calculated.

[0037] In one embodiment, step S21 further includes:

[0038] The camera indirectly locates the center pixel coordinates of the hollow circular hole on the calibration cart by recognizing the marking pattern on the calibration cart.

[0039] The designated lidar directly locates the three-dimensional coordinates of the center point cloud of the calibrated car through point cloud segmentation, point cloud edge extraction, and point cloud center extraction algorithms.

[0040] Based on the point clouds of the camera and LiDAR, matching coordinate information is extracted, and the PnP algorithm is used for joint calibration to solve the rotation matrix and translation matrix of the camera coordinate system relative to the LiDAR coordinate system.

[0041] In one embodiment, step S22 further includes the following steps:

[0042] The relative position and angle information between the millimeter-wave radar and the specified lidar are measured using measuring tools, and the initial translation matrix and initial rotation matrix between the specified lidar and the millimeter-wave radar are obtained.

[0043] Based on the collected calibration data, the millimeter-wave radar data is projected into the specified lidar coordinate system and fused with the lidar point cloud, and a corresponding bird's-eye view is drawn for auxiliary verification.

[0044] To debug the calibration parameters of the millimeter-wave radar and lidar, the method is to check whether the millimeter-wave radar target and the lidar target overlap in the bird's-eye view. If the number of target matches reaches the set threshold, the calibration accuracy requirement is met; otherwise, the extrinsic parameters need to be readjusted.

[0045] In one embodiment, step S23 further includes the following steps:

[0046] Other lidars and the designated lidar point clouds are used to directly locate and calibrate the three-dimensional coordinate values ​​of the point clouds of the three hollow circular holes of the car through point cloud segmentation, point cloud edge extraction, and point cloud center extraction algorithms, and a set of three-dimensional coordinate corresponding points are obtained.

[0047] Based on a set of corresponding points in three-dimensional coordinates, the calibration of other lidars with the specified lidar uses the ICP algorithm to obtain the rotation and translation matrices between the coordinate systems of other lidars and the specified lidar.

[0048] In one embodiment, step S3 further includes the following steps:

[0049] Based on the collected calibration data, the coordinates of the center of the hollow circular hole near the bottom of the vehicle were uniformly extracted from the designated LiDAR point cloud. The GPS coordinate values ​​were expanded from two-dimensional coordinates to three-dimensional coordinates to obtain a set of three-dimensional coordinate corresponding points.

[0050] Based on a set of corresponding points in three-dimensional coordinates, the ICP algorithm is used to solve for the rotation and translation matrices between the lidar coordinate system and the GPS coordinate system.

[0051] To achieve the above objectives, the present invention provides a roadside multi-sensor calibration system, comprising:

[0052] Memory is used to store instructions that can be executed by the processor;

[0053] A processor for executing the instructions to implement the method as described in any of the preceding descriptions.

[0054] To achieve the above objectives, the present invention provides a computer storage medium having computer instructions stored thereon, wherein when the computer instructions are executed by a processor, the method described in any of the preceding claims is performed.

[0055] The roadside multi-sensor calibration method and system provided by this invention first unifies the coordinate systems of all sensors under a lidar coordinate system, and then calibrates the lidar coordinate system under a GPS coordinate system to achieve multi-sensor calibration. It has good engineering practicality and high robustness.

[0056] Meanwhile, this invention proposes a calibration vehicle that can meet the calibration requirements of all sensors in the roadside system, namely, the calibration requirements of cameras and lidar, millimeter-wave radar and lidar, lidar and lidar, and lidar and GPS coordinate system. The calibration vehicle improves the efficiency and accuracy of sensor calibration in the roadside system. Attached Figure Description

[0057] The above and other features, properties and advantages of the present invention will become more apparent from the following description taken in conjunction with the accompanying drawings and embodiments, in which the same reference numerals always denote the same features, wherein:

[0058] Figure 1 A flowchart of a roadside multi-sensor calibration method according to an embodiment of the present invention is disclosed;

[0059] Figure 2 A schematic diagram of a calibration vehicle according to an embodiment of the present invention is disclosed;

[0060] Figure 3 A diagram showing the relationship between the camera coordinate system and the lidar coordinate system according to an embodiment of the present invention is disclosed;

[0061] Figure 4 A flowchart of camera intrinsic parameter calibration according to an embodiment of the present invention is disclosed;

[0062] Figure 5 A block diagram illustrating the principle of a roadside multi-sensor calibration system according to an embodiment of the present invention is disclosed. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0064] Figure 1 A flowchart of a roadside multi-sensor calibration method according to an embodiment of the present invention is disclosed, such as... Figure 1 As shown, the present invention proposes a roadside multi-sensor calibration method, which includes the following steps:

[0065] Step S1: Collect raw data from various sensors of the roadside system;

[0066] Step S2: Calibrate the various sensors of the roadside system with the designated lidar, and transform the coordinate system of the various sensors to the coordinate system of the designated lidar.

[0067] Step S3: Calibrate the specified lidar coordinate system and GPS coordinate system to achieve the calibration of multiple roadside sensors.

[0068] In this embodiment, the roadside system includes various sensors, including lidar, millimeter-wave radar, and cameras.

[0069] The term "LiDAR" includes the specified LiDAR and other LiDARs. The specified LiDAR coordinate system serves as the reference coordinate system for the calibration process.

[0070] First, the coordinates of various sensors are unified under the specified lidar coordinate system, that is, the camera is calibrated with the specified lidar, the millimeter-wave radar is calibrated with the specified lidar, and other lidars are calibrated with the specified lidar; finally, the specified lidar coordinate system is calibrated with the GPS coordinate system.

[0071] To meet the calibration requirements of all sensors in the roadside system, namely, the calibration requirements between cameras and designated LiDARs, millimeter-wave radars and designated LiDARs, other LiDARs and designated LiDARs, and designated LiDARs and GPS coordinate systems, this invention proposes a special calibration tool. The calibration vehicle can meet the calibration requirements between cameras and LiDARs, millimeter-wave radars and LiDARs, other LiDARs and designated LiDARs, and designated LiDARs and GPS coordinate systems. Figure 2 As shown.

[0072] Figure 2 A schematic diagram of a calibration vehicle according to an embodiment of the present invention is shown, such as... Figure 2 The movable calibration trolley shown is equipped with a GPS positioning device and a special plate. The special plate has three hollowed-out circular holes, and a marking pattern is pasted on the left side of each hole.

[0073] It should be noted that the number of perforated round holes can be set, and the position of the marking pattern can be on one side of the round hole, not necessarily on the left side; it can be on the right side or other positions.

[0074] GPS positioning devices are used to obtain GPS coordinate information at different calibrated locations;

[0075] In this embodiment, the marking pattern pasted on the special plate is the Aruco mark, which is a binary square mark consisting of a wide black border and an inner binary matrix. The inner binary matrix determines the identity ID. This marking pattern can facilitate the camera to locate the center pixel coordinates of the three hollow circular holes on the plate.

[0076] Aruco is a small, open-source augmented reality library, currently integrated into OpenCV versions 3.0 and above. Besides augmented reality, it's also used for machine vision applications. In other embodiments, the marker pattern can be other checkerboard markers.

[0077] The hollowed-out circular holes on the special flat plate are used to facilitate the direct positioning of the three-dimensional coordinates of the center of the hollowed-out circular holes by the lidar.

[0078] Millimeter-wave radar is used for the detection and positioning of the calibration vehicle.

[0079] Combination Figure 2 The calibration vehicle shown in this embodiment, and the multi-sensor calibration method for the roadside system, includes the following steps:

[0080] Step S1: Collect raw data from various sensors of the roadside system;

[0081] A common field of view is formed by setting up a LiDAR, millimeter-wave radar, and camera sensor. The calibration vehicle is then set up at multiple calibration positions in the common field of view. The GPS positioning sensor on the calibration vehicle records the GPS positioning information of each calibration position, while the LiDAR, millimeter-wave radar, and camera sensor collect data simultaneously.

[0082] Step S2 further includes steps S21, S22 and S23, and there is no requirement for the order of these three steps.

[0083] Step S21: The camera is calibrated with the designated LiDAR, and the camera coordinate system is placed in the designated LiDAR coordinate system;

[0084] Based on the calibration data, a set of corresponding 2D-3D points of the camera and radar are extracted, and the rotation and translation matrices of the camera coordinate system relative to the specified lidar coordinate system are calculated.

[0085] Step S22: Calibrate the millimeter-wave radar with the designated lidar, and place the millimeter-wave radar coordinate system one under the designated lidar coordinate system;

[0086] The initial values ​​are obtained by measuring the relative position and angle information of the two sensors using measuring tools. Based on the calibration data, the rotation and translation matrices of the millimeter-wave radar coordinate system relative to the specified lidar coordinate system are manually adjusted to continuously optimize the initial values ​​so that the calibration accuracy meets the requirements.

[0087] Step S23: Calibrate other lidars with the designated lidar, and place the coordinate systems of the other lidars under the coordinate system of the designated lidar;

[0088] Based on the calibration data, a set of corresponding 3D-3D point cloud points are extracted, and the rotation and translation matrices between other lidars and the specified lidar coordinate system are calculated.

[0089] Step S3: Calibrate the specified lidar coordinate system and GPS coordinate system to achieve the calibration of multiple roadside sensors.

[0090] Based on the calibration data, a set of corresponding points are extracted, and the rotation and translation matrices of the lidar coordinate system relative to the GPS coordinate system are calculated.

[0091] The above steps will be described in detail below. It should be understood that, within the scope of this invention, the above-described technical features of this invention and the technical features specifically described below (such as in the embodiments) can be combined and related to each other to form preferred technical solutions.

[0092] Step S21: Calibrate the camera with the designated LiDAR.

[0093] Figure 3 A diagram showing the relationship between the camera coordinate system and the lidar coordinate system according to an embodiment of the present invention is disclosed, such as... Figure 3 As shown, point P(X) in the specified lidar coordinate system is... L ,Y L Z L Converting a point to its pixel coordinate system requires the following transformation process:

[0094]

[0095] Where Zc is the scaling factor, [μ,ν,1] T It is the coordinate value of point P in the image pixel coordinate system;

[0096] f x f y These are the number of pixels in the pixel plane at the focal length along the X-axis and the number of pixels in the pixel plane at the focal length along the Y-axis, respectively.

[0097] μ0 and ν0 are the pixel coordinates of the image center in the x and y directions, respectively;

[0098] R and T are the rotation and translation matrices used to transform the LiDAR coordinate system to the camera coordinate system, respectively.

[0099] [X L ,Y L Z L [1] is the homogeneous coordinate value of point P in the lidar coordinate system.

[0100] Furthermore, before calibrating the camera and lidar, it is necessary to complete the camera's intrinsic parameter calibration.

[0101] In this embodiment, the camera intrinsic parameter calibration step adopts the Zhang Zhengyou checkerboard calibration method.

[0102] Zhang Zhengyou's camera calibration method, proposed by Professor Zhang Zhengyou in 1998, is a single-plane checkerboard camera calibration method. It requires only a printed checkerboard grid to calibrate the camera and obtain its intrinsic parameters. The checkerboard grid is one of the most common types of markings and is frequently used for camera intrinsic parameter calibration.

[0103] Figure 4 A flowchart of camera intrinsic parameter calibration according to an embodiment of the present invention is disclosed, such as Figure 4 As shown, multiple chessboard images from different positions and angles were acquired. The Zhang Zhengyou calibration method was used to calibrate the camera's intrinsic parameters, locate the chessboard corner points, extract sub-pixel corner point information for calibration, and draw the found intrinsic corner points on the chessboard for display. After calibration, the extrinsic parameters obtained from the camera calibration were used to project the corresponding 3D corner points onto the image plane, and the results of the back projection and the calibration error of the original image corner points were calculated, while maintaining the calibration error and calibration results.

[0104] Step S21: Camera and designated LiDAR calibration.

[0105] In this embodiment, a calibration vehicle is needed to collect image data and lidar point cloud data at different calibration locations. Specifically, the following steps are also included:

[0106] The camera indirectly locates the center pixel coordinates of the three hollowed-out circular holes on the calibration vehicle by recognizing the three Aruco logo patterns on the calibration vehicle.

[0107] The designated lidar uses algorithms such as point cloud segmentation, point cloud edge extraction, and point cloud center extraction to directly locate and determine the three-dimensional coordinate values ​​of the center point cloud of the three hollowed-out circular holes of the vehicle.

[0108] The image and the LiDAR point cloud are used to extract a set of matching coordinate information for joint calibration. The joint calibration algorithm adopts the PnP (Perspective-n-Point) algorithm, which is to solve the rotation matrix and translation matrix of the camera coordinate system relative to the LiDAR coordinate system, given the camera intrinsic parameter matrix and a set of n point cloud points (3D points) in the LiDAR coordinate system and their corresponding image pixel points (2D points).

[0109] The Perspective-n-Point (PnP) problem is given by the coordinates of n points in the world coordinate system (P1, P2, ..., Pi, ..., Pn) and their corresponding coordinates in the pixel coordinate system (p1, p2, ..., pi, ..., pn), and the intrinsic parameter matrix K of the camera. The goal is to find the pose of the camera coordinate system (Oc, Xc, Yc, Zc) relative to the world coordinate system (Ow, Xw, Yw, Zw).

[0110] Step S22: Specify the calibration of the lidar and millimeter-wave radar.

[0111] First, measuring tools are used to measure the relative position and angle information of the two sensors, the millimeter-wave radar and the designated lidar, to obtain the initial translation matrix and the initial rotation matrix between the lidar and the millimeter-wave radar.

[0112] Based on the collected calibration data, the millimeter-wave radar data is then projected into the specified lidar coordinate system and fused with the lidar point cloud. A corresponding bird's-eye view is then drawn for auxiliary verification. At the same time, the external parameters of the millimeter-wave radar and lidar calibration are manually adjusted. The accuracy is judged by whether the millimeter-wave radar target and the lidar detection target in the bird's-eye view overlap and match. If most targets can be matched, the accuracy requirement is met; otherwise, it is not met, and the parameters need to be manually adjusted again.

[0113] Step S23: Calibrate other lidars with the designated lidar.

[0114] In this embodiment, the calibration of two lidars requires the use of a calibration vehicle to collect point cloud data from two lidars at different calibration locations. Specifically, the calibration also includes the following steps:

[0115] Two lidar point clouds are used to directly locate and calibrate the three-dimensional coordinate values ​​of the point clouds of the three hollow circular holes of the car through algorithms such as point cloud segmentation, point cloud edge extraction, and point cloud center extraction, and a set of three-dimensional coordinate corresponding points are obtained.

[0116] Based on a set of corresponding points in three-dimensional coordinates, the calibration of other lidars with the designated lidar uses the ICP algorithm to obtain the rotation and translation matrices between the two lidar coordinate systems.

[0117] Step S3: Specify the calibration of the lidar and GPS coordinate systems.

[0118] In this embodiment, the calibration of the specified lidar and GPS coordinate system adopts the ICP algorithm. Based on the collected calibration data, the center coordinates of the specified lidar point cloud are uniformly extracted from the circular hole near the bottom of the vehicle. The GPS coordinate values ​​are expanded from (x, y) coordinates to (x, y, z) coordinates and z is uniformly set to 0, thus obtaining a set of three-dimensional coordinate corresponding points.

[0119] Based on a set of corresponding points in three-dimensional coordinates, the ICP algorithm is used to solve for the rotation and translation matrices between the lidar and GPS coordinate systems.

[0120] The ICP algorithm (Iterative Closest Point) is an algorithm based on data registration and using the nearest point search method to solve problems based on freeform surfaces.

[0121] Although the methods described above are illustrated and depicted as a series of actions for the sake of simplicity, it should be understood and appreciated that these methods are not limited by the order of the actions, as some actions may occur in a different order and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but which may be understood by those skilled in the art, according to one or more embodiments.

[0122] Figure 5 A block diagram illustrating the principle of a roadside multi-sensor calibration system according to an embodiment of the present invention is disclosed. The roadside multi-sensor calibration system may include an internal communication bus 501, a processor 502, a read-only memory (ROM) 503, a random access memory (RAM) 504, a communication port 505, and a hard disk 507. The internal communication bus 501 enables data communication between components of the roadside multi-sensor calibration system. The processor 502 can perform judgments and issue prompts. In some embodiments, the processor 502 may consist of one or more processors.

[0123] Communication port 505 enables data transmission and communication between the roadside multi-sensor calibration system and external input / output devices. In some embodiments, the roadside multi-sensor calibration system can send and receive information and data from the network via communication port 505. In some embodiments, the roadside multi-sensor calibration system can transmit data and communicate with external input / output devices via input / output port 506 in a wired manner.

[0124] The roadside multi-sensor calibration system may also include different types of program storage units and data storage units, such as hard disk 507, read-only memory (ROM) 503, and random access memory (RAM) 504, capable of storing various data files used for computer processing and / or communication, as well as possible program instructions executed by processor 502. Processor 502 executes these instructions to implement the main parts of the method. The results processed by processor 502 are transmitted to an external output device via communication port 505 and displayed on the user interface of the output device.

[0125] For example, the implementation process file of the above-mentioned roadside multi-sensor calibration and prediction method can be a computer program, stored in hard disk 507, and can be loaded into processor 502 for execution to implement the method of this application.

[0126] When the implementation process document of the roadside multi-sensor calibration method is a computer program, it can also be stored as an article of manufacture in a computer-readable storage medium. For example, computer-readable storage media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical discs (e.g., compact discs (CDs), digital multifunction discs (DVDs)), smart cards, and flash memory devices (e.g., electrically erasable programmable read-only memory (EPROM), cards, sticks, key drives). Furthermore, the various storage media described herein can represent one or more devices and / or other machine-readable media used for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media (and / or storage media) capable of storing, containing, and / or carrying code and / or instructions and / or data.

[0127] The roadside multi-sensor calibration method and system provided by this invention first unifies the coordinate systems of all sensors under a lidar coordinate system, and then calibrates the lidar coordinate system under a GPS coordinate system to achieve multi-sensor calibration. It has good engineering practicality and high robustness.

[0128] Meanwhile, this invention proposes a calibration vehicle that can meet the calibration requirements of all sensors in the roadside system, namely, the calibration requirements of cameras and lidar, millimeter-wave radar and lidar, lidar and lidar, and lidar and GPS coordinate system. The calibration vehicle improves the efficiency and accuracy of sensor calibration in the roadside system.

[0129] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0130] Those skilled in the art will understand that information, signals, and data can be represented using any of a variety of different techniques and arts. For example, the data, instructions, commands, information, signals, bits, symbols, and chips described throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0131] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.

[0132] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or performed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The general-purpose processor may be a microprocessor, but in alternatives, it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.

[0133] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read and write information to / from the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and storage medium may reside as discrete components in the user terminal.

[0134] In one or more exemplary embodiments, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software as a computer program product, the functionality may be stored or transmitted as one or more instructions or code on or through a computer-readable medium. A computer-readable medium includes both computer storage media and communication media, encompassing any medium that facilitates the transfer of a computer program from one location to another. A storage medium may be any available medium accessible to a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible to a computer. Any connection is also legitimately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. As used in this article, disk and disc include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs. Disks typically reproduce data magnetically, while discs reproduce data optically using lasers. Combinations of these should also be included within the scope of computer-readable media.

[0135] The above embodiments are provided for those skilled in the art to implement or use the present invention. Those skilled in the art can make various modifications or changes to the above embodiments without departing from the inventive concept of the present invention. Therefore, the protection scope of the present invention is not limited to the above embodiments, but should be the maximum scope that conforms to the innovative features mentioned in the claims.

Claims

1. A method for calibrating multiple roadside sensors, characterized in that, Includes the following steps: Step S1: Collect raw data from various sensors of the roadside system; Step S2: Calibrate the various sensors of the roadside system with the designated lidar, and transform the coordinate system of the various sensors to the coordinate system of the designated lidar. Step S3: Calibrate the specified lidar coordinate system and GPS coordinate system to achieve the calibration of multiple roadside sensors; The roadside system includes various sensors such as lidar, millimeter-wave radar, and cameras. LiDAR includes a specified LiDAR and other LiDARs; Step S1 further includes: A common field of view is formed by combining lidar, millimeter-wave radar, and camera sensors. The calibration trolley is sequentially placed at multiple calibration locations within the public field of vision; The GPS positioning sensor on the calibration vehicle records the GPS positioning information of each calibration location; Data is collected simultaneously by lidar, millimeter-wave radar, and camera sensors; The calibration trolley is equipped with a GPS positioning device and a special tablet: The GPS positioning device is used to obtain GPS coordinate information at different calibration locations; The special flat plate has several hollowed-out circular holes; Each perforated circular hole has a marking pattern on one side.

2. The roadside multi-sensor calibration method according to claim 1, characterized in that, Step S2 further includes: Step S21: The camera is calibrated with the designated LiDAR, and the camera coordinate system is placed in the designated LiDAR coordinate system; Step S22: Calibrate the millimeter-wave radar with the designated lidar, and place the millimeter-wave radar coordinate system one under the designated lidar coordinate system; Step S23: Calibrate other lidars with the designated lidar, and place the coordinate systems of the other lidars under the coordinate system of the designated lidar.

3. The roadside multi-sensor calibration method according to claim 1, characterized in that, The marking pattern is an Aruco mark, consisting of a wide black border and an inner binary matrix; The number of the hollowed-out circular holes is at least three.

4. The roadside multi-sensor calibration method according to claim 2, characterized in that: Step S21 further includes extracting a set of 2D-3D corresponding points between the camera and the specified lidar based on the calibration data, and calculating the rotation matrix and translation matrix of the camera coordinate system relative to the specified lidar coordinate system. Step S22 further includes: using a measuring tool to measure the relative position and angle information between the millimeter-wave radar and the specified lidar to obtain initial values; adjusting the rotation matrix and translation matrix of the millimeter-wave radar coordinate system relative to the specified lidar coordinate system according to the calibration data; and continuously optimizing the initial values ​​until the calibration accuracy meets the requirements. Step S23 further includes extracting a set of 3D-3D corresponding points of the point cloud based on the calibration data, and calculating the rotation matrix and translation matrix between other lidars and the specified lidar coordinate system.

5. The roadside multi-sensor calibration method according to claim 4, characterized in that, The step S21 is preceded by a camera intrinsic parameter calibration step: The camera was calibrated using Zhang Zhengyou's checkerboard calibration method to obtain the camera's intrinsic parameters; After calibration, the extrinsic parameters obtained from camera calibration are used to project the corresponding three-dimensional corner points onto the image plane, and then the error between the projected result and the original image corner points is calculated.

6. The roadside multi-sensor calibration method according to claim 4, characterized in that, Step S21 further includes: The camera indirectly locates the center pixel coordinates of the hollow circular hole on the calibration cart by recognizing the marking pattern on the calibration cart. The designated lidar directly locates the three-dimensional coordinates of the center point cloud of the calibrated car through point cloud segmentation, point cloud edge extraction, and point cloud center extraction algorithms. Based on the point clouds of the camera and LiDAR, matching coordinate information is extracted, and the PnP algorithm is used for joint calibration to solve the rotation matrix and translation matrix of the camera coordinate system relative to the LiDAR coordinate system.

7. The roadside multi-sensor calibration method according to claim 4, characterized in that, Step S22 further includes the following steps: The relative position and angle information between the millimeter-wave radar and the specified lidar are measured using measuring tools, and the initial translation matrix and initial rotation matrix between the specified lidar and the millimeter-wave radar are obtained. Based on the collected calibration data, the millimeter-wave radar data is projected into the specified lidar coordinate system and fused with the lidar point cloud, and a corresponding bird's-eye view is drawn for auxiliary verification. To debug the calibration parameters of the millimeter-wave radar and lidar, the method is to check whether the millimeter-wave radar target and the lidar target overlap in the bird's-eye view. If the number of target matches reaches the set threshold, the calibration accuracy requirement is met; otherwise, the extrinsic parameters need to be readjusted.

8. The roadside multi-sensor calibration method according to claim 4, characterized in that, Step S23 further includes the following steps: Other lidars and the designated lidar point clouds are used to directly locate and calibrate the three-dimensional coordinate values ​​of the point clouds of the three hollow circular holes of the car through point cloud segmentation, point cloud edge extraction, and point cloud center extraction algorithms, and a set of three-dimensional coordinate corresponding points are obtained. Based on a set of corresponding points in three-dimensional coordinates, the calibration of other lidars with the specified lidar uses the ICP algorithm to obtain the rotation and translation matrices between the coordinate systems of other lidars and the specified lidar.

9. The roadside multi-sensor calibration method according to claim 4, characterized in that, Step S3 further includes the following steps: Based on the collected calibration data, the coordinates of the center of the hollow circular hole near the bottom of the vehicle were uniformly extracted from the designated LiDAR point cloud. The GPS coordinate values ​​were expanded from two-dimensional coordinates to three-dimensional coordinates to obtain a set of three-dimensional coordinate corresponding points. Based on a set of corresponding points in three-dimensional coordinates, the ICP algorithm is used to solve for the rotation and translation matrices between the lidar coordinate system and the GPS coordinate system.

10. A roadside multi-sensor calibration system, comprising: Memory is used to store instructions that can be executed by the processor; A processor for executing the instructions to implement the method as described in any one of claims 1-9.

11. A computer storage medium having stored thereon computer instructions, wherein when the computer instructions are executed by a processor, the method as described in any one of claims 1-9 is performed.

Images