System and method for remote contactless calibration of roadside sensors

Abstract

Embodiments of the present disclosure may provide a system and method for calibrating extrinsic parameters of an autonomous driving roadside sensor. During operation, the portable lidar device may be brought to a sensor installation location containing one or more roadside sensors to be calibrated, and the portable lidar device may scan an outer surface of the roadside sensors to be calibrated from different angles to generate a stream of frames. The system can spatially align frame streams based on a local reference coordinate system, overlay the aligned frames, and segment a point cloud associated with a roadside sensor to be calibrated. The system can determine extrinsic parameters of the roadside sensor to be calibrated relative to the local reference coordinate system according to the segmented point cloud and convert the extrinsic parameters from the local reference coordinate system to a road-based coordinate system.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to autonomous driving. More specifically, embodiments of the present disclosure relate to efficient non-contact calibration of roadside sensors. Background Art

[0002] The success of autonomous driving depends to a large extent on the vehicle's ability to collect and process a large amount of traffic states, including the vehicle's own state (such as direction and speed) and the states of other objects around the vehicle. The dynamic nature of the spatio-temporal characteristics of traffic states and the limited vehicle perception range often hinder the vehicle's ability to collect traffic scene state information. In addition, on-vehicle sensors may have limitations (e.g., being blocked).

[0003] Roadside sensors have been used to extend the data acquisition capabilities of on-vehicle sensors. For example, sensors installed on roadside traffic facilities (such as streetlight poles or traffic signals) usually have an unobstructed view of the road and can collect traffic states more effectively. Various multi-sensor fusion strategies have been proposed to combine the data collected by roadside sensors with the data collected by on-vehicle sensors. Since the original sensor data is usually based on the respective coordinate systems of each sensor, combining or fusing data from different sensors requires converting these original sensor data into a unified coordinate system. In addition, the processing and analysis of sensor data for autonomous driving purposes are usually road-centered. Therefore, it is crucial to convert all original sensor data from their respective sensor-based coordinate systems into a road-based coordinate system. Traditional calibration methods for establishing the mapping relationship between the sensor-based coordinate system and the road-based coordinate system are often insufficient for autonomous driving applications with a large number of roadside sensors deployed. Summary of the Invention

[0004] Embodiments of the present disclosure may provide a system and method for calibrating the external parameters of roadside sensors in autonomous driving. During operation, a portable light-detection-and-ranging (lidar) unit can be brought to a sensor installation location containing one or more roadside sensors to be calibrated, and the portable lidar device can scan the outer surface of the roadside sensors to be calibrated from different angles to generate a frame stream. The system can spatially align the frame stream according to a local reference coordinate system, stack the aligned frames, and segment out the point cloud related to the roadside sensors to be calibrated from the stacked frames. The system can determine the external parameters of the roadside sensors to be calibrated relative to the local reference coordinate system based on the segmented point cloud, and then convert these external parameters from the local reference coordinate system to a road-based coordinate system.

[0005] In a variation of this embodiment, aligning the frame stream may include applying the Iterative Closest Point (ICP) algorithm.

[0006] In a variation of this embodiment, the portable lidar device may further include one or more position sensors. Aligning the frame stream may include determining the instantaneous pose of the portable lidar device associated with each frame in the frame stream based on the measurements of the position sensors.

[0007] In another variation, the local reference coordinate system may be determined based on the instantaneous pose of the portable lidar device associated with the first frame in the frame stream.

[0008] In another variation, the position sensors include one or more of the following: a Global Positioning System (GPS) sensor; an Inertial Measurement Unit (IMU); and a rotary encoder.

[0009] In a variation of this embodiment, the system may determine the transformation matrix between the local reference coordinate system and the road-based coordinate system.

[0010] In another variation, converting the external parameters from the local reference coordinate system to the road-based coordinate system includes multiplying the external parameters by the transformation matrix.

[0011] In a variation of this embodiment, the portable lidar device may be configured to scan at least two reference objects with unique features in each frame, and spatially aligning the frame stream may include aligning the reference objects.

[0012] In a variation of this embodiment, determining the external parameters of the roadside sensor to be calibrated may include comparing the segmented point cloud with the Computer-Aided Design (CAD) model of the roadside sensor or the point cloud of the roadside sensor obtained by scanning the roadside sensor before installation.

[0013] In a variation of this embodiment, spatially aligning the frame stream may include removing transient objects from each frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Shows an exemplary multi-sensor data acquisition system according to an embodiment of the present application;

[0015] Figure 2 Shows an example scenario for calibrating a roadside sensor according to an embodiment of the present application;

[0016] Figure 3 Illustrates an example scenario for calibrating multiple roadside sensors according to an embodiment of the present application;

[0017] Figure 4 Illustrates an exemplary block diagram of a sensor calibration system according to an embodiment of the present application;

[0018] Figure 5 Presents a flowchart showing an exemplary sensor calibration process according to an embodiment of the present application;

[0019] Figure 6 Illustrates an example computer system that facilitates sensor calibration operations according to an embodiment of the present application.

[0020] In the figures, the same reference numerals represent the same components. Detailed Description

[0021] The following description is intended to enable those skilled in the art to make and use the disclosed embodiments and is provided in the context of one or more specific applications and their requirements. Various modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the scope of the disclosed content. Therefore, the present invention or various aspects thereof are not intended to be limited to the embodiments shown, but should be accorded the widest scope consistent with the disclosed content.

[0022] Overview

[0023] Embodiments of the present disclosure provide a system and method for determining the attitude (i.e., external parameters) of roadside sensors using non-contact technology. To facilitate the efficient and accurate calibration of roadside sensors, a sensor calibration system may include a portable lidar device (referred to simply as lidar) capable of scanning the exterior of roadside sensors to obtain multiple lidar frames. The multiple lidar frames can be aligned and superimposed in the spatial domain to obtain a high-precision three-dimensional (3D) point cloud for each roadside sensor to be calibrated. Then, the attitude (or external parameters) of the roadside sensor in the lidar-based coordinate system can be determined based on the known dimensions of the roadside sensor. Given the attitude of the lidar in the road-based coordinate system, the transformation matrix between the lidar-based coordinate system and the road-based coordinate system can be derived. Thus, the previously obtained external parameters of the roadside sensor can be transformed from the lidar-based coordinate system and the road-based coordinate system, which helps in the calibration of the roadside sensor. Once the external parameters of the roadside sensor are calibrated, the raw sensor data collected by the roadside sensor can be transformed into the road-based coordinate system.

[0024] Roadside-Sensor calibration

[0025] Figure 1 An exemplary multi-sensor data acquisition system according to an embodiment of the present application is shown. In Figure 1 it, the multi-sensor data acquisition system 100 may include a vehicle 102 traveling on a road and roadside sensors 104 and 106 mounted on a permanent traffic structure 108.

[0026] The vehicle 102 may be equipped with various types of sensors, such as visible light cameras, infrared cameras, radars, lidar devices, global positioning system (GPS) sensors, inertial measurement unit (IMU) modules, sound sensors, etc. The roadside sensors 104 and 106 may include similar types of sensors.

[0027] Each sensor, whether it is an on-vehicle sensor or a roadside sensor, may collect data about its surrounding environment (e.g., traffic state), and the surrounding environment may include another vehicle 110 and a pedestrian 112. The raw sensor data itself is expressed in a coordinate system centered on the corresponding sensor. For example, the data collected by sensors (e.g., cameras or lidar devices) on the vehicle 102 may be represented in the vehicle coordinate system, and the origin of the coordinate system may be fixed at a certain fixed point on the vehicle 102. On the other hand, the data collected by the roadside sensor 104 or the roadside sensor 106 may be represented in the corresponding roadside sensor-based coordinate systems with the sensor 104 or the sensor 106 as the origin, respectively.

[0028] To combine data from different sensors, it is necessary to represent the raw sensor data obtained by each sensor in the same coordinate system. Because various tasks involved in autonomous driving (e.g., path planning, positioning, road mapping, obstacle detection, etc.) often need to analyze data in a road-based coordinate system (e.g., Figure 1 the coordinate system 114 shown in it), it makes sense to convert the raw sensor data from different sensors from their respective sensor-based coordinate systems to the road-based coordinate system. In Figure 1 the example shown, the origin of the road-based coordinate system 114 may be set at the bottom of the traffic structure 108, the x-axis is along the road direction, the y-axis is perpendicular to the road, and the z-axis is the vertical axis. There may also be other ways to define the road-based coordinate system.

[0029] Converting the raw data obtained by the roadside sensor from the sensor-based coordinate system to the road-based coordinate system requires knowledge of the external parameters of the roadside sensor that define the sensor pose (position and orientation). The process of determining these external parameters is called calibration.

[0030] Some existing calibration techniques typically rely on using instruments such as real-time kinematic (RTK) GPS to measure the installation location and / or orientation of roadside sensors (such as cameras, lidars, or millimeter-wave radars). These methods require expensive survey-grade equipment and can only determine the position / orientation of one sensor at a time. The measurement process can be very time-consuming and sometimes requires temporary road closures. Some calibration methods also require the use of aerial work platforms, which means additional measures must be taken to ensure the safety of the surveyors. Some other existing calibration methods may rely on additional data (such as high-resolution maps) and auxiliary devices (such as calibration plates, calibration target vehicles with RTK GPS, drones capable of taking images, etc.) for joint calibration, but these methods are not applicable to non-camera sensors (such as thermal imagers, radars, lidars, etc.) and cameras containing non-public data. In addition, existing methods are generally not applicable to the calibration of a large number of sensors.

[0031] To overcome the drawbacks of existing calibration methods, some embodiments of the present application provide a system and method that can calibrate roadside sensors in an efficient and non-contact manner, making it an ideal choice for autonomous driving applications.

[0032] Figure 2 An example scenario for calibrating a roadside sensor according to an embodiment of the present application is shown. In Figure 2 , a user 202 carrying a portable lidar device 204 is walking towards a roadside sensor 206 installed on a permanent traffic structure (such as a streetlight pole) 208. The user 202 can use the portable lidar device 204 to continuously scan the scene including the roadside sensor 206 and its surrounding environment. For example, the user 202 can aim the portable lidar device 204 at the roadside sensor 206 and operate it in such a way that the laser beam of the portable lidar device 204 can repeatedly sweep across the exterior (such as the housing) of the road sensor 206 (such as from left to right or from top to bottom). To improve the calibration accuracy, it should be ensured that the laser beam can scan as many surfaces as possible to obtain a high-density three-dimensional point cloud. In one example, the portable lidar device 204 can generate a lidar frame stream containing at least ten lidar frames, each frame containing a point cloud corresponding to the roadside sensor 206.

[0033] In some embodiments, to obtain a high-density point cloud of the roadside sensor 206, multiple lidar frames can be stacked together, with each frame containing the point cloud of the roadside sensor 206. However, when the portable lidar device 204 scans the surface of the roadside sensor 206, its pose changes, which means that the lidar frames are generated based on different coordinate systems, and the point clouds in different lidar frames are not spatially aligned. More specifically, each lidar frame is generated based on an instantaneous lidar-based coordinate system (i.e., the distance vectors are defined relative to the instantaneous lidar-based coordinate system). Before the lidar frames can be stacked, they should be aligned in the spatial domain. In some embodiments, aligning the lidar frames in the spatial domain can include transforming all lidar frames to the same coordinate system.

[0034] Various techniques can be used to align lidar frames defined in different coordinate systems. In some embodiments, the iterative closest point (ICP) algorithm can be used to align lidar frames, including the point clouds in the lidar frames. To ensure alignment accuracy, in some embodiments, each lidar frame acquired by the portable lidar device 204 should contain multiple reference objects. Each reference object can have distinct and easily recognizable features. Large cylindrical objects (such as utility poles or streetlight poles) are good examples of reference objects. In addition, each lidar frame can also include the ground as a reference plane. For example, two lidar frames can be aligned by aligning the reference objects and the ground in one frame with the corresponding reference objects and the ground in another frame.

[0035] In a variation, a local reference coordinate system can be determined, and all lidar frames acquired by the portable lidar device 204 can be transformed into this local reference coordinate system. The local reference coordinate system can be arbitrarily determined. In one example, the lidar-based coordinate system corresponding to the first frame in the frame stream can be selected as the local reference coordinate system. In other words, the local reference coordinate system can be determined according to the instantaneous pose of the portable lidar device 204 when it acquires the first lidar frame. For example, the origin of the local reference coordinate system can be the position of a preset point on the portable lidar device 204 at this instantaneous moment, and the axes of the local reference coordinate system can be the directions of the optical axes of the portable lidar device 204 at this instantaneous moment. In some embodiments, the portable lidar device 204 can include multiple built-in position sensors, including but not limited to real-time kinematic (RTK) GPS, inertial measurement unit (IMU), rotary encoders, etc. In these scenarios, the lidar frames can be aligned according to the sensor data from various position sensors. For example, the instantaneous pose of the portable lidar device 204 relative to the local reference coordinate system at a given instantaneous moment can be determined based on the position sensor data, and then the corresponding frame acquired at this given instantaneous moment can be transformed into the local reference coordinate system according to this instantaneous pose.

[0036] After the lidar frames or point clouds are transformed into the local reference coordinate system, they can be superimposed on each other to create a high-density point cloud of the scene. When lidar frames acquired from different angles are superimposed, the number of data points on a specific surface of the roadside sensor 206 will increase.

[0037] A segmentation operation can be performed on the aligned and superimposed lidar frames to distinguish the point cloud of the roadside sensor 206 from its surrounding environment. Ideally, the housing of the roadside sensor 206 can be made of a highly reflective material such as metal or glass. More specifically, the reflectivity of the housing of the roadside sensor 206 corresponding to the lidar wavelength can be higher than that of its surrounding objects (such as the street lamp pole 208). In this way, it will be relatively easy to segment the point cloud of the roadside sensor 206 from its surrounding environment. The segmentation task can be completed using various techniques. In some embodiments, a deep learning neural network can be trained to perform the segmentation task.

[0038] In most cases, the size and shape of the roadside sensor 206 are known. Therefore, the pose of the roadside sensor 206 in the local reference coordinate system can be determined by comparing the segmented 3D point cloud of the roadside sensor 206 with its known size and shape. For example, the precise pose of the roadside sensor 206 in the local reference coordinate system can be determined by matching the points in the 3D point cloud with the surface points of the known computer-aided design (CAD) model of the roadside sensor 206. In another example, the roadside sensor 206 can be 3D scanned before installation to generate a known 3D model (such as a point cloud) of the roadside sensor 206.

[0039] As previously mentioned, the local reference coordinate system can be a lidar-based coordinate system corresponding to the pose of the portable lidar device 204 when acquiring a certain frame (such as the first frame). In some embodiments, the portable lidar device 204 can be associated with multiple position sensors, including but not limited to real-time kinematic differential global positioning system (RTK GPS), inertial measurement unit (IMU), rotary encoder, etc. For example, one or more position sensors can be integrated into the portable lidar device 204, or they can be located within the same housing as the portable lidar device 204 and the relative position and / or orientation between them are pre-determined. The pose of the portable lidar device 204 in the road-based coordinate system at a specific instant can be determined based on the data from various position sensors at that instant. For example, at a given specific instant (such as the instant of acquiring the first frame), the RTK GPS in the portable lidar device 204 can provide precise geographical location information, while the IMU or rotary encoder can provide information about the orientation of the portable lidar device 204.

[0040] Once the pose of the portable lidar device 204 is determined, the transformation matrix between the lidar-based coordinate system and the road-based coordinate system can be calculated to transform the pose of the roadside sensor 206 from the local reference coordinate system to the road-based coordinate system. Therefore, the external parameters of the roadside sensor 206 relative to the road-based coordinate system can be obtained, and the calibration of the roadside sensor 206 is completed.

[0041] In Figure 2In the example shown, the portable lidar device 204 is held by the user 202. The user 202 can walk to the intersection where one or more roadside sensors are located and use the portable lidar device 204 to scan these roadside sensors and their surrounding environment. In actual operation, the lidar device can also be installed on a vehicle (e.g., a specially designed data collection vehicle). When the vehicle drives through the intersection, the lidar device can scan the roadside sensors and their surrounding environment. When there are multiple roadside sensors to be calibrated at the intersection, it is preferable to have the lidar scan (which can generate a continuous stream of lidar frames) cover the outer surfaces of all the sensors. Additionally, at any given instant, the lidar frame should include at least two reference objects (e.g., a pole and a cantilever attached to the pole) and the ground, thereby allowing for efficient and accurate alignment and stitching of the point clouds of different lidar frames. The pose of each individual roadside sensor can then be determined based on the corresponding point cloud. For example, after all the lidar frames in the frame stream are aligned, the point cloud of each roadside sensor can be segmented to distinguish it from the background. Then, the point cloud of a specific sensor can be compared with its known shape and dimensions (or a known computer-aided design (CAD) model or a pre-scanned model) to obtain the pose of the sensor relative to the local reference coordinate system.

[0042] Figure 3 An example scenario for calibrating multiple roadside sensors in accordance with an embodiment of the present application is shown. In Figure 3 this figure, the user 302 carrying the portable lidar device 304 is approaching the sensor installation location (e.g., an intersection) where the roadside sensor 306 and the roadside sensor 308 are installed on the cantilever 310 attached to the lamp post 312. The roadside sensor 306 and the roadside sensor 308 can be any type of sensor capable of collecting traffic data, including but not limited to visible light cameras, infrared cameras, radars, lidar devices, global positioning system (GPS) sensors, inertial measurement unit (IMU) modules, sound sensors, etc.

[0043] At the intersection, the user 302 can aim the portable lidar device 304 at the roadside sensor 306 and the roadside sensor 308 to obtain a lidar frame stream. To calibrate all the sensors at the intersection, the portable lidar device 304 should scan the outer surfaces of all the sensors so that all the sensors are included in the lidar frame stream. More specifically, when performing the scan, the user 302 should ensure that the laser beam inside the portable lidar device 304 can illuminate as much of the outer surface of each sensor as possible. Additionally, to facilitate precise frame alignment, each lidar scan (or frame) should include at least two reference objects and the ground. In Figure 3In the example shown, each lidar frame should include the light pole 312 and the cantilever 310.

[0044] Figure 3 Also shown are the various coordinate systems involved in multi-sensor data acquisition, including the road-based coordinate system 314, the lidar-based coordinate system 316, the first roadside sensor-based coordinate system 318 (based on the roadside sensor 306), and the second roadside sensor-based coordinate system 320 (based on the roadside sensor 308). In this example, the origin of the road-based coordinate system 314 can be set at the bottom of the light pole 312. In one instance, the x-y plane of the road-based coordinate system 314 can be the road surface, with the x-axis along the road direction. The z-axis points upward from the ground. The origin of the lidar-based coordinate system 316 can be anchored at a fixed point (e.g., the optical center) on the portable lidar device 304, and the z-axis can be along the optical axis of the portable lidar device 304. Similarly, the origin of the roadside sensor-based coordinate system (e.g., the first coordinate system 318 or the second coordinate system 320) can be anchored at a fixed point on the corresponding roadside sensor, and the z-axis can be along the optical axis of the corresponding roadside sensor.

[0045] The road-based coordinate system 314 can be an absolute, time-invariant coordinate system, and all sensor data should be transformed to the road-based coordinate system 314 so that they can be combined or fused. Since the attitude of each roadside sensor remains unchanged relative to the road-based coordinate system 314, the roadside sensor-based coordinate system is also considered a time-invariant coordinate system. The data collected by each roadside sensor (e.g., distance vectors) are represented according to the corresponding coordinate system. For example, the data collected by the roadside sensor 306 can be represented using values in the first coordinate system 318, while the data collected by the roadside sensor 308 can be represented using values in the second coordinate system 320.

[0046] On the other hand, relative to the road-based coordinate system 314, the lidar-based coordinate system 316 may be dynamic because the attitude of the portable lidar device 304 changes as it scans the roadside sensor 306 and the roadside sensor 308 from different angles. In some embodiments, a local reference coordinate system can be defined according to the lidar-based coordinate system 316. In one example, the local reference coordinate system can be defined based on the initial attitude of the portable lidar device 304 relative to the road-based coordinate system 314 (or its attitude when it acquires the first frame). Since the local reference coordinate system is defined based on the attitude of the portable lidar device 304 at a specific point in time, it is also time-invariant relative to the road-based coordinate system 314. For simplicity of illustration, Figure 3The lidar-based coordinate system 316 therein can also represent a local reference coordinate system.

[0047] Converting the data collected by the roadside sensor to the road-based coordinate system 314 generally requires knowledge of the extrinsic parameters (or pose) of the roadside sensor relative to the road-based coordinate system 314. However, such information is not easily obtained. In some embodiments, the extrinsic parameters of the roadside sensor relative to the local reference coordinate system 316 can be determined first. To achieve this, the lidar frame containing the roadside sensor acquired by the portable lidar device 304 can be aligned according to the local reference coordinate system 316 to obtain a high-density point cloud of the scene. Then, the point cloud corresponding to a specific roadside sensor can be segmented, and the local pose of the roadside sensor (i.e., the pose relative to the local reference coordinate system 316) can be determined based on the segmented point cloud. The extrinsic parameters (or local extrinsic parameters) of the roadside sensor relative to the local reference coordinate system can be expressed as . In Figure 3 the example shown, the local extrinsic parameters of the roadside sensor 306 are expressed as T 1 , and the local extrinsic parameters of the roadside sensor 308 are expressed as T 2 .

[0048] The transformation matrix between the local reference coordinate system 316 and the road-based coordinate system 314 can be derived based on the spatial relationship between the portable lidar 304 and the road. It should be noted that the portable lidar device 304 can be equipped with multiple position sensors, such as RTK GPS, IMU, rotary encoders, etc. The pose of the portable lidar device 304 relative to the road-based coordinate system 314 can be determined based on the outputs of these position sensors. In one example, the local reference coordinate system 316 corresponds to the initial pose of the portable lidar device 304, which can be determined based on the outputs of the position sensors at the corresponding time point. For example, these position sensors can provide the precise position of a predetermined point on the portable lidar device 304, which can be used as the origin of the local reference coordinate system 316. Given the initial pose of the portable lidar device 304, the transformation matrix (denoted as ) between the local reference coordinate system 316 and the road-based coordinate system 314 can be calculated. Given the transformation matrix, a vector in the local reference coordinate system (denoted as ) can be converted to a vector in the road-based coordinate system (denoted as ) according to .

[0049] Then, the pose (or extrinsic parameters) of the roadside sensor relative to the road-based coordinate system 314 can be calculated based on the transformation matrix . More specifically, the extrinsic parameters of the roadside sensor can be obtained through Calculate. In Figure 3 In the example shown, the external parameters of the roadside sensor 306 can be obtained through calculation, and the external parameters of the roadside sensor 308 can be obtained through .

[0050] Figure 4 FIG. shows an exemplary block diagram of a sensor calibration system according to an embodiment of the present application. The sensor calibration system 400 may include a portable lidar device 402, a plurality of position sensors 404, a frame alignment subsystem 406, a point cloud overlay subsystem 408, a segmentation subsystem 410, a sensor attitude determination subsystem 412, a transformation matrix determination subsystem 414, and a parameter conversion subsystem 416.

[0051] The portable lidar device 402 can be carried by a human user or a data collection vehicle. The portable lidar device 402 can be configured to scan a scene (e.g., an intersection) that includes one or more roadside sensors. More specifically, the portable lidar device 402 can be configured to scan the scene from different angles to ensure that the outer surface of each roadside sensor is scanned sufficiently. For sensors at a higher position (e.g., sensors mounted on top of streetlight poles), the portable lidar device 402 may not be able to scan the entire outer surface of the sensor, but should scan as much of its surface area as possible.

[0052] The position sensors 404 may include, but are not limited to, RTK GPS, IMU, rotary encoders, etc. The position sensors 404 can provide accurate attitude information (e.g., position and orientation) about the portable lidar device 402. For example, RTK GPS can provide position information, while IMU can provide orientation information. In some embodiments, the position sensors 404 can be integrated inside the portable lidar device 402. In some embodiments, the position sensors 404 can be independent sensors placed in the same physical housing as the portable lidar device 402.

[0053] The frame alignment subsystem 406 can be responsible for aligning the frames in the frame stream obtained by the portable lidar device 402 so that the point clouds in the frames can be overlaid. In some embodiments, the frame alignment subsystem 406 can apply the ICP algorithm to align the frames. In one example, the ICP algorithm can be used to align two adjacent frames. More specifically, starting from the last frame, the frames in the frame stream can be aligned one by one until all frames are aligned according to the first frame. When the frame contains a reference object with obvious features (e.g., Figure 3The street lamp pole 312 and the cantilever 310 shown in [Figure] and the ground can improve the accuracy of frame alignment. In a variant, the frame alignment subsystem 406 can align frames according to a local reference coordinate system. The local reference coordinate system can be determined based on the pose of the portable lidar device 402 at a preset instant (e.g., the instant when the first frame is acquired). To improve the alignment efficiency, in some embodiments, before aligning the frames, the frame alignment subsystem 406 can remove transient objects (e.g., vehicles or pedestrians) from each frame.

[0054] The point cloud overlay subsystem 408 can be responsible for overlaying the point clouds in the aligned frames to obtain a large high-density point cloud. For a large installation site with many roadside sensors located at different positions, overlaying the frames can also include stitching the frames into a wide-angle frame. The segmentation subsystem 410 can be responsible for segmenting the point cloud of each roadside sensor in the large frame. In some embodiments, the segmentation subsystem 410 can apply machine learning techniques (e.g., by training a deep learning neural network) to perform the segmentation task. To ensure the accuracy of segmentation, the outer surface or housing of the roadside sensor can be made of a highly reflective material (e.g., metal or glass).

[0055] The sensor pose determination subsystem 412 can be responsible for determining the pose of each roadside sensor based on the corresponding segmented point cloud. In some embodiments, determining the pose of the roadside sensor can also include comparing the 3D point cloud of the roadside sensor with its known 3D CAD model or the 3D point cloud obtained by scanning the sensor before its installation (referred to as the pre-scanned point cloud). For example, the pose of the CAD model can be adjusted to find the match between the surface points on the CAD model and the segmented 3D point cloud. Similarly, the pose of the pre-scanned point cloud can be adjusted to match the pose of the segmented point cloud. The determined pose of the roadside sensor is relative to the local reference coordinate system.

[0056] The transformation matrix determination subsystem 414 can be responsible for determining the transformation matrix between two coordinate systems. In some embodiments, the transformation matrix determination subsystem 414 can calculate the transformation matrix between the road-based coordinate system and the local reference coordinate system according to the corresponding instantaneous pose of the portable lidar in the road-based coordinate system (e.g., ). It is worth noting that the instantaneous pose of the portable lidar can be determined according to the outputs of one or more position sensors associated with the portable lidar. The parameter conversion subsystem 416 can be responsible for converting the external parameters (e.g., pose) of each roadside sensor from the roadside sensor-based coordinate system to the local reference coordinate system and then to the road-based coordinate system according to the corresponding transformation matrix . In some embodiments, the external parameters of the roadside sensor can be converted to the road-based coordinate system according to and then to the road-based coordinate system.

[0057] Figure 4 The various subsystems included therein (e.g., subsystems 406-416) may include any combination of hardware and software modules. Additionally, the various subsystems may be integrated within the same computing entity (e.g., a stand-alone computer or server) or distributed across multiple computing entities (e.g., multiple servers). In one example, a lidar frame may be sent to one computer for frame alignment and overlay, and point cloud segmentation may be performed by a different computer. When a portable lidar device is mounted on a data collection vehicle, the computer mounted on the data collection vehicle may include all the subsystems (e.g., subsystems 406-416) required for sensor calibration.

[0058] Figure 5 A flowchart is presented that illustrates an exemplary sensor calibration process according to an embodiment of the present application. In one or more embodiments, Figure 5 one or more of the steps therein may be repeated and / or performed in a different order. Thus, Figure 5 the specific arrangement of the steps shown therein should not be construed as limiting the scope of the technology.

[0059] During operation, a user may place the portable lidar at a sensor mounting location (operation 502). One or more roadside sensors may be mounted at the sensor mounting location. In some examples, the roadside sensors may be mounted at intersections. In different examples, the roadside sensors may be mounted along a section of road. The portable lidar device may be hand-held and carried by a walking human user or may be a vehicle-mounted device mounted on a data collection vehicle that is parked near or driven near the roadside sensors.

[0060] The portable lidar may scan the outer surface of the roadside sensors to generate a series of lidar frames (operation 504). To calibrate all the roadside sensors within a particular mounting location (e.g., an intersection or a section of road), the portable lidar device should scan all the sensors and their surrounding environment. To ensure that the point cloud representing the roadside sensors has a sufficient number of points, the portable lidar device may scan the outer surface of the roadside sensors multiple times from different angles. Additionally, to facilitate subsequent frame alignment, in each scan or lidar frame, the portable lidar device should capture at least two reference objects with distinct and easily recognizable features. Examples of reference objects may include streetlight poles, utility poles, cantilevers, etc.

[0061] The sensor calibration system can align the lidar frames according to a preset local reference coordinate system (operation 506). Various techniques can be applied. In one embodiment, the system can apply the ICP algorithm to align the frames in different coordinate systems. In a variant, the spatial relationship between different coordinate systems can be determined based on the output of a position sensor associated with the portable lidar device, and the frames in different coordinate systems can be aligned according to the determined spatial relationship. In some embodiments, transient objects (such as vehicles or pedestrians) can be removed from the lidar frames to improve the alignment efficiency.

[0062] After aligning the frames according to the local reference coordinate system, the sensor calibration system can overlay the 3D point clouds of different frames to create a high-density 3D point cloud (operation 508). The high-density 3D point cloud can represent the sensor installation positions that include multiple roadside sensors. The data of the high-density point cloud is represented relative to the local reference coordinate system. The sensor calibration system can segment the 3D point cloud of each roadside sensor from the background (operation 510). In some embodiments, multiple roadside sensors can be segmented simultaneously. A machine learning-based method (such as a deep learning neural network) can be used to perform point cloud segmentation.

[0063] The sensor calibration system can then determine the extrinsic parameters (or poses) of each roadside sensor relative to the local reference coordinate system based on the corresponding segmented 3D point clouds (operation 512). In some embodiments, the 3D point cloud of the roadside sensor can be compared with its known shape and size or a known 3D model. In one example, the pose of the 3D CAD model of the roadside sensor can be adjusted to find the match between the surface points of the CAD model and the points in the 3D point cloud. In another example, the pose of the pre-scanned point cloud of the roadside sensor can be adjusted to match the pose of the segmented 3D point cloud. The pose information includes the position and orientation of the roadside sensor. Thus, the extrinsic parameters associated with each roadside sensor can be determined. It should be noted that the extrinsic parameters can include a rotation vector and a translation vector , and both the rotation vector and the translation vector are relative to the local reference coordinate system.

[0064] The sensor calibration system can also determine the transformation matrix between the local reference coordinate system and the road-based coordinate system (operation 514). In some embodiments, the transformation matrix can be determined based on the measured pose of the portable lidar device. The sensor calibration system transforms the external parameters (e.g., rotation vector and translation vector) in the local reference frame into the road-based coordinate system (operation 516). More specifically, the external parameters of each roadside sensor relative to the local reference coordinate system can be transformed into parameters relative to the road-based coordinate system, thus completing the calibration process of the roadside sensors. The external parameters of each roadside sensor relative to the road-based coordinate system can be output as the calibration result. The portable lidar device can also be transported to the next sensor installation location to calibrate the sensors there.

[0065] Compared with the existing roadside sensor calibration methods, the proposed solution does not require prior knowledge (e.g., position or orientation) of the roadside sensors and dedicated equipment. In addition, the calibration process does not cause traffic interruption and can be carried out by an operator in a relatively safe manner. To calibrate the roadside sensors installed at a higher position, there is no need for workers to stand on an aerial work platform for manual calibration. Moreover, when multiple roadside sensors are installed close to each other, these sensors can be calibrated simultaneously.

[0066] Figure 6 An example computer system facilitating sensor calibration operations according to an embodiment of the present application is shown. Computer system 600 includes a processor 602, a memory 604, and a storage device 606. In addition, computer system 600 can be connected to peripheral input / output (I / O) user devices 610, such as a display device 612, a keyboard 614, a pointing device 616, and a portable lidar 618. Storage device 606 can store an operating system 620, a sensor calibration system 622, and data 640. In some embodiments, computer system 600 can be implemented as part of an autonomous driving data acquisition system.

[0067] The sensor calibration system 622 may include a series of instructions that, when executed on the computer system 600, can cause the computer system 600 or the processor 602 to perform the methods and / or processes described in this disclosure. Specifically, the sensor calibration system 622 may include instructions for receiving a frame stream collected by a portable lidar device (frame reception instructions 624), instructions for aligning lidar frames (frame alignment instructions 626), instructions for superimposing point clouds in the aligned frames (point cloud superposition instructions 628), instructions for segmenting the point cloud corresponding to the roadside sensors (segmentation instructions 630), instructions for determining the pose (or extrinsic parameters) of each roadside sensor relative to the local reference coordinate system (sensor pose determination instructions 632), instructions for determining the transformation matrix between the local reference coordinate system and the road-based coordinate system (transformation matrix determination instructions 634), and instructions for converting the extrinsic parameters of each roadside sensor from the local reference coordinate system to the road-based coordinate system (parameter conversion instructions 636). The data 640 may include the shape and size information of the roadside sensors.

[0068] This disclosure presents a solution to the problems associated with calibrating a large number of roadside sensors. The solution allows for the simultaneous calibration of multiple roadside sensors installed at the same location (e.g., a traffic intersection or a section of road). To calibrate the sensors, a portable lidar device can be brought to the sensor installation site to scan the multiple roadside sensors and generate a frame stream. These frames can be aligned in the spatial domain according to the local reference coordinate system, and the aligned frames can be superimposed to generate a large high-density 3D point cloud. The 3D point cloud of each roadside sensor can be segmented from the large high-density 3D point cloud. The pose (or extrinsic parameters) of the roadside sensor to be calibrated relative to the local reference coordinate system can be determined based on the corresponding segmented 3D point cloud. The transformation matrix between the local reference coordinate system and the road-based coordinate system can be calculated, and the extrinsic parameters of the roadside sensor to be calibrated can be converted to the road-based coordinate system and output as the calibration result.

[0069] The data structures and program codes described in this detailed description are generally stored on a non-transitory computer-readable storage medium, which can be any device or medium capable of storing codes and / or data for use by a computer system. Non-transitory computer-readable storage media include, but are not limited to, volatile memories; non-volatile memories; electrical, magnetic, and optical storage devices, solid-state drives, and / or other non-transitory computer-readable media known now or developed in the future.

[0070] The methods and processes described in the detailed description can be embodied as code and / or data, which can be stored in the non-transitory computer-readable storage medium described above. When a processor or computer system reads and executes the code stored on the medium and operates on the data stored on the medium, the processor or computer system will execute the methods and processes embodied in the form of code and data structures and stored on the medium.

[0071] In addition, the optimized parameters obtained from the above methods and processes can be programmed into hardware modules, including but not limited to application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), and other programmable logic devices known now or developed in the future. When such a hardware module is activated, it will execute the methods and processes contained within the module.

[0072] The above embodiments are for illustration and description only and are not intended to be exhaustive or to limit the scope of the disclosure to the disclosed forms. Thus, many modifications and variations will be apparent to those skilled in the art. The scope of the disclosure is defined by the appended claims, rather than the foregoing disclosure.

Claims

1. A method for calibrating external parameters of roadside sensors for autonomous driving, characterized in that: The method comprises: placing a portable laser radar (lidar) device at a sensor mounting location containing one or more roadside sensors to be calibrated; Scanning the outer surface of the roadside sensor to be calibrated from different angles by the portable laser radar device to generate a frame stream; spatially aligning the frame stream based on a local reference coordinate system; Overlay the aligned frames; Segmenting a point cloud associated with the roadside sensor to be calibrated from the superimposed frames; Determining external parameters of the roadside sensor to be calibrated relative to the local reference coordinate system based on the segmented point cloud; and The extrinsic parameters are transformed from the local reference coordinate system to a road-based coordinate system.

2. The method according to claim 1, characterized in that: Aligning the frame streams includes applying an iterative closest point (ICP) algorithm.

3. The method according to claim 1, characterized in that The portable LiDAR device also includes one or more position sensors; and Aligning the frame stream includes determining an instantaneous pose of the portable LiDAR device associated with each frame in the frame stream based on measurements from a position sensor.

4. The method according to claim 3, characterized in that The local reference coordinate system is determined based on the instantaneous pose of the portable lidar device associated with a first frame in the frame stream.

5. The method according to claim 3, characterized in that: The position sensor includes one or more of the following sensors: Global Positioning System (GPS) sensors; Inertial Measurement Unit (IMU); and Rotary encoder.

6. The method according to claim 1, characterized in that Also included is determining a transformation matrix between the local reference coordinate system and a road-based coordinate system.

7. The method according to claim 6, characterized in that Converting the extrinsic parameters from the local reference coordinate system to the road-based coordinate system includes multiplying the extrinsic parameters with the transformation matrix.

8. The method according to claim 1, characterized in that The portable laser radar device is configured to scan at least two reference objects having distinct features in each frame; as well as Spatially aligning the frame streams includes aligning the reference object.

9. The method according to claim 1, characterized in that: Determining the external parameters of the roadside sensor to be calibrated includes comparing the segmented point cloud with a computer-aided design (CAD) model of the roadside sensor or a point cloud of the roadside sensor obtained by scanning the roadside sensor before installation.

10. The method according to claim 1, characterized in that Spatially aligning the stream of frames also includes removing transient objects from each frame.

11. A system for calibrating external parameters of roadside sensors for autonomous driving, characterized in that: include: A portable laser radar (lidar) device is brought to a sensor installation location containing one or more roadside sensors to be calibrated, wherein the portable laser radar is configured to scan the outer surface of the roadside sensors to be calibrated from different angles to generate a frame stream; A frame alignment subsystem, for spatially aligning the frame stream based on a local reference coordinate system; Frame overlay subsystem, used to overlay aligned frames; A segmentation subsystem for segmenting a point cloud associated with a roadside sensor to be calibrated from the superimposed frames; A sensor attitude determination subsystem, used to determine the external parameters of the roadside sensor to be calibrated relative to the local reference coordinate system based on the segmented point cloud; as well as A parameter conversion subsystem is used to convert the external parameters from the local reference coordinate system to a road-based coordinate system.

12. The system according to claim 11, characterized in that The frame alignment subsystem applies an iterative closest point (ICP) algorithm to align the frame streams.

13. The system according to claim 11, characterized in that The portable LiDAR device also includes one or more position sensors; and The frame alignment subsystem determines an instantaneous pose of the portable lidar device associated with each frame in a frame stream based on measurements from a position sensor.

14. The system according to claim 13, characterized in that The local reference coordinate system is determined based on the instantaneous pose of the portable lidar device associated with a first frame in the frame stream.

15. The system according to claim 13, characterized in that The position sensor includes one or more of the following sensors: Global Positioning System (GPS) sensors; Inertial Measurement Unit (IMU); and Rotary encoder.

16. The system according to claim 11, characterized in that A transformation matrix determination subsystem is also included for determining a transformation matrix between the local reference coordinate system and a road-based coordinate system.

17. The system according to claim 16, characterized in that The parameter conversion subsystem converts the extrinsic parameters from the local reference coordinate system to the road-based coordinate system by multiplying the extrinsic parameters with the transformation matrix.

18. The system according to claim 11, characterized in that The portable laser radar device is configured to scan at least two reference objects having distinct features in each frame; as well as The frame alignment subsystem aligns the reference object when aligning the frame streams.

19. The system according to claim 11, characterized in that The sensor posture determination subsystem determines the external parameters of the roadside sensor to be calibrated by comparing the segmented point cloud with a computer-aided design (CAD) model of the roadside sensor or a point cloud obtained by scanning the roadside sensor before installation.

20. The system according to claim 11, characterized in that When aligning the frame streams, the frame alignment subsystem removes transient objects from each frame.

Images