Point Cloud Map-Based LiDAR Recalibration System for Autonomous Vehicles

Abstract

Embodiments of the present disclosure disclose methods and systems for notifying an operator that a perception sensor of an autonomous driving vehicle (ADV) needs to be recalibrated. In one embodiment, the system senses a surrounding environment of the autonomous driving vehicle (ADV) that includes one or more obstacles. The system extracts feature information from a point cloud that previously stored and depicts a three-dimensional surrounding environment of the ADV. The system identifies one or more matching features between the extracted feature information and features of the one or more obstacles. The system determines an average offset distance based on each of the matching features. The system determines an average offset distance distribution over a period of time based on the average offset distance. If the average offset distance distribution meets a predetermined condition, the system issues an alert to the ADV to warn that one or more sensors are recommended or need to be recalibrated.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to operating an autonomous vehicle. More specifically, embodiments of the present disclosure relate to a point cloud-based light detection and ranging (LIDAR) recalibration system for an autonomous driving vehicle (ADV). Background Art

[0002] Vehicles operating in an automatic mode (e.g., driverless) can relieve certain driving-related responsibilities of passengers, especially drivers. When operating in an automatic mode, the vehicle is capable of navigating to various locations using on-board sensors, allowing the vehicle to travel with minimal human interaction or in some cases without any passengers.

[0003] Perception and prediction are key operations in autonomous driving. Perception allows the ADV to sense its environment. The perception system of the ADV generally includes cameras, LIDAR, RADAR, sensors, an inertial measurement unit (IMU), etc. These sensors / devices need to be calibrated to achieve high perception accuracy. Sometimes, the devices / sensors need to be recalibrated, for example when the operator cleans and / or reinstalls / replaces the sensors for maintenance. Summary of the Invention

[0004] In one aspect of the present application, there is provided a computer-implemented method for operating an autonomous driving vehicle (ADV), the method may include: using one or more sensors to sense the surrounding environment of the autonomous driving vehicle (ADV), the surrounding environment may include one or more obstacles; extracting feature information from a previously stored point cloud related to the surrounding environment of the ADV; identifying one or more matching features between the extracted feature information and the features of the one or more obstacles; determining an average offset distance for a planning period based on each of the matching features; and determining an average offset distance distribution for a predetermined number of planning periods based on the average offset distance, wherein if the average offset distance meets a predetermined condition, one or more sensors need to be recalibrated.

[0005] In another aspect of the present application, there is provided a non-transitory machine-readable medium, in which instructions may be stored, and when the instructions are executed by a processor, the operations may be caused to be performed, the operations including: using one or more sensors to sense the surrounding environment of the autonomous driving vehicle (ADV), the surrounding environment may include one or more obstacles; extracting feature information from a previously stored point cloud related to the surrounding environment of the ADV; identifying one or more matching features between the extracted feature information and the features of the one or more obstacles; determining an average offset distance for a planning period based on each of the matching features; and determining an average offset distance distribution for a predetermined number of planning periods based on the average offset distance, wherein if the average offset distance distribution meets a predetermined condition, one or more sensors need to be recalibrated.

[0006] In yet another aspect of the present application, a data processing system is provided. The system may include a processor and a memory coupled to the processor for storing instructions. When the instructions are executed by the processor, they cause the processor to perform operations, which may include sensing the surrounding environment of an autonomous driving vehicle (ADV) using one or more sensors. The surrounding environment may include one or more obstacles; extracting feature information from a previously stored point cloud related to the surrounding environment of the ADV; identifying one or more matching features between the extracted feature information and the features of the one or more obstacles; determining an average offset distance for a planning period based on each of the matching features; and determining an average offset distance distribution for a predetermined number of planning periods based on the average offset distance. If the average offset distance distribution meets a predetermined condition, then one or more sensors need to be recalibrated.

[0007] In yet another aspect of the present application, a computer program product is provided, including a computer program which, when executed by a processor, implements the method as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Embodiments of the present disclosure are shown by way of example and not limitation in the figures of the accompanying drawings, in which like reference numerals indicate like elements.

[0009] Figure 1 is a block diagram showing a networked system according to one embodiment.

[0010] Figure 2 is a block diagram showing an example of an autonomous driving vehicle according to one embodiment.

[0011] Figures 3A to 3B is a block diagram showing an example of an autonomous driving system used with an autonomous driving vehicle according to one embodiment.

[0012] Figure 4 is a block diagram showing a sensor calibration module according to one embodiment.

[0013] Figure 5A shows the perception of an ADV according to one embodiment.

[0014] Figure 5B is a diagram showing some feature information corresponding to the perception of the ADV in Figure 5A according to one embodiment.

[0015] Figure 5C is a top view of Figure 5A and Figure 5B according to one embodiment.

[0016] Figure 5DIt is a point cloud diagram showing characteristic information according to an embodiment.

[0017] Figure 6A It is an example probability density function showing the average offset distance distribution according to an embodiment.

[0018] Figure 6B It is an example probability density function showing the average offset distance distribution for triggering a re - calibration recommendation alert according to an embodiment.

[0019] Figure 6C It is an example probability density function showing the average offset distance distribution for triggering a re - calibration requirement alert according to an embodiment.

[0020] Figure 7 It is a flowchart showing a method according to an embodiment. Detailed Description

[0021] Various embodiments and aspects of the present disclosure will be described with reference to the details discussed below, and the accompanying drawings will illustrate the various embodiments. The following description and drawings are illustrative of the present disclosure and should not be construed as limiting the present disclosure. A variety of specific details are described to provide a thorough understanding of the various embodiments of the present disclosure. However, in some cases, well - known or conventional details are not described in order to provide a concise discussion of the embodiments of the present disclosure.

[0022] The phrase "an embodiment" or "embodiments" mentioned in the specification means that the specific features, structures, or characteristics described in connection with the embodiment can be included in at least one embodiment of the present disclosure. The phrase "in an embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment.

[0023] Embodiments of the present disclosure disclose a method and system for notifying and / or recalibrating the perception sensors of an autonomous driving vehicle (ADV). In one embodiment, the system uses one or more sensors to sense the surrounding environment of the autonomous driving vehicle (ADV), and the surrounding environment includes one or more obstacles. The system extracts characteristic information from a previously stored point cloud that maps the three - dimensional surrounding environment of the ADV. The system identifies one or more matching features between the extracted characteristic information and the characteristics of one or more obstacles. The system determines an average offset distance for a planning cycle based on each matching feature. The system determines an average offset distance distribution for a predetermined number of planning cycles based on the average offset distance. The system can send an alert to the ADV to alert one or more sensors for recommendation, or if the average offset distance distribution meets a predetermined condition, a recalibration is required.

[0024] Figure 1is a block diagram showing an autonomous driving network configuration according to an embodiment of the present disclosure. Referring to Figure 1 , the network configuration 100 includes an autonomous driving vehicle (ADV) 101, which is communicatively coupled to one or more servers 103 and 104 via a network 102. Although one ADV is shown, multiple ADVs can be coupled to each other and / or to servers 103 and 104 via the network 102. The network 102 can be any type of wired or wireless network, such as a local area network (LAN), a wide area network (WAN) such as the Internet, a cellular network, a satellite network, or a combination thereof. The servers 103 and 104 can be any type of server or cluster of servers, such as a Web or cloud server, an application server, a backend server, or a combination thereof. The servers 103 and 104 can be data analysis servers, content servers, traffic information servers, map and point of interest (MPOI) servers, or location servers, etc.

[0025] An ADV refers to a vehicle that can be configured to drive in an autonomous mode, in which the vehicle navigates through an environment with little or no input from a driver. Such an ADV can include a sensor system having one or more sensors configured to detect information about the environment in which the vehicle operates. The vehicle and its associated controller use the detected information to navigate through the environment. The ADV 101 is capable of operating in a manual mode, a fully autonomous mode, or a partially autonomous mode.

[0026] In one embodiment, the ADV 101 includes, but is not limited to, an autonomous driving system (ADS) 110, a vehicle control system 111, a wireless communication system 112, a user interface system 113, and a sensor system 115. The ADV 101 may also include certain common components included in a normal vehicle, such as an engine, wheels, a steering wheel, a transmission, etc., which can be controlled by the vehicle control system 111 and / or the ADS 110 using various communication signals and / or commands, such as an acceleration signal or command, a deceleration signal or command, a steering signal or command, a braking signal or command, etc.

[0027] The components 110 to 115 can be communicatively coupled to each other via an interconnect, a bus, a network, or a combination thereof. For example, the components 110 to 115 can be communicatively coupled to each other via a Controller Area Network (CAN) bus. The CAN bus is a vehicle bus standard designed to allow microcontrollers and devices to communicate with each other in applications without a host. It is a message-based protocol, originally designed for multiplexed electrical wiring in automobiles, but also used in many other environments.

[0028] Now referring to Figure 2, in one embodiment, the sensor system 115 includes, but is not limited to, one or more cameras 211, a Global Positioning System (GPS) unit 212, an Inertial Measurement Unit (IMU) 213, a radar unit 214, and a Light Detection and Ranging (LIDAR) unit 215. The GPS system 212 may include a transceiver operable to provide information about the position of the ADV. The IMU unit 213 may sense changes in the position and orientation of the ADV based on inertial acceleration. The radar unit 214 may represent a system that uses radio signals to sense objects within the local environment of the ADV. In some embodiments, in addition to sensing objects, the radar unit 214 may also sense the speed and / or heading of the objects. The LIDAR unit 215 may use lasers to sense objects in the environment in which the ADV is located. The LIDAR unit 215 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components. The camera 211 may include one or more devices for capturing images of the environment surrounding the ADV. The camera 211 may be a static camera and / or a video camera. For example, the camera may be mechanically movable by mounting it on a rotating and / or tilting platform.

[0029] The sensor system 115 may also include other sensors, such as sonar sensors, infrared sensors, steering sensors, throttle sensors, brake sensors, and audio sensors (e.g., microphones). The audio sensor may be configured to capture sounds from the environment surrounding the ADV. The steering sensor may be configured to sense the steering angle of the steering wheel, the wheels of the vehicle, or a combination thereof. The throttle sensor and the brake sensor sense the throttle position and the brake position of the vehicle, respectively. In some cases, the throttle sensor and the brake sensor may be integrated into an integrated throttle / brake sensor.

[0030] In one embodiment, the vehicle control system 111 includes, but is not limited to, a steering unit 201, a throttle unit 202 (also referred to as an acceleration unit), and a brake unit 203. The steering unit 201 is used to adjust the direction or heading of the vehicle. The throttle unit 202 is used to control the speed of the engine or motor, and thus control the speed and acceleration of the vehicle. The brake unit 203 decelerates the vehicle by providing friction to slow down the wheels or tires of the vehicle. It should be noted that the components shown in Figure 2 may be implemented in hardware, software, or a combination thereof.

[0031] Returning to the reference Figure 1, the wireless communication system 112 is used to allow communication between the ADV 101 and external systems such as devices, sensors, other vehicles, etc. For example, the wireless communication system 112 can communicate wirelessly with one or more devices directly or via a communication network (e.g., communicate wirelessly with the server 103 and the server 104 through the network 102). The wireless communication system 112 can use any cellular communication network or a wireless local area network (WLAN) such as WiFi to communicate with another component or system. For example, the wireless communication system 112 can communicate directly with devices (e.g., a passenger's mobile device, a display device, a speaker in the vehicle 101) using an infrared link, Bluetooth, etc. The user interface system 113 can be part of the peripheral devices implemented in the vehicle 101, for example, including a keyboard, a touch screen display device, a microphone, and a speaker, etc.

[0032] Some or all of the functions of the ADV 101 can be controlled or managed by the ADS 110, especially when operating in the autonomous driving mode. The ADS 110 includes the necessary hardware (e.g., processors, memories, storage devices) and software (e.g., operating systems, planning and routing programs) that receive information from the sensor system 115, the control system 111, the wireless communication system 112, and / or the user interface system 113, process the received information, plan a route or path from a starting point to a destination point, and then drive the vehicle 101 based on the planning and control information. Optionally, the ADS 110 can be integrated with the vehicle control system 111.

[0033] For example, a user as a passenger can specify the starting location and destination of a trip via a user interface such as the user interface. The ADS 110 obtains trip-related data. For example, the ADS 110 can obtain location and route data from an MPOI server, and the MPOI server can be part of the server 103 and the server 104. The location server provides location services, while the MPOI server provides map services and POIs of certain locations. Optionally, such location and MPOI information can be locally cached in the permanent storage device of the ADS 110.

[0034] When the ADV 101 moves along a route, the ADS 110 can also obtain real-time traffic information from a traffic information system or server (TIS). It should be noted that the server 103 and the server 104 can be operated by a third-party entity. Optionally, the functions of the server 103 and the server 104 can be integrated with the ADS 110. Based on the real-time traffic information, the MPOI information, the location information, and the real-time local environment data (e.g., obstacles, objects, nearby vehicles) detected or sensed by the sensor system 115, the ADS 110 can plan an optimal route and drive the vehicle 101 along the planned route via a control system such as the control system 111 to safely and effectively reach the specified destination.

[0035] Server 103 may be a data analysis system that performs data analysis services for various clients. In one embodiment, the data analysis system 103 includes a data collector 121 and a machine learning engine 122. The data collector 121 collects driving statistics 123 from a variety of vehicles, which may be ADV or conventional vehicles driven by human drivers. The driving statistics 123 include information indicating issued driving commands (e.g., throttle command, brake command, steering command) and information indicating the responses of the vehicle captured by the vehicle's sensors at different time points (e.g., speed, acceleration, deceleration, direction). The driving statistics 123 may also include information describing the driving environment at different time points, such as the route (including the starting position and the destination position), MPOI, road conditions, weather conditions, etc.

[0036] Based on the driving statistics 123, the machine learning engine 122 generates or trains a set of rules, algorithms, and / or prediction models 124 for various purposes. In one embodiment, the algorithms 124 may include algorithms for obstacle detection, classification, feature extraction, etc.

[0037] The algorithms 124 may include algorithms for detecting obstacles based on images from the ADV's perception system. The algorithms 124 may include algorithms for classifying objects into static elements (e.g., landmarks, buildings, lamp posts, traffic signs / lights, trees, etc.) and / or dynamic elements (e.g., vehicles, pedestrians, etc.). The algorithms 124 may include algorithms for extracting feature information from the detected obstacles. Then, the algorithms 124 may be uploaded to the ADV for real-time utilization during autonomous driving.

[0038] Figure 3A and Figure 3B is a block diagram showing an example of an autonomous driving system used with an ADV according to one embodiment. The system 300 may be implemented as Figure 1 a part of the ADV 101, and the system 300 includes, but is not limited to, an ADS 110, a control system 111, and a sensor system 115. Referring to Figure 3A and Figure 3B , the ADS 110 includes, but is not limited to, a positioning module 301, a perception module 302, a prediction module 303, a decision module 304, a planning module 305, a control module 306, a route arrangement module 307, and a sensor calibration module 308.

[0039] Some or all of the modules 301 to 308 may be implemented in software, hardware, or a combination thereof. For example, these modules may be installed in a permanent storage device 352, loaded into a memory 351, and executed by one or more processors (not shown). It should be noted that some or all of these modules may be communicatively coupled to Figure 2Some or all of the modules of the vehicle control system 111, or integrated with Figure 2 Some of the modules among modules 301 to 308 may be integrated together as an integrated module. For example, module 308 may be implemented as part of the prediction module 302.

[0040] The positioning module 301 determines the current position of the ADV 300 (e.g., by means of the GPS unit 212) and manages any data related to the user's trip or route. The positioning module 301 (also referred to as the map and route arrangement module) manages any data related to the user's trip or route. The user can log in via, for example, a user interface and specify the starting position and destination of the trip. The positioning module 301 communicates with other components of the ADV 300 (e.g., the map and route arrangement data 311) to obtain trip-related data. For example, the positioning module 301 can obtain location and route data from a location server and a map and POI (MPOI) server. The location server provides location services, and the MPOI server provides map services and POIs of certain locations, and the POIs of certain locations can be cached as part of the map and route data 311. When the ADV 300 moves along a route, the positioning module 301 can also obtain real-time traffic information from a traffic information system or server.

[0041] Based on the sensor data provided by the sensor system 115 and the location information obtained by the positioning module 301, the perception module 302 determines the perception of the surrounding environment. The perception information can represent what an ordinary driver would perceive around the vehicle the driver is driving. The perception can include, for example, lane configurations in the form of objects, traffic light signals, the relative position of another vehicle, pedestrians, buildings, sidewalks, or other traffic-related signs (e.g., stop signs, yield signs). Lane configurations include information describing one or more lanes, such as the shape of the lane (e.g., straight or curved), the width of the lane, how many lanes there are in the road, one-way or two-way lanes, merging or bifurcating lanes, exit lanes, etc.

[0042] The perception module 302 may include a computer vision system or the functions of a computer vision system to process and analyze the images captured by one or more cameras in order to identify objects and / or features in the environment of the ADV. The objects may include traffic signals, road boundaries, other vehicles, pedestrians, and / or obstacles, etc. The computer vision system can use object recognition algorithms, video tracking, and other computer vision technologies. In some embodiments, the computer vision system can draw the environment, track objects, and estimate the speed of objects, etc. The perception module 302 can also detect objects based on other sensor data provided by other sensors (e.g., radar and / or LIDAR).

[0043] For each of the objects, the prediction module 303 predicts how the object will behave in the environment. The prediction is performed based on the perception data of the driving environment received at a time point for a set of map / route information 311 and traffic rules 312. For example, if the object is a vehicle in the opposite direction and the current driving environment includes an intersection, the prediction module 303 will predict whether the vehicle will likely move straight ahead or turn. If the perception data indicates that there is no traffic light at the intersection, the prediction module 303 may predict that the vehicle may have to come to a complete stop before entering the intersection. If the perception data indicates that the vehicle is currently in a left-turn only lane or a right-turn only lane, the prediction module 303 may predict that the vehicle is more likely to make a left turn or a right turn, respectively.

[0044] For each of the objects, the decision-making module 304 makes a decision on how to handle the object. For example, for a specific object (e.g., another vehicle in an intersection) and its metadata describing the object (e.g., speed, direction, steering angle), the decision-making module 304 determines how to encounter the object (e.g., overtake, yield, stop, pass). The decision-making module 304 may make such decisions based on a set of rules (e.g., traffic rules or driving rules 312) that can be stored in the permanent storage device 352.

[0045] The route arrangement module 307 is configured to provide one or more routes or paths from a starting point to a destination point. For a given trip from a starting position to a destination position received from a user, for example, the route arrangement module 307 obtains route and map information 311 and determines all possible routes or paths from the starting position to the destination position. The route arrangement module 307 may generate a reference line in the form of a topographic map for each route from the starting position to the destination position. The reference line refers to an ideal route or path without any interference from other factors (e.g., other vehicles, obstacles, or traffic conditions).

[0046] That is, if there are no other vehicles, pedestrians, or obstacles on the road, the ADV should exactly or approximately follow the reference line. Then the topographic map is provided to the decision-making module 304 and / or the planning module 305. The decision-making module 304 and / or the planning module 305 check all possible routes for other data provided by other modules, such as traffic conditions from the positioning module 301, the driving environment sensed by the perception module 302, and the traffic conditions predicted by the prediction module 303, to select and modify one of the best routes. Depending on the specific driving environment at that time point, the actual path or route for controlling the ADV may be close to or different from the reference line provided by the route arrangement module 307.

[0047] Based on the decisions for each perceived object, the planning module 305 uses the reference line provided by the routing module 307 as a basis to plan the path or route of the ADV, as well as driving parameters (e.g., distance, speed, and / or steering angle). That is, for a given object, the decision-making module 304 determines what to do with the object, while the planning module 305 determines how to do it. For example, for a given object, the decision-making module 304 may decide to pass the object, and the planning module 305 may determine whether to pass on the left or right side of the object. The planning and control data is generated by the planning module 305, and this data includes information describing how the vehicle 300 will move in the next movement cycle (e.g., the next route / path segment). For example, the planning and control data may indicate that the vehicle 300 moves at a speed of 30 miles per hour (mph) for 10 meters, and then changes to the right lane at a speed of 25 mph.

[0048] Based on the planning and control data, the control module 306 controls and drives the ADV by sending appropriate commands or signals to the vehicle control system 111 according to the route or path defined by the planning and control data. The planning and control data includes sufficient information to drive the vehicle from the first point to the second point of the route or path using appropriate vehicle settings or driving parameters (e.g., throttle, brake, steering commands) at different time points along the path or route.

[0049] In one embodiment, the planning phase is executed in multiple planning cycles, which are also referred to as driving cycles. For example, it is executed at an interval of every 100 milliseconds (ms). For each planning cycle or driving cycle, one or more control commands will be issued based on the planning and control data. That is, for every 100 ms, the planning module 305 plans the next road segment or path segment, for example, including the target position and the time required for the ADV to reach the target position. Optionally, the planning module 305 may also define specific speed, direction, and / or steering angle, etc. In one embodiment, the planning module 305 plans the route segment or path segment for the next predetermined time period (e.g., 5 seconds). For each planning cycle, the planning module 305 plans the target position for the current cycle (e.g., the next 5 seconds) based on the target position planned in the previous cycle. Then, the control module 306 generates one or more control commands (e.g., throttle control command, brake control command, steering control command) based on the planning and control data of the current cycle.

[0050] It should be noted that the decision-making module 304 and the planning module 305 can be integrated into an integrated module. The decision-making module 304 / planning module 305 can include a navigation system or the functions of a navigation system to determine the driving path of the ADV. For example, the navigation system can determine a series of speeds and direction headings to affect the movement of the ADV along the path, which basically avoids the perceived obstacles while generally making the ADV drive along a road-based path leading to the final destination. The destination can be set according to user input via the user interface system 113. When the ADV is in operation, the navigation system can dynamically update the driving path. The navigation system can combine data from the GPS system and one or more maps in order to determine the driving path of the ADV.

[0051] Figure 4 is a block diagram showing an example of a sensor calibration module according to one embodiment. The sensor calibration module 308 can be implemented as Figure 3A a part of the perception module. The sensor calibration module 308 can determine and / or notify the operator whether any of the perception sensors need to be recalibrated. Referring to Figure 4 , the sensor calibration module 308 can include sub-modules such as the environmental perception 401, the feature information extractor 402, the feature matching determiner 403, the offset distance determiner 404, the average offset distance determiner 405, the offset distance distribution determiner 406, the threshold determiner 407, and the alarm sender 408. It should be noted that some of these modules can be integrated into fewer modules or a single module.

[0052] The environmental perception 401 can perceive the environment around the ADV. The environment can be perceived by various sensors / image capture devices of the ADV, including RGB cameras, LIDAR, RADAR, time-of-flight cameras, etc. installed for the ADV to capture three-dimensional environmental information around the ADV. The feature information extractor 402 can extract feature information from the previously stored three-dimensional (3D) environmental information. The previously stored 3D environmental information can be a point cloud / RGB image / depth image, or a combination thereof, etc. Here, the 3D environmental information can be captured in advance by various sensors of the ADV 101 and / or other ADVs. The feature information extracted from the previously stored 3D environmental information can be used by the ADV 101 to match the features perceived by the perception system of the ADV 101.

[0053] The feature matching determiner 403 can determine whether there are matching features / obstacles between the image currently captured by the ADV 101 and the retrieved images (RGB images, point cloud maps, etc.) previously captured by the ADV 101 and / or other ADVs with calibrated sensors. The offset distance determiner 404 can determine the offset distance of a specific matching feature / obstacle. The average offset distance determiner 405 can calculate the average value of the offset distances based on all the matching features / obstacles of the image frame (or planning cycle). Here, an image frame can be captured for each planning cycle (about one frame every 100 milliseconds). The average offset distance distribution determiner 406 can determine the average offset distance distribution for a predetermined time interval based on the average offset distance. The threshold determiner 407 can determine one or more thresholds, where the average value of the average offset distance distribution deviating from zero offset triggers a recalibration alert / notification. The alert (notification) sender 408 can send a notification from module 308 to the ADV 101 to notify the operator that sensor recalibration is recommended and / or required.

[0054] Figure 5A Shows the perception of the ADV according to one embodiment. Figure 5B Is shown corresponding to Figure 5A Some feature information of the perception of the ADV in Figure 5C Is Figure 5A And Figure 5B Top view of. Refer to Figures 5A to 5C , for one example, the ADV101 (not shown in Figure 5A And Figure 5B ) can travel along a road 500 with two distinguishable obstacles 501 and 502. The ADV 101 can include one or more perception sensors, including RGB cameras, LIDAR, RADAR, time-of-flight (ToF) cameras, or a combination thereof, etc. In one embodiment, the ADV 101 perceives the surrounding environment by capturing red-green-blue (RGB) images from an RGB camera, ToF images from a ToF camera, and / or point clouds from a LIDAR sensor. In one embodiment, for multi-sensor fusion perception, the images can be fused together. Based on the measurements from the IMU (IMU previously calibrated to the sensor) and the captured images, the ADV 101 can generate a quaternion representing the 3D position and the heading information of the ADV 101. When calibrated, the static obstacles perceived by the perception system of the ADV 101 should appear in the same position. It should be noted that although the following discloses embodiments of recalibration based on camera images / LIDAR point clouds, the same recalibration embodiments can be applied to other images and / or multi-sensor fusion images.

[0055] In one embodiment, if one or more sensors are no longer aligned with the IMU, the perceived point cloud (or image) will indicate an offset compared to the image previously captured by the sensors of ADV 101 or other ADV. In one embodiment, ADV 101 can automatically determine whether the misalignment is within an acceptable threshold. If not, ADV 101 sends a recommendation and / or an alert requesting recalibration to the operator of ADV 101.

[0056] Referring Figures 5A to 5C , in one embodiment, ADV 101 retrieves a point cloud / image depicting the 3D surrounding environment of the ADV based on the position of the ADV (the position determined by GPS / IMU, etc.). The point cloud / image can be retrieved as part of Figure 3A the map and route arrangement data 311. Next, ADV 101 extracts feature information from the retrieved point cloud. In this case, referring Figures 5A to 5C , the feature information can be the direction / edge / curve / features of obstacles captured by the point cloud / image (e.g., lamp post 501 and traffic sign 502). In one embodiment, a predetermined number of features are selected for extraction. In another embodiment, all features are extracted. In one embodiment, the features are classified as static or dynamic elements based on a classification algorithm, and only static elements (e.g., buildings, trees, traffic signs, lane lines, and lamp posts) are selected for extraction, and dynamic elements (e.g., vehicles, pedestrians) are not selected for extraction.

[0057] Next, based on the real-time point cloud / image captured by ADV 101 depicting the 3D surrounding environment of ADV 101, ADV 101 extracts features from the real-time point cloud / image, where the features reflect the obstacles perceived by the ADV (e.g., obstacles 501 and 502). Here, the obstacles can be perceived through an image recognition algorithm (e.g., edge detection algorithm, machine learning algorithm, including but not limited to deep convolutional neural network). It should be noted that a neural network is a machine learning model that can learn to perform a task by considering examples (e.g., training with input / output scenarios) without being programmed with any task-specific rules. A neural network is a computational method based on a large number of neural units or neurons in a series of hidden layers or inner layers. Each hidden layer consists of a set of neurons, where each neuron is connected to one or more neurons in the previous layer, and where the neurons in a single layer can act completely independently and do not share any connections with other neurons in that layer. A neural network is self-learning and self-training, rather than explicitly programmed. A convolutional neural network is a deep neural network with fully connected layers. A fully connected layer is an inner layer with neurons that are fully connected to all neurons in the previous layer.

[0058] In one embodiment, ADV 101 identifies one or more matching features between the extracted feature information of the currently captured point cloud / image and the retrieved point cloud / image depicting the static elements 501 and 502. Based on the matching features, ADV 101 determines an average offset distance for each matching feature. The average offset distance for a planning cycle can be calculated by averaging the offset distances of all the matching features. In one embodiment, the offset distance of a feature can be calculated based on the distance between the ADV and the extracted features of the previously stored point cloud / image and the distance between the ADV and the extracted features of the real-time point cloud / image. In one embodiment, the offset distance is a vector offset distance (e.g., for a two-dimensional vector, a distance with an x-component and a y-component). For example, the distance from the ADV to the feature extracted in real time and the distance from the ADV to the feature extracted from the retrieved point cloud are vector distances. In one embodiment, the vector distance is a 2D plane distance (e.g., ignoring height to simplify the calculation).

[0059] In one embodiment, based on the average offset distances over multiple planning cycles (or a predetermined time period), ADV 101 generates an average offset distance distribution for the multiple planning cycles. Here, the distribution has a sequence of points, and each point is the average offset distance for a given planning cycle. It should be noted that the number of planning cycles can be set to 100 milliseconds (or 1 cycle), 1 minute, 10 minutes, half an hour, etc. Then this distribution can be used to determine whether there is a perception sensor misalignment of any of the sensors capturing the point cloud / image. In one embodiment, the distribution can be discretized to reduce the storage capacity. For example, the distribution includes a discrete number of bins, and the average offset distance points near the bins are represented by the bins, where each bin is represented by a center value and a range.

[0060] In one embodiment, ADV 101 calculates one or more thresholds, where a deviation of the distribution from zero offset will trigger an alarm. In one embodiment, ADV 101 calculates an average value based on the distribution. The average value can be calculated by averaging the points in the distribution. In one embodiment, the first threshold is one standard deviation, and the second threshold is two standard deviations. The standard deviation measures the amount of variation or dispersion of a set of values and can be calculated based on the average value and a set of points in the distribution. In one embodiment, a zero offset within one standard deviation of the average value of the average offset distance distribution does not trigger an alarm. In one embodiment, a zero offset that is more than one standard deviation but less than two standard deviations from the average value of the average offset distance distribution triggers a recommended alarm. In one embodiment, a zero offset that is more than two standard deviations from the average value of the average offset distance distribution triggers a demand alarm.

[0061] In one embodiment, the average offset distance distribution is a vector average offset distance distribution. For example, the distribution has an x - component and a y - component for a 2D planar distribution. In one embodiment, a zero offset within one standard deviation of the average of the two vector components of the distribution does not trigger an alarm. In one example, for any vector component of the distribution, a zero offset from the average that is greater than one standard deviation but less than two standard deviations triggers a recommended alarm. In one embodiment, for any vector component of the distribution, a zero offset from the average that is greater than two standard deviations triggers a demand alarm. In one embodiment, the ADV automatically recalibrates by determining an offset vector calculated as the average of each vector average offset distance distribution component. The offset vector is added to the misaligned sensor readings to compensate for the misalignment. Here, the misaligned sensor introduces a bias offset / error. However, different sensors can include random errors for each reading. The averaging of the offset distances and the generation of the distribution based on the average offset distance can highlight the bias offset / error for recalibration determination.

[0062] Figure 5D is a point cloud diagram showing characteristic information according to one embodiment. Refer to Figure 5D , feature 511 is similar to a building, which will be classified as a static element using a classification algorithm, and features 512 to 514 are similar to vehicles. Here, features 512 to 514 will be classified as dynamic elements, and only feature 511 (instead of features 512 to 514) will be extracted and used for feature matching for sensor recalibration.

[0063] Figure 6A is an example probability density function showing the average offset distance distribution according to one embodiment. Figure 6B is an example probability density function showing the average offset distance distribution that triggers a recalibration recommended alarm according to one embodiment. Figure 6C is an example probability density function showing the average offset distance distribution that triggers a recalibration demand alarm according to one embodiment. Refer to Figure 6A , since the average of distribution 601 is within the distance of one standard deviation threshold from the zero offset, no notification of sensor recalibration is sent to the operator of the ADV 101. Refer to Figure 6B , since the zero offset from the average of distribution 602 is greater than one standard deviation but less than two standard deviations, the ADV 101 sends a first - level alarm to the operator to recommend sensor recalibration. Refer to Figure 6C, since the average value of the zero-offset distribution 603 is greater than two standard deviations, the ADV 101 sends a second-level alert to the operator to request recalibration of the sensor. Although not shown, the alerts at both levels can be any type of alert, such as displaying text (e.g., recommended recalibration and / or recalibration required), color change (yellow and / or red), or a combination thereof, or sound / harmonic, or touch feedback, etc. on the dashboard display of the ADV.

[0064] Figure 7 is a flowchart showing a method according to one embodiment. The process 700 can be executed by processing logic that can include software, hardware, or a combination thereof. For example, the process 700 can be executed by Figure 3A the sensor calibration module 308. Referring to Figure 7 , at block 701, the processing logic uses one or more sensors to sense the surrounding environment of the autonomous driving vehicle (ADV), which includes one or more obstacles. At block 702, the processing logic extracts feature information from a previously stored point cloud that maps the three-dimensional surrounding environment of the ADV. At block 703, the processing logic identifies one or more matching features between the extracted feature information and the features of one or more obstacles. At block 704, the processing logic determines an average offset distance for a planning cycle (e.g., a cycle of 100 milliseconds) based on each matching feature. At block 705, the processing logic determines an average offset distance distribution for a predetermined number of planning cycles (e.g., 10 minutes) based on the average offset distance. At block 706, if the average offset distance distribution meets a predetermined condition, the processing logic sends a warning to the ADV to warn that it is recommended or necessary for one or more sensors to be recalibrated.

[0065] In one embodiment, if the average value of the average offset distance distribution is greater than a threshold distance away from zero offset, an alert is sent to the ADV to warn that it is recommended or necessary for one or more sensors to be recalibrated. In one embodiment, determining an average offset distance for a planning cycle based on each matching feature includes: for each matching feature, determining the offset distance between the feature of the obstacle and its corresponding matching feature, and determining the average offset distance for the planning cycle by averaging the determined offset distances of the matching features.

[0066] In one embodiment, the offset distance is an offset vector distance. In one embodiment, determining the offset distance and its corresponding matching feature with the obstacle includes determining a first vector distance between the ADV and the feature of the obstacle, determining a second vector distance between the ADV and the corresponding matching feature, and determining the offset distance as the difference vector between the first vector distance and the second vector distance.

[0067] In one embodiment, the average offset distance distribution includes vectors of the average offset distance distribution, where if the average value of any vector component of the average offset distance distribution is greater than a threshold distance from zero offset for the corresponding vector component, the operator of the ADV is warned that one or more sensors need to be recalibrated. In one embodiment, the threshold distance is approximately two standard deviations.

[0068] In one embodiment, one or more sensors are recalibrated based on the average offset distance distribution such that for a predetermined number of planning cycles, the average offset distance has an offset that is approximately zero. In one embodiment, the one or more sensors include a light detection and ranging (LIDAR) sensor and an inertial measurement unit (IMU) sensor, where the LIDAR sensor is recalibrated with reference to the IMU. In one embodiment, the feature information includes features of static landmarks of the surrounding environment (including buildings, traffic signs, lane lines, and lamp posts).

[0069] It should be noted that some or all of the components shown and described above may be implemented in software, hardware, or a combination thereof. For example, these components may be implemented as software installed and stored in a permanent storage device, which may be loaded and executed by a processor (not shown) in a memory to perform the processes or operations described throughout this application. Alternatively, these components may be implemented as executable code programmed or embedded in dedicated hardware, such as an integrated circuit (e.g., a dedicated IC or ASIC), a digital signal processor (DSP), or a field-programmable gate array (FPGA), and the executable code may be accessed from an application via the corresponding drivers and / or operating system. Additionally, these components may be implemented as specific hardware logic in a processor or processor core, as part of an instruction set that can be accessed by software components via one or more specific instructions.

[0070] Some portions of the foregoing detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. In this context, an algorithm is generally conceived to be a self-consistent sequence of operations leading to a desired result. These operations are those requiring physical manipulation of physical quantities.

[0071] However, it should be borne in mind that all these and similar terms are intended to be associated with appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise expressly stated in the above discussion, it should be understood that throughout the specification, discussions using terms such as those set forth in the appended claims refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities within the registers and memories of the computer system and transforms that data into other data similarly represented as physical quantities within the memories or registers of the computer system or other such information storage, transmission, or display devices.

[0072] Embodiments of the present disclosure also relate to apparatus for performing the operations herein. Such a computer program is stored in a non-transitory computer-readable medium. Machine-readable media include any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, machine-readable (e.g., computer-readable) media include machine (e.g., computer) readable storage media (e.g., read only memory (“ROM”), random access memory (“RAM”), magnetic disk storage media, optical storage media, flash memory devices).

[0073] The processes or methods depicted in the foregoing figures may be executed by processing logic that includes hardware (e.g., circuits, dedicated logic, etc.), software (e.g., embodied on a non-transitory computer-readable medium), or a combination of both. Although the processes or methods have been described above in terms of some sequential operations, it should be understood that some of the operations may be performed in a different order. In addition, some operations may be performed in parallel rather than sequentially.

[0074] Embodiments of the present disclosure have not been described with reference to any particular programming language. It should be appreciated that a variety of programming languages may be used to implement the teachings of the embodiments of the present disclosure as described herein.

[0075] In the foregoing specification, embodiments of the present disclosure have been described with reference to specific exemplary embodiments of the present disclosure. It will be apparent that various modifications may be made thereto without departing from the broader spirit and scope of the present disclosure set forth in the appended claims. Accordingly, the specification and drawings are to be interpreted in an illustrative rather than a restrictive sense.

Claims

1. A computer - implemented method for operating an autonomous driving vehicle, comprising: Using one or more sensors to sense the surrounding environment of the autonomous driving vehicle (ADV), including one or more obstacles; Extracting feature information from a point cloud previously stored and related to the surrounding environment of the ADV; Identifying one or more matching features between the extracted feature information and the features of the one or more obstacles, the features of the one or more obstacles being sensed by the one or more sensors; Based on each of the matching features, determining an average offset distance for a planning period, the planning period corresponding to a driving period with a predetermined time interval; And Based on the average offset distance, determining an average offset distance distribution for a predetermined number of planning periods, the predetermined number of planning periods being greater than 1, wherein if the average offset distance meets a predetermined condition, the one or more sensors need to be recalibrated.

2. The method according to claim 1, wherein If the average value of the average offset distance is greater than a threshold distance away from zero offset, the one or more sensors need to be recalibrated.

3. The method according to claim 2, wherein Determining the average offset distance for the planning period based on each of the matching features includes: For each of the matching features, determining an offset distance from the feature of the obstacle and its corresponding matching feature; and Determining the average offset distance for the planning period by averaging the determined offset distances of the matching features.

4. The method according to claim 3, wherein The offset distance is an offset vector distance.

5. The method according to claim 4, wherein Determining the offset distance from the feature of the obstacle and its corresponding matching feature includes: Determining a first vector distance between the ADV and the feature of the obstacle; Determining a second vector distance between the ADV and the corresponding matching feature; and Determining the offset distance between the first vector distance and the second vector distance as a difference vector.

6. The method according to claim 5, wherein, The average offset distance distribution includes vectors of the average offset distance distribution, wherein if the average value of any vector component of the average offset distance distribution is greater than the threshold distance away from the zero offset of the corresponding vector component, the operator of the ADV is warned that the one or more sensors need to be recalibrated.

7. The method according to claim 2, wherein, The threshold distance is two standard deviations.

8. The method according to claim 2, further comprising recalibrating the one or more sensors based on the average offset distance distribution such that subsequent sensor readings have approximately zero offset.

9. The method according to claim 2, wherein, The one or more sensors include a light detection and ranging (LIDAR) sensor and an inertial measurement unit (IMU) sensor, wherein the LIDAR sensor is recalibrated relative to the IMU sensor.

10. The method according to claim 1, wherein, The feature information includes features of static landmarks in the surrounding environment, the static landmark features including building landmarks, traffic signs, lane lines, and lamp posts, etc.

11. A non - transitory machine - readable medium storing instructions that, when executed by a processor, cause the processor to perform operations, the operations including: Using one or more sensors to sense the surrounding environment of an autonomous driving vehicle (ADV), the surrounding environment including one or more obstacles; Extract feature information from a previously stored point cloud related to the surroundings of the ADV; Identify one or more matching features between the extracted feature information and the features of the one or more obstacles, the features of the one or more obstacles being sensed by the one or more sensors; Determine an average offset distance for a planning period based on each of the matching features, the planning period corresponding to a driving period having a predetermined time interval; And Determine an average offset distance distribution for a predetermined number of planning periods based on the average offset distance, the predetermined number of planning periods being greater than 1, wherein if the average offset distance distribution meets a predetermined condition, the one or more sensors need to be recalibrated.

12. The non-transitory machine-readable medium according to claim 11, wherein, If the average value of the average offset distance distribution is greater than a threshold distance away from zero offset, the one or more sensors need to be recalibrated.

13. The non-transitory machine-readable medium according to claim 12, wherein, Determining the average offset distance for the planning period based on each of the matching features includes: For each of the matching features, determine an offset distance from the feature of the obstacle and its corresponding matching feature; and Determine the average offset distance for the planning period by averaging the determined offset distances of the matching features.

14. The non-transitory machine-readable medium according to claim 13, wherein, The offset distance is an offset vector distance.

15. The non-transitory machine-readable medium according to claim 14, wherein, Determining the offset distance from the feature of the obstacle and its corresponding matching feature includes: Determine a first vector distance between the ADV and the feature of the obstacle; Determine a second vector distance between the ADV and the corresponding matching feature; and Determine the offset distance between the first vector distance and the second vector distance as a difference vector.

16. A data processing system, comprising: A processor; And A memory coupled to the processor and storing instructions that, when executed by the processor, cause the processor to perform operations, the operations including Sense the surroundings of an autonomous driving vehicle (ADV) using one or more sensors, the surroundings including one or more obstacles; Extract feature information from a previously stored point cloud related to the surroundings of the ADV; Identify one or more matching features between the extracted feature information and the features of the one or more obstacles, the features of the one or more obstacles being sensed by the one or more sensors; Determine an average offset distance for a planning period based on each of the matching features, the planning period corresponding to a driving period having a predetermined time interval; And Determine an average offset distance distribution for a predetermined number of planning periods based on the average offset distance, the predetermined number of planning periods being greater than 1, wherein if the average offset distance distribution meets a predetermined condition, the one or more sensors need to be recalibrated.

17. The system according to claim 16, wherein, If the average value of the average offset distance is greater than a threshold distance away from zero offset, the one or more sensors need to be recalibrated.

18. The system according to claim 17, wherein, Determining the average offset distance for the planning period based on each of the matching features includes: For each of the matching features, determine an offset distance from the obstacle and its corresponding matching feature; and Determine the average offset distance for the planning period by averaging the determined offset distances of the matching features.

19. The system according to claim 18, wherein, The offset distance is an offset vector distance.

20. The system according to claim 19, wherein Determining the offset distance from the feature of the obstacle and its corresponding matching feature includes: Determine a first vector distance between the ADV and the feature of the obstacle; Determine a second vector distance between the ADV and the corresponding matching feature; and Determine the offset distance between the first vector distance and the second vector distance as a difference vector.

21. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-10.

Images