Cloud-based scanning for detecting autonomous vehicle sensor failure

Abstract

In one embodiment, a system obtains first point cloud data generated by a first scanning device of a first autonomous driving vehicle (ADV). The system determines anomalies of the first point cloud data by applying one or more anomaly detection algorithms to the first point cloud data. The system determines an anomaly type of the anomaly. The system determines if the anomaly is above a severity threshold based on the anomaly type. In response to determining that the anomaly is above a severity threshold, the system issues an alert to check for any hardware failure of the first scanning device.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to operating autonomous vehicles and more particularly to cloud-based local scanning for detecting autonomous driving vehicle (ADV) sensor failures. Background Art

[0002] A vehicle operating in an automated mode (e.g., driverless mode) can relieve occupants, especially the driver, of some driving-related tasks. When a vehicle is operating in an automated mode, onboard sensors can be used to navigate the vehicle to different locations, allowing the vehicle to be driven with minimal human intervention, or in some cases without any occupants.

[0003] On-board LiDAR (Light Detection and Ranging) devices / sensors are key components for autonomous driving. Failure of LiDAR devices may lead to potential accidents on the road, and identifying early signs of failure can prevent potential accidents caused by failure. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Embodiments of the present disclosure are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements.

[0005] Figure 1 A block diagram of a networking system according to one embodiment is shown.

[0006] Figure 2 A block diagram of an example of an autonomous vehicle is shown according to one embodiment.

[0007] Figure 3A and Figure 3B A block diagram of an example of an autonomous driving system for use with an autonomous vehicle is shown according to one embodiment.

[0008] Figure 4 A block diagram showing an example of a device anomaly detection module according to one embodiment is shown.

[0009] Figure 5 A block diagram of a network for an autonomous vehicle is shown according to one embodiment.

[0010] Figure 6 A block diagram illustrating an example interface for an anomaly signature profile according to one embodiment.

[0011] 7A to 7E is an example of a point cloud data anomaly according to some embodiments.

[0012] Figure 8A flow chart showing an example of a method for detecting anomalies in point cloud data according to one embodiment. DETAILED DESCRIPTION

[0013] Various embodiments and aspects of the present disclosure will be described with reference to the following details, and the accompanying drawings will illustrate various embodiments. The following description and drawings are illustrative and should not be construed as limiting the present disclosure. In order to thoroughly understand the various embodiments of the present disclosure, many specific details are described. However, in some cases, in order to briefly discuss the embodiments of the present disclosure, well-known or conventional details are not described.

[0014] The "one embodiment" or "an embodiment" mentioned in the specification means that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present disclosure. The "in one embodiment" appearing in different places in the specification does not necessarily refer to the same embodiment.

[0015] According to some embodiments, cloud server resources are utilized to process raw sensor data to detect anomalies. Anomalies can be analyzed on a timeline for equipment fault detection. Embodiments utilize the scalability of the cloud's computing resources to process raw sensor data. In some embodiments, raw sensor data is analyzed locally on the autonomous vehicle (ADV), and the analysis results are streamed to the cloud server. These analysis results enable operators to visualize changes in anomalies over time and determine the severity of anomalies based on different types of anomalies. In some embodiments, the cloud server automatically identifies situations where anomalies are above a severity threshold and issues a warning signal to the ADV operator.

[0016] Early fault detection is possible for lidar devices because lidar device failures occur gradually. For example, a lidar device may occasionally miss scans at small angles at first. After a few days or weeks, the lidar device begins to miss scans at larger angles more frequently.

[0017] Current state-of-the-art systems deploy hardware fault detection mechanisms on the vehicle for real-time detection. The advantage of real-time detection is that hardware fault detection can be performed immediately. The disadvantage is that the fault detection mechanism shares power and processing resources with the vehicle's autonomous driving system. In addition, due to limited processing resources, the on-board detection algorithm is also limited and cannot cover all types of anomalies.

[0018] According to one embodiment, a system obtains first point cloud data generated by a first scanning device of a first autonomous vehicle (ADV). The system determines an anomaly of the first point cloud data by applying one or more anomaly detection algorithms to the first point cloud data. The system determines an anomaly type of the anomaly. The system determines, based on the anomaly type, that the anomaly is above a severity threshold. In response to determining that the anomaly is above the severity threshold, the system issues an alert to check for any hardware failure of the first scanning device.

[0019] Figure 1 FIG. 1 is a block diagram showing an autonomous driving network configuration according to an embodiment of the present disclosure. Figure 1 , the network configuration 100 includes an autonomous driving vehicle (ADV) 101, which can be communicatively connected to one or more servers 103~104 via a network 102. One ADV is shown in the figure, but multiple ADVs can be connected to each other and / or to servers 103~104 via network 102. Network 102 can be any type of wired or wireless network, such as a local area network (LAN), a wide area network (WAN) such as the Internet, a cellular network, a satellite network, or a combination thereof (wired or wireless). Servers 103~104 can be any type of server or server cluster, such as a Web or cloud server, an application server, a back-end server, or a combination thereof. Servers 103~104 can be data analysis servers, content servers, traffic information servers, map and point of interest (MPOI) servers, or location servers, etc.

[0020] An ADV refers to a vehicle that can be configured for an automated mode, in which the vehicle requires little or no input from the driver to navigate an environment. Such an ADV may include a sensor system having one or more sensors configured to detect information related to the environment in which the vehicle operates. The vehicle and its associated controller use the detected information to navigate the environment. The ADV 101 can operate in a manual mode, a fully automated mode, or a partially automated mode.

[0021] In one embodiment, ADV 101 includes, but is not limited to, an autonomous driving system (ADS) 110, a vehicle control system 111, a wireless communication system 112, a user interface system 113, and a sensor system 115. ADV 101 may also include certain common components included in an ordinary vehicle, such as an engine, wheels, a steering wheel, a transmission, etc., which may be controlled by the vehicle control system 111 and / or ADS 110 using various communication signals and / or commands, such as acceleration signals or commands, deceleration signals or commands, steering signals or commands, braking signals or commands, etc.

[0022] The components 110-115 may be communicatively connected to each other via an interconnect, a bus, a network, or a combination thereof. For example, the components 110-115 may be communicatively connected to each other via a controller area network (CAN) bus. The CAN bus is a vehicle bus standard designed to allow microcontrollers and devices to communicate with each other in applications without a host computer. The CAN bus is a message-based protocol originally designed for multi-way electrical wiring within automobiles, but is also used in many other situations.

[0023] See also Figure 2 In one embodiment, the sensor system 115 includes, but is not limited to, one or more cameras 211, a global positioning system (GPS) unit 212, an inertial measurement unit (IMU) 213, a radar unit 214, and a laser radar (LIDAR) unit 215. The GPS unit 212 may include a transceiver for providing information about the position of the ADV. The IMU unit 213 may sense the position and orientation changes of the ADV based on inertial acceleration. The radar unit 214 may represent a system that senses objects within the local environment of the ADV using radio signals. In some embodiments, in addition to sensing objects, the radar unit 214 may also sense the speed and / or heading of the object. The laser radar unit 215 may sense objects in the environment where the ADV is located using lasers. The laser radar unit 215 may include one or more laser sources, a laser scanner, and one or more detectors and other system components. The camera 211 may include one or more devices for capturing images of the environment surrounding the ADV. The camera 211 may be a still camera and / or a video camera. The camera may be mechanically moved by, for example, mounting the camera on a rotating and / or tilting platform.

[0024] The sensor system 115 may also include other sensors, such as a sonar sensor, an infrared sensor, a steering sensor, a throttle sensor, a brake sensor, and an audio sensor (e.g., a microphone). The audio sensor may be used to capture sounds from the surrounding environment of the ADV. The steering sensor may be used to sense the steering angle of the steering wheel, the wheels, or a combination thereof. The throttle sensor and the brake sensor may sense the throttle position and the brake position of the vehicle, respectively. In some cases, the throttle sensor and the brake sensor may be integrated into an integrated throttle / brake sensor.

[0025] In one embodiment, the vehicle control system 111 includes but is not limited to a steering unit 201, a throttle unit 202 (also referred to as an acceleration unit), and a brake unit 203. The steering unit 201 is used to adjust the direction or heading of the vehicle. The throttle unit 202 is used to control the speed of the motor or engine, thereby controlling the speed and acceleration of the vehicle. The brake unit 203 slows down the speed of the wheels or tires of the vehicle by providing friction, thereby decelerating the vehicle. It should be noted that Figure 2 The components shown in may be implemented as hardware, software, or a combination thereof.

[0026] See again Figure 1 , the wireless communication system 112 allows the ADV 101 to communicate with external systems, such as devices, sensors, and other vehicles. For example, the wireless communication system 112 can communicate wirelessly with one or more devices directly, or with one or more devices through a communication network such as servers 103~104 on the network 102. The wireless communication system 112 can use any cellular communication network or wireless local area network (Wireless Local Area Network, WLAN), such as WiFi, to communicate with another component or system. The wireless communication system 112 can communicate directly with devices (e.g., mobile devices, display devices, speakers of passengers in the vehicle 101) using, for example, infrared links, Bluetooth, etc. The user interface system 113 can be part of a peripheral device implemented in the vehicle 101, including, for example, a keyboard, a touch screen display device, a microphone, and a speaker.

[0027] Some or all of the functions of ADV 101 may be controlled or managed by ADS 110, especially when operating in an autonomous driving mode. ADS 110 includes necessary hardware (e.g., processor, memory, storage) and software (e.g., operating system, planning and routing programs) to receive information from sensor system 115, control system 111, wireless communication system 112, and / or user interface system 113, process the received information, plan a route or path from a starting point to a destination, and then drive vehicle 101 based on the planning and control information. Alternatively, ADS 110 may be integrated with vehicle control system 111.

[0028] For example, a user as a passenger may specify a starting location and a destination of a trip, for example, through a user interface. ADS 110 obtains trip-related data. For example, ADS 110 may obtain location and route data from an MPOI server, which may be part of servers 103-104. The location server provides location services, and the MPOI server provides map services and points of interest at some locations. Alternatively, these locations and MPOI information may be cached locally in a persistent storage device of ADS 110.

[0029] When ADV 101 moves along the route, ADS 110 can also obtain real-time traffic information from a traffic information system or server (TIS). It should be noted that the servers 103~104 can be operated by a third-party entity. Alternatively, the functions of servers 103~104 can be integrated into ADS 110. Based on real-time traffic information, MPOI information, location information, and real-time local environmental data (such as obstacles, objects, nearby vehicles) detected or sensed by sensor system 115, ADS 110 can plan the best route and, for example, drive vehicle 101 according to the planned route through control system 111 to reach the designated destination safely and efficiently.

[0030] The server 103 may be a data analysis system that performs data analysis services for various clients. In one embodiment, the data analysis system 103 includes a data collector 121 and a machine learning engine 122. The data collector 121 collects driving statistics 123 from various vehicles such as ADS or ordinary vehicles driven by drivers. The driving statistics 123 include the following information: driving commands issued at different times (e.g., throttle commands, brake commands, steering commands) and vehicle responses captured by vehicle sensors (e.g., speed, acceleration, deceleration, direction). The driving statistics 123 may also include information describing the driving environment at different times, such as routes (including starting and target locations), MPOI, road conditions, weather conditions, etc.

[0031] The machine learning engine 122 generates or trains a set of rules, algorithms, and / or predictive models 124 based on driving statistics data 123 for various purposes. In one embodiment, the algorithm / model 124 may include a scanning algorithm or a machine learning model for detecting various types of lidar data anomalies. In one embodiment, the equipment anomaly detection module 125 processes the raw lidar data using the model / algorithm 124. In one embodiment, the algorithm 124 includes configuration of severity thresholds for each type of lidar data anomaly to alert the operator.

[0032] In one embodiment, the algorithm / model 124 can be streamed to the ADV to perform anomaly detection in real time. In one embodiment, the device anomaly detection module 308 of the ADV 101 processes the raw LiDAR data using the model / algorithm 124. The device anomaly detection module 125 or 308 will Figure 4 Further details in.

[0033] Figure 3A and Figure 3B FIG. 3 is a block diagram showing an example of an automated driving system for use with an ADV according to one embodiment. The system 300 may be implemented as Figure 1 A portion of ADV 101 includes, but is not limited to, ADS 110, control system 111, and sensor system 115. Figure 3A and Figure 3B The ADS 110 includes but is not limited to a positioning module 301 , a perception module 302 , a prediction module 303 , a decision module 304 , a planning module 305 , a control module 306 , a route selection module 307 and an equipment anomaly detection module 308 .

[0034] Some or all of the modules 301 to 308 may be implemented in software, hardware, or a combination thereof. For example, these modules may be installed in the persistent storage device 352, loaded into the memory 351, and executed by one or more processors (not shown). It should be noted that some or all of these modules may be implemented in conjunction with Figure 2 Some or all modules of the vehicle control system 111 are communicatively connected or integrated together. Some modules in modules 301-308 can be integrated together to form an integrated module.

[0035] The positioning module 301 determines the current location of the ADV 101 (e.g., using the GPS unit 212) and manages any data related to the user's itinerary or route. The positioning module 301 (also known as the map and route module) manages any data related to the user's itinerary or route. The user can log in, for example, through a user interface, and specify the starting location and destination of the itinerary. The positioning module 301 communicates with other components of the ADV 101, such as the map and route data 311, to obtain itinerary related data. For example, the positioning module 301 can obtain location and route data from a location server and an MPOI server. The location server provides location services, and the MPOI server provides map services and POIs of some locations, which can be cached as part of the map and route data 311. When the ADV 101 moves along the route, the positioning module 301 can also obtain real-time traffic information from a traffic information system or server.

[0036] Based on the sensor data provided by the sensor system 115 and the positioning information obtained by the positioning module 301, the perception module 302 determines the perception information of the surrounding environment. The perception information may represent the information that an average driver perceives about the surrounding environment of the vehicle he is driving. The perception information may include, for example, lane configuration in the form of objects, traffic light signals, relative positions of other vehicles, pedestrians, buildings, crosswalks or other traffic-related signs (e.g., stop signs, yield signs), etc. The lane configuration includes information describing one or more lanes, such as lane shape (e.g., straight or curved), lane width, number of lanes on the road, one-way or two-way lanes, merging or splitting lanes, exit lanes, etc.

[0037] The perception module 302 may include a computer vision system or functionality of a computer vision system to process and analyze images captured by one or more cameras to identify objects and / or features in the ADV environment. Objects may include traffic signals, road boundaries, other vehicles, pedestrians, and / or obstacles, etc. The computer vision system may use object recognition algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system may map the environment, track objects, and estimate the speed of objects, etc. The perception module 302 may also detect objects based on other sensor data provided by other sensors such as radar and / or lidar.

[0038] For each object, the prediction module 303 predicts the behavior of the object in a specific situation. The prediction is performed based on the perception data, which combines a set of map / route information 311 and traffic rules 312 to perceive the driving environment at a certain moment. For example, if the object is a vehicle traveling in the opposite direction and the current driving environment includes an intersection, the prediction module 303 will predict whether the vehicle is likely to go straight or turn. If the perception data indicates that there is no traffic light at the intersection, the prediction module 303 can predict that the vehicle may have to come to a complete stop before entering the intersection. If the perception data indicates that the vehicle is currently in a left-turn lane or a right-turn lane, the prediction module 303 will predict that the vehicle will be more likely to turn left or right, respectively.

[0039] For each object, decision module 304 makes a decision on how to respond to the object. For example, for a particular object (e.g., another vehicle in an intersecting path) and its metadata describing the object (e.g., speed, direction, turn angle), decision module 304 decides how to respond to the object (e.g., overtake, yield, stop, pass). Decision module 304 can make such decisions based on a set of rules such as traffic rules or driving rules 312, which can be stored in persistent storage device 352.

[0040] The route selection module 307 is configured to provide one or more routes or paths from the starting point to the destination. For a given trip from the starting location to the destination location, for example, received from a user, the route selection module 307 obtains the route and map information 311 and determines all possible routes or paths from the starting location to the destination location. The route selection module 307 can generate a reference line in the form of a topographic map for each route it determines from the starting location to the destination location. The reference line is an ideal route or path that is not interfered by other vehicles, obstacles, or traffic conditions. That is, if there are no other vehicles, pedestrians, or obstacles on the road, the ADV should follow the reference line completely or approximately. Thereafter, the topographic map is provided to the decision module 304 and / or the planning module 305. The decision module 304 and / or the planning module 305 examine all possible routes to select and modify one of the optimal routes based on other data provided by other modules, such as traffic conditions from the positioning module 301, the driving environment perceived by the perception module 302, and the traffic conditions predicted by the prediction module 303. The actual path or route of controlling the ADV may be close to or different from the reference line provided by the route selection module 307, depending on the specific driving environment at the time.

[0041] Based on the decision for each perceived object, the planning module 305 plans the path or route of the ADV and driving parameters (such as distance, speed and / or turning angle) using the reference line provided by the route selection module 307 as a basis. That is, for a given object, the decision module 304 decides what to do with the object, and the planning module 305 determines how to do it. For example, for a given object, the decision module 304 may decide to pass through the object, and the planning module 305 may determine whether to pass from the left or right side of the object. The planning module 305 generates planning and control data, including information describing how the vehicle 101 moves in the next movement cycle (such as the next route / path segment). For example, the planning and control data may instruct the vehicle 101 to move 10 meters at a speed of 30 miles per hour (mph), and then change to the right lane at a speed of 25 miles per hour.

[0042] Based on the planning and control data, the control module 306 controls and drives the ADV according to the route or path defined by the planning and control data by sending appropriate commands or signals to the vehicle control system 111. The planning and control data includes sufficient information to drive the vehicle from a first point on the route or path to a second point along the route or path using appropriate vehicle settings or driving parameters (e.g., throttle, brake, steering commands) at different times.

[0043] In one embodiment, the planning stage is performed in multiple planning cycles, which are also called driving cycles, for example, every 100 milliseconds. For each planning cycle or driving cycle, one or more control commands will be issued based on the planning and control data. That is, every 100 milliseconds, the planning module 305 plans the next route segment or path segment, for example, including the target position and the time required for the ADV to reach the target position. Alternatively, the planning module 305 can also specify a specific speed, direction and / or steering angle, etc. In one embodiment, the planning module 305 plans the route segment or path segment for the next predetermined time period of, for example, 5 seconds. For each planning cycle, the planning module 305 plans the target position of the current cycle (for example, the next 5 seconds) based on the target position planned in the previous cycle. Afterwards, the control module 306 generates one or more control commands (for example, throttle, brake, steering control commands) based on the planning and control data of the current cycle.

[0044] It should be noted that the decision module 304 and the planning module 305 can be integrated into an integrated module. The decision module 304 / planning module 305 can include a navigation system or the functionality of a navigation system to determine the driving path of the ADV. For example, the navigation system can determine a series of speeds and directional headings so that the ADV generally proceeds along a lane-based path to the final destination while its driving path substantially avoids perceived obstacles. The destination can be set based on user input via the user interface system 113. The navigation system can dynamically update the driving path while the ADV is running. The navigation system can combine data from a GPS system and one or more maps to determine the driving path of the ADV.

[0045] The device anomaly detection module 308 can process the raw LiDAR data to detect anomalies, thereby obtaining an overview of the raw LiDAR data. In some embodiments, the raw LiDAR data is processed when the processing resources of the ADV 101 are idle or when the ADV is parked or stationary. Once the processing is complete, the overview or analysis results can be uploaded to the server 103 for further analysis.

[0046] Figure 4 FIG. 4 is a block diagram showing an example of a device abnormality detection module according to an embodiment. Figure 1 Device anomaly detection module 125 or Figure 3AThe device anomaly detection module 308. The device anomaly detection module 400 can detect anomalies in the raw laser radar point cloud data and analyze these anomalies on the timeline. In one embodiment, the device anomaly detection module 400 includes a point cloud acquisition submodule 401, an anomaly detection submodule 403, an anomaly type determination submodule 405, a severity determination submodule 407, and a warning submodule 409. The point cloud acquisition submodule 401 can acquire raw point cloud data from the laser radar device. The point cloud data may include one or more frames of points in a three-dimensional (3D) space. The 3D point can represent the distance to the obstacle, and the distance is detected by the incident light signal reflecting back to the light detection sensor of the laser radar device after a certain delay. In one embodiment, the point cloud data captures the 360-degree environment around the ADV (rotating laser radar). In one embodiment, the point cloud captures a 60-180 degree viewing angle in front, behind, or to the side of the ADV (flash laser radar). The anomaly detection submodule 403 can apply a scanning algorithm to the point cloud data to detect anomalies in the data. The anomaly type determination submodule 405 can classify the detected anomalies into specific types. Some examples of abnormality types include noisy point clouds, small missing point cloud data (e.g., angles less than 10 degrees), large missing point cloud data (e.g., angles greater than 90 degrees), discontinuous scan lines, etc. The severity determination submodule 407 can evaluate the severity of the abnormality. The severity can be evaluated by a severity threshold for each abnormality type. If the severity of the abnormality is higher than the threshold, the warning submodule 409 can issue a warning signal. It should be noted that some modules in modules 401-409 can be integrated together as an integrated module.

[0047] Figure 5 FIG. 5 shows a block diagram of a network 500 of autonomous driving vehicles (ADVs) 101A-101D according to one embodiment. Figure 5 As shown, in one embodiment, ADV 101A~101D can upload raw LiDAR data to server 103 for processing. Raw LiDAR data can be uploaded from ADV 101A~101D to server 103 via a WIFI network, a cellular network, or manually uploaded by an operator via a hot-swappable mobile hard disk. In one embodiment, the upload can be scheduled to be performed regularly, such as every day, every hour, etc. In one embodiment, a scanning algorithm can be applied to the raw LiDAR data by the corresponding ADV, and the processing results can be uploaded to server 103 regularly.

[0048] When the raw lidar data is uploaded to the server 103, the server 103 may apply one or more algorithms to the lidar data to identify anomalies in the lidar data, and further classify the anomalies into one or more anomaly types. Examples of anomaly types include a small portion of data missing (e.g., less than 10 degrees, or less than 1 / 8 of the data in the frame is missing), a large portion of data missing (e.g., greater than 10 degrees, or more than 1 / 8 of the data in the frame is missing), noisy data, missing ground information, discontinuous scan lines, etc. In some embodiments, the algorithm may correspond to a machine learning model trained to detect different types of anomalies. The machine learning model may include a trainable deep learning convolutional neural network model, a recurrent neural network model, a long short-term memory network, a gated recurrent unit, etc. The model may be supervised to learn 1) to identify anomalies and / or 2) to classify anomalies into specific types in a list of anomaly types. In one embodiment, the scanning algorithm may include a pattern recognition algorithm configured to identify recognizable patterns for each anomaly type. When the raw lidar data is scanned to detect anomalies, the raw lidar data within a time period may be analyzed, and a status indication may be given for each data frame within the time period (in Figure 6 Once the analysis is complete, the server 103 may process the profile to issue an early warning signal, and / or an operator may view the profile on a timeline via an interface such as Figure 6 shown.

[0049] Figure 6A block diagram of an example interface 600 for an abnormal feature profile 602 according to one embodiment is shown. An abnormal feature profile (or profile) can describe the health of raw lidar data captured over a period of time. Profile 602 can be drawn in interface 600, where each row can represent a status indication of a lidar device and each column represents a status indication within a day. For example, interface 600 includes status indications of raw lidar data of lidar devices 601A~601K over three days. The status within the time period can be indicated by multiple indicator bars of different colors / hues. The status indicator bars can include the following examples: processed+normal 605, processed+abnormal (and its type) 603~604, unprocessed 609, or unavailable 607. Processed+normal indicator 605 can correspond to a raw lidar data frame that has been scanned by an anomaly algorithm and no anomalies are found. Processed+abnormal indicators 603~604 can correspond to raw lidar data frames that have been scanned by an anomaly algorithm and at least one anomaly is found. Indicator 603 may correspond to a Class 1 anomaly (e.g., a small portion of data is missing), while indicator 604 may correspond to a Class 2 anomaly (e.g., the data is noisy). Unprocessed indicator 609 may indicate a raw lidar data frame that has not yet been processed by any anomaly algorithm. Unavailable indicator 607 may indicate that there is no raw lidar data available for processing during this time period.

[0050] In one embodiment, the server 103 (one or more cloud server nodes) can determine the type of anomaly for each detected anomaly. The server can then determine a severity threshold associated with the detected anomaly based on the determined anomaly type. In one embodiment, the severity threshold at which a small portion of the lidar data is considered severe is two or more indications within three days. In one embodiment, the severity threshold for a large portion of the lidar data being missing is one indication. If the severity is above the threshold, the server 103 can send an early warning signal to alert the operator, or the server 103 can send a signal to the ADV to alert the operator of the ADV to check for a fault in the lidar equipment. In one embodiment, if an anomaly is observed to appear and then disappear on two or more ADVs within the same time period, the server 103 can identify the anomaly as a systemic anomaly (e.g., weather-related) and ignore the anomaly.

[0051] like Figure 6As shown, profile 602 contains three days of raw lidar data for lidar devices 601A~601K corresponding to ADV 101A~101K. Among them, lidar device 601A can be installed on ADV 101A, lidar device 601B can be installed on ADV 101B, and so on. For lidar device 601A, server 103 observed two independent data frames with abnormal indications 603 (e.g., a small portion of data is missing) on ​​August 8, 2022 and August 10, 2022. The detection module 125 of server 103 can perform anomaly identification and classification by applying one or more scanning models / algorithms to the raw lidar data. Here, the severity threshold of Class 1 anomalies is two or more occurrences within three days. Since there are two anomalies 603 within three days, the indication 603 on August 10 is marked as severe, and the server 103 can send an early warning signal to ADV 101A, alerting the operator of ADV 101A to check the lidar device 601A. The warning signal can be an instruction to play a visual indication or a sound prompt in the user interface of ADV 101A.

[0052] See also Figure 6 , the laser radar device 601B has a Class 2 anomaly (e.g., most of the data is missing) on ​​August 9, 2022. Here, the severity threshold of the Class 2 anomaly is one occurrence. When it is determined that the Class 2 anomaly occurs and reaches the severity threshold, the server 103 sends an early warning signal to the ADV 101B, alerting the operator to check for potential hardware failures in the laser radar device 601B.

[0053] See also Figure 6 , the laser radar devices 601A~601B, 601E~601G and 601I observed abnormal indication 611 at approximately the same time on August 10, 2022. Since two or more ADVs observed indication 611 at the same time, the server 103 can classify indication 611 as a systematic indication (e.g., weather-related). In some embodiments, the server 103 uses the location of the corresponding ADV as a condition for the systematic indication. For example, if the distance between the ADVs is less than a threshold distance (e.g., 10 kilometers), the indications that appear at the same time from the laser radar devices of these ADVs are systematic indications. In some embodiments, the server 103 uses the weather conditions (e.g., rain, fog, snow, etc.) at the driving location of the corresponding ADV as a condition for the systematic indication.

[0054] In addition, the laser radar device 601D has no laser radar data available for processing. The laser radar device 601K has available data but has not yet processed it. The other laser radar devices 601C, 601E~601J have processed the signals and found no abnormalities. Therefore, no warning signal is sent to ADV 101C, 101E~101J.

[0055] In one embodiment, the operator can view the overview 602 on the server 103 or through a client accessing the server 103 to manually identify the anomaly. In one embodiment, the overview 602 can be generated by the device anomaly detection module 125 of the server 103, or can be generated by the device anomaly detection module of any of the ADVs 101A~101K (e.g., the device anomaly detection module 308). It should be noted that the examples of the severity threshold being two occurrences within three days and the severity threshold being one occurrence are shown for illustrative purposes only, and the server 103 can configure any value as the severity threshold corresponding to each anomaly type. Although Figure 6 One lidar device is shown associated with one ADV, but each ADV can be associated with multiple lidar devices.

[0056] 7A to 7E is an example of point cloud data anomaly according to one embodiment. 7A to 7E As shown, point cloud frames 700 , 710 , 720 , 730 and 740 show different anomaly types of a 360-degree field of view rotating laser radar device. For each point cloud frame, ADV is located at the center position 703 .

[0057] In one embodiment, the anomaly included in frame 700 corresponds to a small missing portion 701. Here, the angle of the missing portion is about 2 degrees and reaches the leftmost edge of the point cloud data, indicating that there is an anomaly in the lidar data. In some cases, the anomaly of the missing angle may be related to dirt or bird droppings on the lidar device. In some cases, the missing angle may be related to a problem with the processing circuit of the lidar device. In some cases, the data is missing due to weather (e.g., rain, snow, etc.).

[0058] Referring to frame 710, anomaly 711 may correspond to a large portion of data missing. In one embodiment, a large portion of data missing means that the angle of potential data missing is greater than 10 degrees, or the data missing in frame 710 exceeds 1 / 8. Referring to frame 720, another anomaly 721 may correspond to noisy data. For example, noisy data in the lidar data may result in the inability to identify the lidar scan line. Referring to frame 730, another anomaly 731 may correspond to a large portion of ground plane data missing, where the ADV expects to observe the road surface at the ground plane of the ADV. Referring to frame 740, another anomaly 741 may correspond to discontinuity of point cloud data, for example, the scan line discontinuity in the frame reaches a threshold of lidar scan lines (e.g., greater than 50%). The cause of the scan line discontinuity may be a hardware failure. Only five types of anomaly are shown for rotating lidar devices, but the algorithm 124 included in the server 103 can detect other types of anomaly. In addition, anomaly detection can be associated with other types of ranging devices (e.g., flash lidar, time of flight (TOF), radar, etc.).

[0059] Figure 8 800 is a flowchart showing an example of a method for detecting anomalies in point cloud data according to an embodiment. The method 800 may be performed by processing logic, which may include software, hardware, or a combination thereof. For example, Figure 4 The device abnormality detection module 400 executes method 800.

[0060] At block 801 , processing logic acquires first point cloud data generated by a first scanning device of a first autonomous vehicle (ADV).

[0061] At block 803 , processing logic determines anomalies of the first point cloud data by applying one or more anomaly detection algorithms 124 to the first point cloud data.

[0062] At block 805 , processing logic determines the exception type of the exception.

[0063] At block 807 , processing logic determines, based on the exception type, whether the exception is above a severity threshold.

[0064] In one embodiment, the processing logic further determines a severity threshold based on the anomaly type. Different severity thresholds may be pre-configured for each anomaly type and the configurations may be stored in the server 103 .

[0065] At block 809 , in response to determining that the anomaly is above the severity threshold, processing logic issues an alert to check for any hardware failures of the first scanning device.

[0066] In one embodiment, processing logic obtains second point cloud data generated by a second scanning device of a second ADV. Processing logic applies one or more anomaly detection algorithms to the first point cloud data and the second point cloud data to determine one or more anomalies in the first point cloud data and the second point cloud data. Processing logic determines a situation where the anomaly occurs in both the first point cloud data and the second point cloud data. In response to determining that the anomaly occurs in both the first point cloud data and the second point cloud data, processing logic indicates that the anomaly corresponds to a common event.

[0067] For example, if the anomaly is a systemic anomaly (e.g., weather-related), then ignore the anomaly.

[0068] In one embodiment, the first point cloud data is uploaded to a cloud server node, and the cloud server node analyzes the first point cloud data.

[0069] In one embodiment, the abnormality type is one of a plurality of abnormality types, and the plurality of abnormality types include: a small portion of the point cloud data is missing, a large portion of the point cloud data is missing, the point cloud data is noisy, or the point cloud data is discontinuous.

[0070] In one embodiment, the scanning device includes a lidar device, which is a rotating lidar device or a flash lidar device.

[0071] In one embodiment, the processing logic further acquires first point cloud data generated by the first scanning device within a predetermined time period, wherein the situation where the anomaly is higher than the severity threshold includes: the severity of the anomaly increases within the predetermined time period.

[0072] In one embodiment, processing logic provides a user interface to an operator to review point cloud data corresponding to one or more ADVs over a predetermined period of time.

[0073] It should be noted that some or all of the components shown and described above may be implemented in software, hardware, or a combination thereof. For example, these components may be implemented as software installed and stored in a persistent storage device, which may be loaded by a processor (not shown) and executed in a memory to perform the methods or operations described in this application. Alternatively, these components may be implemented as executable code that is programmed or embedded in dedicated hardware, such as an integrated circuit (e.g., a dedicated IC or ASIC), a digital signal processor (DSP), or a field programmable gate array (FPGA), which may be accessed from an application program via a corresponding driver and / or operating system. In addition, these components may be implemented as specific hardware logic in a processor or processor core as part of an instruction set accessible to a software component via one or more specific instructions.

[0074] Some portions of the foregoing detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally is conceived to be a self-consistent sequence of operations leading to a desired result. These operations are those requiring physical manipulation of physical quantities.

[0075] It should be remembered, however, that all of these terms and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless expressly noted in the above discussion, it should be understood that throughout the specification, discussions using terms such as those set forth in the appended claims refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities within the computer system registers and memories and converts that data into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage devices, transmission devices, or display devices.

[0076] Embodiments of the present disclosure also relate to apparatus for performing the operations herein. Such a computer program is stored in a non-transitory computer-readable medium. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, a machine-readable (e.g., a computer-readable) medium includes a machine (e.g., a computer) readable storage medium (e.g., a read-only memory (ROM), a random access memory (RAM), a disk storage medium, an optical storage medium, a flash memory device).

[0077] The processes or methods depicted in the foregoing figures may be performed by processing logic, which may include hardware (e.g., circuits, dedicated logic, etc.), software (e.g., embodied on a non-transitory computer-readable medium), or a combination of both. The processes or methods are described above in terms of some sequential operations, but it should be understood that some of the operations described may be performed in a different order. In addition, some operations may be performed in parallel rather than sequentially.

[0078] The embodiments of the present disclosure are not described with reference to any particular programming language. It should be appreciated that a variety of programming languages ​​can be used to implement the teachings of the embodiments of the present disclosure described herein.

[0079] In the foregoing description, embodiments of the present disclosure have been described with reference to specific exemplary embodiments of the present disclosure. Obviously, various modifications may be made to the present disclosure without departing from the broader spirit and scope of the present disclosure as described in the appended claims. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A computer-implemented method comprising: Acquire first point cloud data generated by a first scanning device of a first autonomous driving vehicle; determining anomalies of the first point cloud data by applying one or more anomaly detection algorithms to the first point cloud data; determining an exception type of the exception; Based on the type of the anomaly, determining that the anomaly is above a severity threshold; as well as In response to determining that the anomaly is above the severity threshold, an alert is issued to check for any hardware failures of the first scanning device.

2. The method according to claim 1, further comprising: Acquire second point cloud data generated by a second scanning device of a second autonomous driving vehicle; applying one or more anomaly detection algorithms to the first point cloud data and the second point cloud data to determine one or more anomalies in the first point cloud data and the second point cloud data; Determining that the first point cloud data and the second point cloud data both have the abnormality; as well as In response to determining that the anomaly occurs in both the first point cloud data and the second point cloud data, it is indicated that the anomaly corresponds to a common event.

3. The method according to claim 1, wherein: The first point cloud data is uploaded to a cloud server node, and the cloud server node analyzes the first point cloud data.

4. The method according to claim 1, wherein: The abnormality type is one of a plurality of abnormality types, and the plurality of abnormality types include: a small portion of point cloud data is missing, a large portion of point cloud data is missing, point cloud data has noise, or point cloud data is discontinuous.

5. The method according to claim 1, wherein: The scanning device includes a laser radar device, which is a rotating laser radar device or a flash laser radar device.

6. The method according to claim 1, further comprising: The first point cloud data generated by the first scanning device within a predetermined time period is acquired, wherein the situation in which the anomaly is higher than the severity threshold comprises: the severity of the anomaly increases within the predetermined time period.

7. The method according to claim 1, further comprising: An operator is provided with a user interface to examine point cloud data corresponding to one or more autonomous vehicles over a predetermined period of time.

8. A non-transitory machine-readable medium having stored therein instructions, which when executed by a processor cause the processor to perform operations comprising: Acquire first point cloud data generated by a first scanning device of a first autonomous driving vehicle; determining anomalies of the first point cloud data by applying one or more anomaly detection algorithms to the first point cloud data; determining an exception type of the exception; Based on the type of the anomaly, determining that the anomaly is above a severity threshold; as well as In response to determining that the anomaly is above the severity threshold, an alert is issued to check for any hardware failures of the first scanning device.

9. The machine-readable medium of claim 8, wherein: The operations include: Acquire second point cloud data generated by a second scanning device of a second autonomous driving vehicle; applying one or more anomaly detection algorithms to the first point cloud data and the second point cloud data to determine one or more anomalies in the first point cloud data and the second point cloud data; Determining that the first point cloud data and the second point cloud data both have the abnormality; and In response to determining that the anomaly occurs in both the first point cloud data and the second point cloud data, it is indicated that the anomaly corresponds to a common event.

10. The machine-readable medium of claim 8, wherein: The first point cloud data is uploaded to a cloud server node, and the cloud server node analyzes the first point cloud data.

11. The machine-readable medium of claim 8, wherein: The abnormality type is one of a plurality of abnormality types, and the plurality of abnormality types include: a small portion of point cloud data is missing, a large portion of point cloud data is missing, point cloud data has noise, or point cloud data is discontinuous.

12. The machine-readable medium of claim 8, wherein: The scanning device includes a laser radar device, which is a rotating laser radar device or a flash laser radar device.

13. The machine-readable medium of claim 8, wherein: The operation further includes: acquiring the first point cloud data generated by the first scanning device within a predetermined time period, wherein the situation where the anomaly is higher than the severity threshold includes: the severity of the anomaly increases within the predetermined time period.

14. The machine-readable medium of claim 8, wherein: The operations also include providing a user interface to an operator to review point cloud data corresponding to one or more autonomous vehicles within a predetermined time period.

15. A data processing system comprising: processor; as well as A memory connected to the processor, the memory being used to store instructions, wherein when the instructions are executed by the processor, the processor performs operations, including: Acquire first point cloud data generated by a first scanning device of a first autonomous driving vehicle; determining anomalies of the first point cloud data by applying one or more anomaly detection algorithms to the first point cloud data; determining an exception type of the exception; Based on the anomaly type, determining that the anomaly is above a severity threshold; and In response to determining that the anomaly is above the severity threshold, an alert is issued to check for any hardware failures of the first scanning device.

16. The system of claim 15, wherein: The operations also include: Acquire second point cloud data generated by a second scanning device of a second autonomous driving vehicle; applying one or more anomaly detection algorithms to the first point cloud data and the second point cloud data to determine one or more anomalies in the first point cloud data and the second point cloud data; Determining that the first point cloud data and the second point cloud data both have the abnormality; and In response to determining that the anomaly occurs in both the first point cloud data and the second point cloud data, it is indicated that the anomaly corresponds to a common event.

17. The system of claim 15, wherein: The first point cloud data is uploaded to a cloud server node, and the cloud server node analyzes the first point cloud data.

18. The system of claim 15, wherein: The abnormality type is one of a plurality of abnormality types, and the plurality of abnormality types include: a small portion of point cloud data is missing, a large portion of point cloud data is missing, point cloud data has noise, or point cloud data is discontinuous.

19. The system of claim 15, wherein: The scanning device includes a laser radar device, which is a rotating laser radar device or a flash laser radar device.

20. The system of claim 15, wherein: The operation further includes: acquiring the first point cloud data generated by the first scanning device within a predetermined time period, wherein the situation where the anomaly is higher than the severity threshold includes: the severity of the anomaly increases within the predetermined time period.

Images