Systems and methods for monitoring the functionality of a LiDAR sensor in uncontrolled environments

Abstract

A functional monitoring system for a vehicle's light-based detection and distance (LiDAR) sensor comprises a LiDAR sensor configured to generate light pulses and receive feedback. A location identification module is configured to identify locations visited by the vehicle, select those locations as frequently visited, generate metrics for the frequently visited locations based on feedback from the LiDAR sensor, and, based on these metrics, select a further group of frequently visited locations to monitor the LiDAR sensor's health. A functional monitoring module is configured to generate functional metrics for the LiDAR sensor when the vehicle is at the selected locations.

Description

introduction

[0001] The information provided in this section serves the purpose of presenting the general context of the disclosure. The work of the inventors mentioned herein, insofar as it is described in this section, as well as aspects of the description that cannot be classified as prior art at the time of filing, are neither expressly nor implicitly recognized as prior art with respect to the present disclosure.

[0002] The present disclosure relates to driver assistance systems and in particular to driver assistance systems that include sensors for light-based detection and distance measurement (LiDAR).

[0003] Vehicles feature varying levels of driver assistance (such as fully or partially autonomous vehicles). Autonomous vehicles often rely on light-based detection and distance measurement (LiDAR) systems to detect and avoid objects in the vehicle's path. The functionality or health of the LiDAR sensor is typically evaluated by the manufacturer using a dedicated test setup that provides a controlled environment and targets with known reflectivity. However, once the vehicle is sold, this type of test environment is not readily available to assess the health of the LiDAR sensor. Summary

[0004] A functional monitoring system for a vehicle's light-based detection and distance (LiDAR) sensor comprises a LiDAR sensor configured to generate light pulses and receive feedback. A location identification module is configured to identify locations visited by the vehicle, select those locations as frequently visited, generate metrics for the frequently visited locations based on feedback from the LiDAR sensor, and, based on these metrics, select a further group of frequently visited locations to monitor the LiDAR sensor's health. A functional monitoring module is configured to generate functional metrics for the LiDAR sensor when the vehicle is at the selected locations.

[0005] In other features, one or more metrics are selected from a group consisting of a signal-to-noise ratio, a number of feedback points, and a variation or deviation in reflectivity. The location identification module is configured to identify the selected locations from among the frequently visited locations by assigning usability ratings to the frequently visited locations based on the metrics, ranking the frequently visited locations based on the usability ratings, and selecting the chosen locations based on the ranking.

[0006] In other features, the functionality monitoring module is configured to select a visit to each of the chosen locations as corresponding reference data. The functionality monitoring module is configured to apply release criteria to each visit to the selected locations. The release criteria are selected from a group consisting of the vehicle's location, temperature, light intensity, time of day, the location of a reference object, and combinations thereof.

[0007] In other features, the functionality monitoring module is configured to combine metrics from complementary data from the selected locations. The functionality monitoring module is configured to apply a function to the functionality metrics for each of the selected locations to generate mature functionality metrics. The functionality monitoring module is configured to fuse one or more of the mature functionality metrics for two or more of the selected locations to generate a fused metric. The functionality monitoring module is configured to compare the fused metric to a predetermined threshold to evaluate the health of the LiDAR sensor.

[0008] A method for evaluating the health of a light-based detection and distance measurement (LiDAR) sensor includes generating light pulses and receiving feedback using a vehicle's LiDAR sensor, identifying locations visited by the vehicle, identifying selected locations as frequently visited, generating metrics for the frequently visited locations based on feedback from the LiDAR sensor, selecting, based on these metrics, selected locations from the frequently visited as locations to monitor the health of the LiDAR sensor, and generating operational metrics for the LiDAR sensor when the vehicle is at the selected locations.

[0009] In other characteristics, the procedure involves selecting metrics from a group consisting of a signal-to-noise ratio, a number of feedback points, and a reflectivity deviation. Selecting the most frequently visited locations involves assigning usability ratings to these locations based on the metrics, ranking the frequently visited locations based on the usability ratings, and selecting the chosen locations based on the ranking.

[0010] In other features, the procedure includes selecting one visit to each of the chosen locations as reference data. The procedure includes applying release criteria to each of the visits to the selected locations. The procedure includes selecting the release criteria from a group consisting of vehicle location, temperature, light intensity, time of day, the location of a reference object, and combinations thereof. The procedure includes combining metrics from complementary values ​​of the selected locations. The procedure includes applying a function to the capability metrics for each of the selected locations to generate mature capability metrics.

[0011] In other features, the procedure includes merging metrics for two or more of the selected locations to create a merged metric and comparing the merged metric to a predetermined threshold to assess the condition of the LiDAR sensor.

[0012] A vehicle includes a light-based detection and distance measurement (LiDAR) sensor configured to generate light pulses and receive feedback, and a global positioning system (GPS). A functional assessment module, which communicates with the LiDAR sensor and the GPS, includes a location identification module configured to identify locations visited by the vehicle, identify selected locations as frequently visited, generate metrics for the frequently visited locations based on feedback from the LiDAR sensor, and, based on these metrics, select specific locations from the frequently visited list to monitor the health of the LiDAR sensor.The functionality assessment module includes a functionality monitoring module configured to generate functionality metrics for the LiDAR sensor when the vehicle is at the selected locations.

[0013] Further applications of the present disclosure will become apparent from the detailed description, the claims, and the drawings. The detailed description and specific examples serve only for illustrative purposes and are not intended to limit the scope of the disclosure. Brief description of the drawings

[0014] The present disclosure is more fully understood from the detailed description and the accompanying drawings, whereby: Fig. 1 a functional block diagram of an example of a vehicle that includes a functional assessment module configured to evaluate the health of a light-based detection and distance measurement (LiDAR) sensor according to the present disclosure; Fig. 2 a flowchart of an example of a method for selecting frequently visited locations for evaluating the functionality of the LiDAR sensor and assessing the health of the LiDAR sensor at the selected locations according to the present disclosure; Fig. 3A to 3C illustrate examples of the first, second and third frequently visited locations of the vehicle according to the present disclosure; Fig. 4A to 4C illustrate examples of variations encountered during different visits to the same frequently visited location, according to the present disclosure; Fig. 5A to 5C illustrate an example of the selection or rejection of the first, second and third frequently visited locations as selected locations for assessing the health of the LiDAR sensor according to the present disclosure; Fig. 6 is a flowchart of an example of a procedure for identifying frequently visited locations according to the present disclosure; Fig. 7A is a flowchart of an example of a procedure for selecting suitable locations for a LiDAR functionality assessment according to the present disclosure; Fig. 7B is a flowchart of an example of a procedure for assigning usability ratings in accordance with the present disclosure; Fig. 7C is a flowchart of an example of a method for classifying frequently visited locations suitable for evaluating the functionality of the LiDAR sensor according to the present disclosure; Fig. 7D is a flowchart of an example of a method for storing properties, performance indicators and an orientation for each of the selected sites according to the present disclosure; Fig. 8 a flowchart of an example of a procedure for eliminating variations or deviations over multiple visits according to the present disclosure; and Fig. 9 is a flowchart of an example of a method for monitoring the health and functional deterioration of the LiDAR sensor according to the present disclosure.

[0015] Reference numbers can be reused in the drawings to identify similar and / or identical elements. Detailed description

[0016] Systems and methods according to the present disclosure monitor the functionality, health, and deterioration of one or more light-based detection and distance measurement (LiDAR) sensors. In some examples, the LiDAR sensor(s) is / are part of a system for autonomous driving of a vehicle.

[0017] The functional monitoring systems and procedures identify locations frequently visited by the vehicle and select one or more of these locations to evaluate the health of the LiDAR sensor in order to detect any deterioration. The systems identify, classify, and group frequently visited locations. Some of these locations are selected and regularly used for evaluating the health of the LiDAR sensor. The reference locations are far removed from controlled environments typically used by a manufacturer or service facility to evaluate the LiDAR sensor.

[0018] If the vehicle is in the same location and orientation, the feedback data from the LiDAR sensor (e.g., after adjusting for changes due to environmental factors, etc.) should be approximately the same. One visit to each of the selected locations is designated as the reference data. Data from other visits to the selected locations are compared to the reference data. Differences in the feedback data during the other visits are used to evaluate the degradation of the LiDAR sensor's functionality.

[0019] Referring now to Fig. In this example, a vehicle 100 has a driver assistance controller 110 that includes a functional assessment module 112. A global positioning (GPS) / compass system 120 determines the position, path, and / or orientation of the vehicle 100 and outputs the GPS / compass data to the driver assistance controller 110. A radar system 122 optionally generates radio-frequency pulses and outputs radar feedback signals to the driver assistance controller 110. In some examples, the driver assistance controller 110 includes an autonomous driving module 160 that is configured to operate the vehicle 100 fully and / or partially in an autonomous driving mode by controlling vehicle controls 164 (such as a steering wheel, brake pedal, accelerator pedal, etc.) based on outputs from a LiDAR sensor 124 and / or other sensors.

[0020] The LiDAR sensor 124 generates light pulses in the vehicle's path and receives feedback signals. In some examples, the LiDAR sensor includes one or more lasers 130 and one or more scanners 128 that probe or scan the one or more lasers 130 in the vehicle's path or in the driver's field of vision. The capability assessment module 112 includes a location identification module 142, which is configured to identify frequently visited locations, select one or more of the locations as selected capability assessment locations, and perform other functions described herein. A capability deterioration module 144 is configured to initiate an inspection at the selected locations and assess the health and / or capability deterioration of the LiDAR sensor 124.

[0021] Referring now to Fig. Figure 2 describes a procedure for monitoring the health of the LiDAR sensor 124. In Figure 210, locations frequently visited by the vehicle are identified. In Figure 214, one or more frequently visited locations are selected for a LiDAR functionality assessment. In Figure 218, compensation for deviations (such as different vehicle orientation, position, environmental factors, etc.) is performed across multiple visits to the frequently visited locations. In Figure 222, the LiDAR health is assessed, and any deterioration of the LiDAR sensor's health over time is monitored.

[0022] Referring now to Fig. Sections 3A to 5C present examples of identifying and selecting frequently visited locations. Fig. 3A A vehicle location, such as an operator's driveway, is evaluated as a possible location for a functionality assessment of the LiDAR sensor. Fig. 3B evaluates a vehicle location, such as a country road, as a possible location for the LiDAR sensor. Fig. 3C evaluates a vehicle location, such as an office building or the operator's workplace, as a possible location for the LiDAR sensor.

[0023] In Fig. 4A to 4C, when the same location is visited, the data generated by the LiDAR sensor 124 may vary or differ due to different conditions at the selected location (e.g., orientation and position of the vehicle in relation to the objects being detected, environmental factors, etc.). Fig. 4A is a vehicle location, such as an operator's entrance, a selected location, and feedback data is collected. During this visit, the vehicle's orientation relative to the garage is approximately perpendicular. During the visit in Fig. 4B describes the alignment of the vehicle at a first offset angle relative to the garage. During the visit to Fig. 4C specifies the vehicle's orientation at a second offset angle relative to the garage. In some examples, the functionality assessment module 112 performs a transformation on the feedback from the LiDAR sensor 124 to reduce deviations due to the vehicle's orientation. Similar compensation can be provided for environmental factors.

[0024] In Fig. In steps 5A to 5C, one or more locations are selected for LiDAR functionality assessment. Fig. 5A selects a vehicle location such as the operator's driveway because it is frequently visited and provides good feedback data for analyzing the health of the LiDAR sensor 124. Fig. In 5B, a vehicle location such as a country road is not selected because it does not provide good feedback data for analyzing the health of the LiDAR sensor 124. Fig. 5C selects a vehicle location such as an office building or the operator's workplace because it is frequently visited and provides good feedback data for analyzing the health of the LiDAR sensor 124.

[0025] Referring now to Fig. 6. A procedure for identifying frequently visited locations is described in step 210 of Fig. Figure 2 is shown. At step 310, GPS data is used to record locations visited by the vehicle. At step 314, the locations are monitored over time to identify frequently visited locations. Frequently visited locations include, for example, home, workplace, charging stations, grocery stores, daycare centers, etc. At step 318, the feedback data from the LiDAR sensor 124 for the frequently visited locations is saved.

[0026] Referring now to Fig. Sections 7A to 7D describe a procedure for selecting locations suitable for the LiDAR functionality assessment in step 214 of Fig. 2 are suitable. At 340 in Fig. 7A assigns usability ratings to each site based on feedback data collected over time. 344 ranks the sites and groups them into complementary locations. 348 stores characteristics, functionality indicators, and / or optimal positions for each selected site.

[0027] In Fig. 7B are steps for assigning usability ratings in step 340 of Fig. Figure 7A illustrates this. At Figure 350, point cloud data from the LiDAR sensor 124 at each of the frequently visited locations are used to calculate functionality metrics. Examples of functionality metrics include the signal-to-noise ratio (SNR), the number of feedback points, reflectivity variation, and / or other metrics. At Figure 354, the frequently visited locations are evaluated for a LiDAR functionality assessment using the generated metrics. At Figure 358, the usability ratings are determined based on the calculated metrics at each location.

[0028] In Fig. 7C are steps for classifying and grouping in step 344 of Fig. Figure 7A illustrates this. At 400, the most frequently visited locations are ranked based on the usability rating. Some of the frequently visited locations can be selected to measure a subset of the metrics (based on the quality of the feedback data), but cannot be used for at least one of the metrics. Other frequently visited locations can be selected to measure all of the metrics and / or another subset of the metrics.

[0029] In 404, a forward selection process is used to determine whether two or more sites complement each other and can be merged or combined. In other words, the functionality assessment module can combine data from two or more of the frequently visited sites for a given metric. In some examples, the forward selection criterion includes an overall usability assessment for the combined sites using all or a subset of the metrics. In 408, a list of selected sites is generated for functionality monitoring.

[0030] In Fig. 7D are steps for saving properties, functionality indicators, and an optimal position for each of the selected sites in step 348 of Fig. Figure 7A illustrates this. At 410, feedback data, metrics, and / or assessments based on them are collected and stored for the selected locations. At 414, the data is used to identify an optimal position (according to the vehicle's position and orientation at the selected location) for a functional assessment. At 418, one visit to each of the selected locations is chosen as the reference data for each location. The reference data is used as a baseline for comparison with subsequent visits. The differences in metrics over time are used to assess health and functional decline.

[0031] In Fig. 8 are steps to eliminate discrepancies across multiple visits in step 218 of Fig. Figure 2 illustrates this. In Figure 460, release criteria are applied to subsequent visits to increase the accuracy of the operational readiness assessment. Examples of release criteria include environmental criteria such as GPS location (is the vehicle in a suitable location and / or orientation) and / or detection of a reference object. Other examples of release criteria include temperature, time of day, illuminance, reflectivity, etc. In Figure 464, the data are adjusted for variations across multiple visits to the same location. In some examples, a transformation is applied to the feedback data to adjust for different vehicle positions or orientations relative to objects at the locations. In some examples, adjustment is made for intensity measurements.

[0032] In Fig. 9 are steps to monitor the health and deterioration of the LiDAR sensor in step 222 of Fig. Figure 2 illustrates this. At 480, one or more performance indicators are calculated and stored after the vehicle has visited one of the selected locations. Examples of performance indicators include angular resolution, field of view (FOV), frame rate, and average intensity. At 484, the calculations are repeated during multiple visits to the same location. The calculations from multiple visits to each of the selected locations are aggregated using a function such as the average, median, or other mathematical and / or logical functions.

[0033] In 486, the functional status indicators across all selected sites are optionally merged (or combined using one or more mathematical or logical functions) and stored. The mature functional status indicators are merged (or combined using one or more mathematical or logical functions) across all selected sites.

[0034] At 488, the operational status of the LiDAR sensor 124 is determined. The fused operational status indicators are monitored over time. In some examples, one or more fixed or adaptive thresholds are applied to the fused operational status indicators to evaluate the operational status of the LiDAR sensor 124. For example, the operational status can be declared adequate if the fused operational status indicators are greater than (or less than) one of several corresponding predetermined thresholds. Conversely, the operational status can be declared inadequate if the fused operational status indicators are less than (or greater than) one of several corresponding predetermined thresholds.

[0035] The preceding description is merely illustrative and intended to limit the scope of the revelation, its application, or uses. The comprehensive doctrine of revelation can be implemented in a multitude of forms. Therefore, although this revelation contains particular examples, the true scope of the revelation should not be so limited, since other modifications will become apparent upon study of the drawings, the description, and the following claims. It should be understood that one or more steps within a process may be carried out in a different order (or simultaneously) without altering the principles of the present revelation.Furthermore, although each of the embodiments described above is characterized by certain features, one or more of these features described in relation to any embodiment of the disclosure may be implemented in one of the other embodiments and / or combined with features of one of the other embodiments, even if this combination is not expressly described. In other words, the described embodiments are not mutually exclusive, and permutations of one or more embodiments with each other remain within the scope of this disclosure.

[0036] Spatial and functional relationships between elements (for example, between modules, circuit elements, semiconductor layers, etc.) are described using various terms, including "connected," "interlocked," "coupled," "adjacent," "near or beside," "on," "above," "below," and "arranged." Unless explicitly described as "direct," when a relationship between first and second elements is described in the above disclosure, this relationship may be a direct relationship, in which no other intervening elements exist between the first and second elements, or it may be an indirect relationship, in which one or more intervening elements (either spatial or functional) exist between the first and second elements.As used here, the phrase “at least one of A, B and C” should be understood as meaning a logical (A OR B OR C) using a non-exclusive logical OR, and should not be understood as meaning “at least one of A, at least one of B and at least one of C”.

[0037] In the diagrams, the direction of an arrow, as indicated by its tip, generally illustrates the flow of information (e.g., data or instructions) that is relevant to the illustration. For example, if Element A and Element B exchange a variety of information, but information transferred from Element A to Element B is important for the illustration, the arrow may point from Element A to Element B. This unidirectional arrow does not imply that no other information is transferred from Element B to Element A. Furthermore, Element B may send requests for, or acknowledgments of, information to Element A in connection with the information transferred from Element A to Element B.

[0038] In this application, including the definitions below, the term "module" or the term "controller" may be replaced by the term "circuit". The term "module" may refer to, be part of, or include an application-specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field-programmable gate array (FPGA); a processor circuit (shared, dedicated, or group) that executes code; a memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the described functionality; or a combination of some or all of the above components, such as in a system-on-a-chip.

[0039] The module may contain one or more interface circuits. In some examples, the interface circuits may include wired or wireless interfaces connected to a local area network (LAN), the internet, a wide area network (WAN), or combinations thereof. The functionality of any given module of this disclosure may be distributed among multiple modules connected via interface circuits. For example, multiple modules may enable load balancing. In another example, a server module (also known as a remote or cloud module) may perform some functions for a client module.

[0040] The term "code," as used above, can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, data structures, and / or objects. The term "shared processor circuit" refers to a single processor circuit that executes some or all of the code from multiple modules. The term "group processor circuit" refers to a processor circuit that, in combination with additional processor circuits, executes some or all of the code from one or more modules. References to multiple processor circuits include multiple processor circuits on separate chips, multiple processor circuits on a single chip, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or a combination of the above.The term shared memory circuit refers to a single memory circuit that stores some or all of the code from multiple modules. The term group memory circuit refers to a memory circuit that, in combination with additional memory, stores some or all of the code from one or more modules.

[0041] The term storage circuit is a subset of the term computer-readable medium. The term computer-readable medium, as used here, does not include transitory electrical or electromagnetic signals that propagate through a medium (such as on a carrier wave); the term computer-readable medium can therefore be considered material and non-transient.Non-restrictive examples of a non-transient, physical, computer-readable medium include non-volatile memory circuits (such as a flash memory circuit, a erasable programmable read-only memory circuit, or a mask read-only memory circuit), volatile memory circuits (such as a static random access memory circuit or a dynamic random access memory circuit), magnetic storage media (such as an analog or digital magnetic tape or a hard disk drive), and optical storage media (such as a CD, a DVD, or a Blu-ray Disc).

[0042] The devices and methods described in this application can be partially or fully implemented by means of a special-purpose computer created by configuring a general-purpose computer to perform one or more specific functions embodied in computer programs. The functional blocks, flowchart components, and other elements described above serve as software specifications that can be translated into computer programs through the routine work of a person skilled in the art or a programmer.

[0043] Computer programs contain instructions executable by processors, stored on at least one non-transient, physical, computer-readable medium. Computer programs may also contain or rely on stored data. Computer programs may include a basic input / output system (BIOS) that interacts with the computer's special-purpose hardware, device drivers that interact with specific devices of the computer for special purposes, one or more operating systems, user applications, background services, background applications, etc.

[0044] The computer programs can contain: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation); (ii) assembly code; (iii) object code generated from source code by a compiler; (iv) source code for execution by an interpreter; (v) source code for compilation and execution by a just-in-time compiler; etc. For example, source code can be written using syntax from languages ​​such as C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, JavaScript®, HTML5 (Hypertext Markup Language 5th Revision), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, and Visual Basic®. Include Lua, MATLAB, SIMULINK and Python®.

Claims

[1] Functionality monitoring system for a vehicle light-based detection and distance measurement (LiDAR) sensor, comprising: a LiDAR sensor configured to generate light pulses and receive feedback; a location identification module configured to: identify locations visited by the vehicle; selected locations identified as frequently visited locations; Metrics for frequently visited locations are generated based on feedback from the LiDAR sensor; and, based on these metrics, selected frequently visited locations are chosen as locations to monitor the health of the LiDAR sensor; and a functionality monitoring module configured to generate functionality metrics for the LiDAR sensor when the vehicle is at the selected locations. [2] Functionality monitoring system according to claim 1, wherein one or more of the metrics are selected from a group consisting of a signal-to-noise ratio, a number of feedback points and a reflectivity deviation. [3] Functionality monitoring system according to claim 1, wherein the location identification module is configured to select the frequently visited locations as selected locations by: assigns usability ratings to frequently visited locations based on metrics; ranks the most frequently visited locations based on usability ratings; and selects the chosen locations based on the classification. [4] Functionality monitoring system according to claim 1, wherein the functionality monitoring module is configured to select a visit to each of the selected locations as the corresponding reference data. [5] Functionality monitoring system according to claim 1, wherein the functionality monitoring module is configured to apply release criteria for each of the visits to the selected locations. [6] Functionality monitoring system according to claim 5, wherein the release criteria are selected from a group consisting of the vehicle location, the temperature, the light intensity, the time of day, the location of a reference object and combinations thereof. [7] Functionality monitoring system according to claim 1, wherein the functionality monitoring module is configured to combine metrics from complementary of the selected sites. [8] Functionality monitoring system according to claim 1, wherein the functionality monitoring module is configured to apply a function to the functionality metrics for each of the selected sites to generate mature functionality metrics. [9] Functionality monitoring system according to claim 8, wherein the functionality monitoring module is configured to fuse one or more of the mature functionality metrics for two or more of the selected sites to produce a fused metric. [10] Functionality monitoring system according to claim 9, wherein the functionality monitoring module is configured to compare the fused metric with a predetermined threshold to evaluate the state of the LiDAR sensor.

Images