Dynamic and variable learning by determining and using most trustworthy inputs
Patent Information
- Application Number
- JP2020171118
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2019-10-25
- Filing Date
- 2020-10-09
- Publication Date
- 2024-11-14
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing vehicle trajectory prediction systems rely on unreliable sensor inputs, leading to reduced reliability of warning systems due to unfavorable trust or lack of trust in certain sensor data, particularly when ambient conditions affect sensor performance.
A method and system for determining and utilizing the most reliable sensor inputs by assessing sensor confidence levels based on ambient conditions, using a trust estimator to adjust confidence in subsystem outputs accordingly.
Enhances the reliability of vehicle trajectory prediction by dynamically adjusting confidence levels based on ambient conditions, ensuring accurate and trustworthy sensor input utilization.
Abstract
Description
Technical Field
[0001] Certain aspects of the present disclosure generally relate to vehicle notifications, and more particularly, to systems and methods for dynamic and variable learning by determining and using the most reliable inputs.
Background Art
[0002] A system / subsystem predictor (e.g., a trajectory predictor) may trust inputs from devices (e.g., sensors and / or cameras) that should not be trusted by the predictor, or may not trust inputs from some devices that should be trusted by the predictor. For example, in a well-known vehicle collision, a trajectory predictor (or perhaps other subsystems) may trust sensor inputs from a camera and ignore sensor inputs from a lidar, even though the lidar should have provided the most reliable inputs in that situation. This undesirable trust or lack of trust reduces the reliability of the warning system.
Summary of the Invention
[0003] A method for providing dynamic and variable learning to an ego vehicle by determining and using the most reliable inputs is described. The method includes determining a reliability level of sensor values obtained from one or more ego vehicle sensors based on ambient conditions of the environment of the ego vehicle. The method also includes determining a credibility level in the accuracy of the output of a subsystem of the ego vehicle. The output of the subsystem is based on the sensor values. The credibility level is based on the reliability level of the sensor values.
[0004] A system is described that provides dynamic and variable learning to an ego vehicle by determining and using the most reliable input. The system has memory and one or more processors coupled to that memory. The processor is configured to determine the reliability level of sensor values obtained from one or more ego vehicle sensors based on the ambient conditions of the ego vehicle's environment. The processor is also configured to determine the reliability level in accuracy of the outputs of the ego vehicle's subsystems. The subsystem outputs are based on sensor values. The reliability level is based on the reliability level of the sensor values.
[0005] A non-temporary computer-readable medium on which non-temporary program code is recorded is described. The program code provides dynamic and variable learning to the ego vehicle by determining and using the most reliable inputs. The program code is executed by a processor and includes program code that determines the reliability level of sensor values obtained from one or more ego vehicle sensors based on the surrounding environment of the ego vehicle. The program code also includes program code that determines the reliability level in accuracy of the outputs of the ego vehicle subsystems. The subsystem outputs are based on sensor values. The reliability level is based on the reliability level of the sensor values.
[0006] This broad outlines the features and technical advantages of the disclosure so that the subsequent detailed descriptions may be better understood. Additional features and advantages of the disclosure are described below. Those skilled in the art should recognize that the disclosure can be readily used as a basis for modifying or designing other structures to perform the same purposes as the disclosure. Those skilled in the art should also recognize that such equivalent structures will not deviate from the teachings of the disclosure as described in the accompanying claims. Novel features of the disclosure that are considered to be features of its structure and method of operation, along with further purposes and advantages, will be better understood from the following description when considered in conjunction with the accompanying drawings. However, it should be clearly understood that each feature is provided for illustrative and descriptive purposes only and is not intended to define any limitation of the disclosure. [Brief explanation of the drawing]
[0007] The features, properties, and advantages of this disclosure will become more apparent from the detailed description below, when similar reference letters are linked to drawings that identify their corresponding counterparts through the drawings.
[0008] [Figure 1A] This document shows examples of vehicles in the environment relating to the aspects of this disclosure. [Figure 1B] This is a top view of an example of a road, relating to one aspect of the disclosure, which includes a host vehicle with an external environment and surrounding vehicles located within it. [Figure 1C] This is a top view of an example of a road, relating to one aspect of the disclosure, which includes a host vehicle with an external environment and surrounding vehicles located within it. [Figure 2] Figures 1B and 1C are schematic block diagrams of an example of a host vehicle configured to predict the trajectory of surrounding vehicles or a host vehicle, relating to one aspect of the disclosure. [Figure 3] This figure shows an example of a hardware implementation for a dynamic and variable learning system according to the aspects of this disclosure. [Figure 4] This disclosure illustrates a method for providing dynamic and variable learning of an ego vehicle by determining and using the most reliable input. [Modes for carrying out the invention]
[0009] The detailed descriptions provided below, in conjunction with the accompanying drawings, are intended to describe various configurations and are not intended to represent only configurations that can implement the concepts described herein. The detailed descriptions include certain details for the purpose of providing a complete understanding of the various concepts. However, it will be obvious to those skilled in the art that these concepts can be implemented without these specific details. In some examples, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.
[0010] In recent years, various attempts have been made to realize the automated operation of moving objects such as four-wheeled vehicles. To achieve automated operation of moving objects, it is crucial to accurately detect objects such as vehicles, pedestrians, and obstacles around the moving object and to avoid hazards while driving based on the detection results. Of these two factors, object detection technology using various sensors and radars is known as a technology for accurately detecting surrounding objects.
[0011] A system / subsystem predictor (e.g., a trajectory predictor) may trust input from devices it should not trust (e.g., sensor systems including sensors and / or cameras), or it may fail to trust input from some devices it should trust. For example, in a well-known vehicle collision, the trajectory predictor (or perhaps another subsystem) trusted sensor input from a camera and ignored sensor input from lidar, even though lidar would have provided the most reliable input in that situation. This undesirable trust or lack of trust reduces the reliability of the warning system.
[0012] Aspects of this disclosure are directed to an ego vehicle configured to determine the reliability of sensor values obtained from one or more sensors, cameras, or other devices based on the surrounding conditions determined by the ego vehicle. With respect to subsystem outputs generated based on the obtained sensor values, the ego vehicle can determine the reliability in accuracy of the outputs generated by the subsystem, which is based (at least in part) on the reliability of the obtained sensor values. The reliability in accuracy of the subsystem outputs can be separate from the reliability in fidelity of the model (for example, separate from the reliability in accuracy of a given prediction generated by a trajectory predictor).
[0013] As the surrounding conditions change, the Ego vehicle can update the reliability of the sensor values, and therefore update the reliability of the output accuracy of a given subsystem.
[0014] In one embodiment, the Ego vehicle obtains sensor values from one or more sensors. For example, the Ego vehicle can be equipped with a variety of sensors, and the Ego vehicle can obtain sensor values from one or more of these sensors. The sensors may include, among many other possibilities, LiDAR, radar, cameras, photoelectric sensors, or microphones.
[0015] The Ego Vehicle determines one or more of its surrounding conditions. For example, the Ego Vehicle can determine whether it is on a highway, in a congested area, inside a tunnel, at an intersection, or about to change lanes. The Ego Vehicle can also determine cloud cover, the current time of day, and whether it is daytime or nighttime. The Ego Vehicle can determine any combination of these or other surrounding conditions.
[0016] A vehicle's sensor system may also include ambient condition sensors, which provide sensor signals or ambient condition data associated with the surrounding environment, such as external temperature, precipitation (like rain or snow), road conditions (e.g., road roughness, or the presence of water or ice).
[0017] The EGO vehicle determines the reliability of the obtained sensor values based on the determined surrounding conditions. For example, if the EGO vehicle is in a congested or crowded area, sensor values obtained from radar are relatively unreliable because, for example, the radar may be jammed and / or the sensor values may contain an unacceptable amount of noise. On the other hand, in such situations, sensor values obtained from a RIDER mounted on the roof rack are relatively reliable. However, if the EGO vehicle is traveling on a highway, sensor values obtained from radar are relatively reliable.
[0018] As another example, sensor values obtained from a camera are relatively unreliable in low-light conditions (resulting in a black image) or high-light conditions (resulting in a wash-out white image).
[0019] Reliability can be determined by a trust estimator trained based on pre-collected driving data. For example, the driving data may show the predicted trajectory for the vehicle and the actual trajectory taken by the vehicle. The trajectory can be predicted based on sensor values from one or more sensors that are the same as (or similar to) the ego vehicle and obtained under the same (or similar) surrounding conditions. The trust estimator can be adjusted based on the calculated difference between the predicted trajectory and the actual trajectory. Based on this adjustment, the trust estimator can determine the reliability of sensor values obtained from one or more sensors under the current conditions. For example, reliability can be the difference between the predicted and actual occurrences of events in all modes, or the probability of an event occurring considering the prediction. Modes may include trajectory position, the position of other vehicles, etc.
[0020] In one embodiment, the credibility estimator generates a credibility level for the output of an ego vehicle subsystem. The credibility level indicates the credibility of the subsystem's output in terms of accuracy. The subsystem's output is based on obtained sensor values, and the credibility level is generated based (at least partially) on the reliability of the obtained sensor values.
[0021] For example, the Ego vehicle can provide sensor values to a trajectory predictor, which then generates a predicted trajectory for the Ego vehicle (or another vehicle) based on those sensor values. The Ego vehicle may determine that the obtained sensor values are relatively unreliable, and therefore the credibility estimator can generate a low credibility level for the predicted trajectory. Thus, even if the trajectory predictor predicts that the Ego vehicle is highly likely to take a given trajectory, the Ego vehicle may decide not to pass this predicted trajectory to a different subsystem due to the low credibility of the highly probable trajectory.
[0022] In one embodiment, the trajectory predictor generates a confidence level separate from the confidence level generated by the confidence estimator. Similar to the confidence level generated by the confidence estimator, the confidence level generated by the trajectory predictor indicates the confidence in the accuracy of the predicted trajectory (i.e., output) generated by the trajectory predictor (i.e., subsystem). However, unlike the confidence level generated by the confidence estimator, the confidence level generated by the trajectory predictor does not separately consider the reliability of the obtained sensor values. By considering the surrounding circumstances, the Ego vehicle determines not only the fidelity of the model (e.g., the trajectory predictor) but also the reliability of the entire subsystem output.
[0023] Figure 1A shows an example of a vehicle 100 (ego or host vehicle) in an environment 150 according to an aspect of the present disclosure. In this example, vehicle 100 is an autonomous vehicle. As shown in Figure 1A, vehicle 100 can travel on road 110. The first vehicle 104 is in front of vehicle 100, and the second vehicle 116 can be adjacent to the ego vehicle 100. In this example, vehicle 100 can include a 2D camera 108, such as a 2D RGB camera, and a lidar sensor 106. Other sensors, such as radar and / or ultrasonic, are also conceivable. Additionally or alternatively, vehicle 100 can include one or more additional 2D cameras and / or lidar sensors. For example, the additional sensors can be sensors facing the side and / or the rear.
[0024] In one configuration, the 2D camera 108 captures a 2D image that includes objects in the field of view 114 of the 2D camera 108. The lidar sensor 106 can generate one or more output streams. The first output stream can include a 3D point cloud of objects in a first field of view, such as a 360° field of view 112 (e.g., a bird's-eye view). The second output stream 124 can include a 3D point cloud of objects in a second field of view, such as a forward-facing field of view.
[0025] The 2D image captured by the 2D camera includes a 2D image of the first vehicle 104 because the first vehicle 104 is in the field of view 114 of the 2D camera 108. As is known to those skilled in the art, the lidar sensor 106 uses laser light to sense the shape, size, and position of objects in the environment. The lidar sensor 106 can scan the environment in the vertical and horizontal directions. In this example, the artificial neural network (e.g., an autonomous driving system) of vehicle 100 can extract height and / or depth features from the first output stream. The autonomous driving system of vehicle 100 can also extract height and / or depth features from the second output stream.
[0026] The information obtained from sensors 106, 108 can be used to evaluate the driving environment. For example, the information obtained from sensors 106, 108 can be used to identify objects in locations not visible from vehicle 100. This information can be used to generate one or more location-specific notifications.
[0027] Referring now to FIGS. 1B and 1C, a top view of an example of a road is shown that includes a host vehicle 100 having an external environment 12 and surrounding vehicles 14 located therein. The system can include the host vehicle 100. The host vehicle 100 can be of any suitable type of vehicle. The host vehicle 100 can be of any shape of a powered transportation vehicle, at least in part. In one or more arrangements, the host vehicle 100 can be an automobile. Although the arrangements are described herein with respect to automobiles, it should be understood that the examples and implementations are not limited to automobiles. In one or more arrangements, the host vehicle 100 can be a ship, an aircraft, a spacecraft, a golf cart, a motorcycle, and / or any other shape of a powered transportation vehicle, at least in part.
[0028] The host vehicle 100 can be located anywhere. For example, the host vehicle 100 can travel along a road. The host vehicle 100 can have an associated external environment 12. The external environment 12 can be the area surrounding the host vehicle 100 and / or any part thereof. One or more objects can be located in the external environment 12 of the host vehicle 100. For example, one or more objects can be vehicles surrounding the host vehicle 100, also traveling along a road. The external environment 12 of the host vehicle 100 may have any number of surrounding vehicles 14. Figures 1B and 1C show only one surrounding vehicle 14, but it will be understood that the arrangement described herein is not limited in this respect. In practice, two or more surrounding vehicles 14 can be located in the external environment 12 of the host vehicle 100. Additionally or alternatively, there may be no surrounding vehicles in the external environment 12 of the host vehicle 100.
[0029] As described herein, the host vehicle 100 can monitor its external environment 12. The host vehicle 100 can detect objects located in the external environment 12. The host vehicle 100 can classify these objects as surrounding vehicles 14. The host vehicle 100 can predict the trajectory 16 of the surrounding vehicles 14. The predicted trajectory 16 can be a set of actions, directions, orientations, and / or travel paths relative to the surrounding vehicles 14. As will be considered below, the predicted trajectory 16 can be determined based on one or more predetermined vehicle characteristics and one or more current driving characteristics relative to the surrounding vehicles 14. The predetermined vehicle characteristics can be based on the type of surrounding vehicles 14. The current driving characteristics can be obtained from sensor data of the host vehicle 100's sensor system.
[0030] Referring here to Figure 2, a schematic diagram of an example of a host vehicle 100 configured to predict the trajectory (e.g., trajectory 16) of the host vehicle 100 or surrounding vehicle 14 shown in Figures 1B and 1C is shown. The host vehicle 100 can include various elements. Some of the possible elements of the host vehicle 100 are shown and described in Figure 2. However, it should be understood that the host vehicle 100 does not have to include all of the elements shown in or described in Figure 2. The host vehicle 100 can have any combination of the various elements shown in Figure 2. Furthermore, the host vehicle 100 can have additional elements in addition to the elements shown in Figure 2. Furthermore, although the various elements are shown as being located inside the host vehicle 100 in Figure 2, it should be understood that one or more of these elements can be located outside the host vehicle 100. Furthermore, the elements shown can be physically separated by a large distance from each other.
[0031] The host vehicle 100 may include one or more processors 18. The processor 18 may be any component or group of components configured to execute any of the processes described herein, or any form of instructions that cause such processes to be executed. The processor 18 can be one or more general-purpose and / or special-purpose processors. Examples of suitable processors 18 include microprocessors, microcontrollers, digital signal processors, and other circuits capable of executing software. Further examples of suitable processors 18, but not limited to, include central processing units (CPUs), array processors, vector processors, digital signal processors (DSPs), field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), application-specific integrated circuits (ASICs), programmable logic circuits, and controllers. The processor 18 may include at least one hardware circuit (e.g., an integrated circuit) configured to execute instructions contained in program code. In an arrangement with multiple processors 18, such processors may operate independently of each other, or one or more processors may operate in combination with each other. In one or more configurations, the processor 18 may be the main processor of the host vehicle 100. For example, the processor 18 may be an electronic control unit (ECU).
[0032] The host vehicle 100 may include a computer-readable medium. In one or more configurations, the computer-readable medium may be memory 19. Memory 19 may include volatile / non-volatile memory. Examples of suitable memory 19 include RAM (random access memory), flash memory, ROM (read-only memory), PROM (programmable read-only memory), EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), registers, magnetic disks, optical disks, hard drives, or any other suitable storage medium, or any combination thereof. Memory 19 may contain instructions in the program code stored thereon. Such instructions can be executed by the processor 18 and / or one or more modules of the host vehicle 100. In one or more configurations, memory 19 may be a component of the processor 18. In one or more configurations, memory 19 may be functionally connected to the processor 18 and / or one or more modules of the host vehicle 100 for use by them. Functional connection can include direct or indirect connections, including connections that do not involve direct physical contact.
[0033] The host vehicle 100 may include one or more databases 20 for storing one or more types of data. The databases 20 may be components of the memory 19 and the processor 18, or the databases 20 may be functionally connected to them for use by the processor 18 and / or the memory 19.
[0034] The host vehicle 100 may also include a sensor system 22. The sensor system 22 may include one or more sensors. One or more sensors may be configured to monitor something in the external environment 12 (shown in Figures 1B and 1C) of the host vehicle 100. In one or more configurations, one or more sensors may be configured to monitor in real time. Real time can be a level of processing response that allows the user or system to sense quickly enough to perform a particular process or make a decision, and allows the processor to keep up with some external process without delay. The sensor system 22 may be located anywhere in or on the host vehicle 100. The sensor system 22 may, to show some possibilities, include at least in part existing systems of the host vehicle 100 such as backup sensors, lane keeping sensors, and / or front sensors.
[0035] In an arrangement where the sensor system 22 includes multiple sensors, the multiple sensors can be distributed around the host vehicle 100 in any suitable manner. The sensors can operate independently of each other or in combination. In such cases, two or more sensors can form a sensor network.
[0036] The sensor system 22 may include one or more sensors configured to sense the external environment 12 of the host vehicle 100, or a portion thereof. For example, the sensor system 22 may be configured to acquire data from at least the front portion and / or at least the rear portion of the external environment 12 of the host vehicle 100. For example, the sensor system 22 may monitor the front portion of the host vehicle 100 along longitudinal α, and / or the rear portion of the host vehicle 100 along longitudinal α.
[0037] Additionally or alternatively, the sensor system 22 may be configured to acquire data from at least the lateral portion of the external environment 12 of the host vehicle 100. The lateral portion may be, for example, the portion of the external environment 12 located between the front and rear portions of the host vehicle 100. For example, the sensor system 22 may be configured to monitor the left and / or right portions along the lateral direction β of the host vehicle 100.
[0038] The sensor system 22 can be functionally connected to the processor 18, database 20, memory 19, and / or any other components and / or modules of the host vehicle 100. Various examples of possible sensors of the sensor system 22 are described herein. However, it will be understood that the disclosure is not limited to the special sensors described. The sensor system 22 may include categories of sensors such as active sensors 24, passive sensors 26, and / or vehicle motion sensors 28.
[0039] The active sensor 24 can be any sensor or group of sensors configured to actively emit a signal for the purpose of determining the quantity or properties of some object. For example, the active sensor 24 can emit electromagnetic signals, radio signals, laser signals, etc. Such signals can be emitted into the external environment 12 of the host vehicle 100. Examples of such active sensors 24 include radar sensors 30 and lidar sensors 32.
[0040] In one or more configurations, the active sensor 24 may include one or more radar sensors 30. The radar sensor 30 can be any device, component and / or system that can detect something using at least in part radio signals. The radar sensor 30 can be configured to detect the presence of one or more objects in the external environment 12 of the host vehicle 100, the position of each detected object relative to the host vehicle 100, the distance between each detected object and the host vehicle 100 in one or more directions (e.g., longitudinal α, lateral β, and / or other directions), the altitude of each detected object, the velocity of each detected object, and / or the movement of each detected object. The radar sensor 30, or the data obtained therefrom, can determine, or can be used to determine, the velocity, position, and / or orientation of objects in the external environment 12 of the host vehicle 100. The radar sensor 30 may have three-dimensional coordinate data associated with the objects.
[0041] In one or more configurations, the active sensor 24 may include a lidar sensor 32. The lidar sensor 32 may be any device, component, and / or system that can detect an object using, at least in part, an electromagnetic signal. In one or more configurations, the electromagnetic signal may be a laser signal. The lidar sensor 32 may include a laser source and / or laser scanner configured to emit a laser signal, and a detector configured to detect reflections of the laser signal. The lidar sensor 32 may be configured to operate in coherent or non-coherent detection modes.
[0042] The lidar sensor 32 can be configured to detect the presence of one or more objects in the external environment of the host vehicle 100, the position of each detected object relative to the host vehicle 100, the distance between each detected object and the host vehicle 100 in one or more directions, the altitude of each detected object, the speed of each detected object, and / or the movement of each detected object. An example of a lidar sensor 32 is, for example, the Velodyne® lidar system.
[0043] The passive sensor 26 can be any sensor or group of sensors configured to receive signals delivered from something independent of the passive sensor 26. Such received signals can be delivered from something located in the external environment 12 of the host vehicle 100. Additionally or alternatively, such received signals can be delivered from any component or group of components of the host vehicle 100 independent of the passive sensor 26. For example, the passive sensor 26 can receive electromagnetic signals, radio signals, laser signals, etc. Examples of such passive sensors 26 include a positioning sensor 34, a camera 36, an IMU sensor 38, a CAN sensor 40, and the like.
[0044] In one or more configurations, the passive sensor 26 may include a positioning sensor 34. The positioning sensor 34 may be one or more components or a group of components configured to determine the geographical location of the host vehicle 100.
[0045] The positioning sensor 34 may include a global positioning system, a local positioning system, and / or a geolocation system. The positioning sensor 34 may include a transceiver configured to estimate the position of the host vehicle 100 relative to the Earth. For example, the positioning sensor 34 may include a GPS transceiver for determining the latitude, longitude, and / or altitude of the vehicle. The positioning sensor 34 may use other systems for determining the position of the host vehicle 100 (e.g., laser-based positioning systems, inertial-assisted GPS, and / or camera-based positioning). It should be understood that many different systems or components can replace or complement the positioning sensor 34 for determining the position of the host vehicle 100 without departing from the scope of this disclosure.
[0046] In one or more configurations, the passive sensor 26 may include one or more cameras. The camera 36 may be any device, component, and / or system capable of capturing visual data. The visual data may include video and / or image information / data. The visual data may be in any appropriate shape. In one or more configurations, the visual data may include thermal signs, thermal images, and / or thermal images of parts of the external environment 12 of the host vehicle 100.
[0047] Camera 36 can be any suitable type of camera. For example, camera 36 can be a high-resolution camera, a high dynamic range (HDR) camera, an infrared camera, and / or a thermal imaging camera.
[0048] In one or more configurations, the camera 36 can be positioned to capture visual data from at least a portion of the external environment 12 of the host vehicle 100. In one or more configurations, the camera 36 can be positioned to capture visual data from at least the rear portion of the external environment of the host vehicle 100. As a further example, the camera 36 can be positioned to acquire visual data from at least the left and / or right portion of the external environment of the host vehicle 100.
[0049] Camera 36 can be positioned on any suitable part of the host vehicle 100. For example, camera 36 can be positioned inside the host vehicle 100. One or more cameras 36 can be positioned outside the host vehicle 100. One or more cameras 36 can be positioned outside the host vehicle 100 or exposed to the outside. Additionally or alternatively, one or more cameras 36 can be positioned on the side of the host vehicle 100. As another example, one or more cameras 36 can be positioned on the roof of the host vehicle 100.
[0050] In one or more configurations, camera 36 may be one or more backup cameras. The backup cameras may be configured to acquire visual data of the rear portion of the external environment 12 of the host vehicle 100. In some configurations, one or more backup cameras may capture visual data of at least a portion of the rear portion of the external environment when the host vehicle 100 is not in reverse gear mode and / or is not moving backward. Alternatively or additionally, camera 36 may include other cameras that can be used in the host vehicle 100.
[0051] In one or more configurations, the passive sensor 26 may include an inertial measuring unit (IMU) sensor 38. The IMU sensor 38 can be functionally connected to the IMU (not shown) of the host vehicle 100. The IMU can be one or more mechanisms, devices, elements, components, systems, applications, and / or combinations thereof configured to determine the inertial properties of the host vehicle 100. Such inertial properties may include, for example, velocity, orientation, gravity, etc. The IMU sensor 38 can determine the inertial properties of the host vehicle 100 using an accelerometer, gyroscope, etc. The IMU sensor 38 can be configured to monitor data received from the IMU of the host vehicle 100.
[0052] In one or more configurations, the passive sensor 26 may include a Controller Area Network (CAN) sensor 40. The CAN sensor 40 can be functionally connected to the CAN bus (not shown) of the host vehicle 100. The CAN bus may be one or more components that are functionally connected to each other and communicate via a network in the host vehicle 100. The network may be a wired network and / or a wireless network. The CAN sensor 40 may be configured to monitor, access, and / or evaluate data transmitted on the CAN bus of the host vehicle 100.
[0053] The vehicle motion sensor 28 can be any component or group of components configured to detect the position, velocity (as a vector), velocity (as a scalar), and / or acceleration of the host vehicle 100. The vehicle motion sensor 28 can detect the position, velocity (as a vector), velocity (as a scalar), and / or acceleration of the host vehicle 100 using a number of sensors already present in the vehicle. Such sensors include, for example, a tachometer, steering angle sensor, wheel angle sensor, wheel speed sensor, or any other sensor capable of detecting the position, velocity (as a vector), velocity (as a scalar), and / or acceleration of the host vehicle 100.
[0054] The sensor system 22 can be configured to monitor the external environment 12 of the host vehicle 100, including objects detected in the external environment 12. In response to detecting objects in the external environment 12 of the host vehicle 100, the host vehicle 100 can classify the objects.
[0055] In one or more configurations, the host vehicle 100 may include an object classification database 42. The object classification database 42 may include data corresponding to the classification of various objects. For example, the object classification database 42 may include data corresponding to the characteristics of various vehicles, on-road or off-road objects, and road attributes. Such characteristics may be the shape of an image stored in the object classification database 42, a typical numerical representation from one or more sensors in the sensor system 22 indicating a particular type of object, and / or any other shape of data useful for classifying objects. Examples of various vehicles, on-road or off-road objects, and road attributes include, for example, vehicles, motorcycles, trees, pedestrians, people riding bicycles, animals, road signs, obstacles, or any other objects typically found on or along a road.
[0056] The features stored in the object classification database 42 can be compared with data obtained by the sensor system 22. For example, an image of a vehicle stored in the object classification database 42 can be compared with an image of an object in the external environment 12 obtained by the camera 36. Such a comparison can be performed by image processing software commonly known in this technology. The image processing software is implemented on the processor 18. In response to the comparison that the vehicle is substantially the same as the object in the external environment 12, the host vehicle 100 can determine that the object in the external environment 12 is the surrounding vehicle 14.
[0057] Additionally or alternatively, the object classification database 42 can store data representation values that are typical for specific types of objects. For example, the object classification database 42 can store data from typical lidar sensor representation values indicating the presence of a tree. The data from typical lidar sensor representation values can be compared with data received from the lidar sensor 32. In response that the comparison of typical lidar sensor representation values is substantially the same as the object in the external environment 12 detected by the lidar sensor 32, the host vehicle 100 can determine that the object in the external environment is a tree. Substantially the same could be, for example, within one standard deviation, within half a standard deviation, within a quarter of a standard deviation, etc. Although the two examples above are provided for clarity, any type of data can be stored in the object classification database 42 for comparison with data obtained via the sensor system 22. As a result of the comparison, the object detected in the external environment 12 of the host vehicle 100 can be classified.
[0058] The object classification database 42 may further include data indicating various types of vehicles. Examples of vehicle types include sedans, sport utility vehicles (SUVs), convertibles, pickup trucks, semi-trucks, campers, motorcycles, tractors, specific brands or models of vehicles, and / or any other subclasses of vehicles. Thus, in response to the detection that an object is a surrounding vehicle 14, the host vehicle 100 can determine the type of the surrounding vehicle 14. For example, the processor 18 can receive data from the sensor system 22. The data may indicate that an object located in the external environment 12 is a surrounding vehicle 14. In response to the determination that the object is a surrounding vehicle 14, the processor 18 can compare the data with various types of vehicles stored in the object classification database 42. Based on a comparison of the data received from the sensor system 22 with data stored in the object classification database 42, which shows typical sensor readings when a convertible is present in the external environment 12, the processor 18 can determine that the surrounding vehicle 14 is, for example, a convertible.
[0059] The host vehicle 100 may further include a predetermined vehicle characteristics database 44. The predetermined vehicle characteristics database 44 may include one or more predetermined vehicle characteristics for various types of vehicles. A predetermined vehicle characteristic can indicate driving behavior for a vehicle type. A predetermined characteristic can indicate at least one driving behavior associated with multiple vehicles of the same type. For example, a predetermined vehicle characteristic for a convertible may be more aggressive than a predetermined vehicle characteristic for a semi-truck.
[0060] A given vehicle characteristic can be a numerical value on a scale, a percentage of aggression, a weighting factor compared to a standard sedan, or any other way that characterizes the driving style. For example, a given vehicle characteristic for a standard sedan might be 50% aggression, while a given vehicle characteristic for a convertible might be 80% aggression. Additionally or alternatively, a given vehicle characteristic for a standard sedan might be an aggression factor of 1.0, while a given vehicle characteristic for a semi-truck might be an aggression factor of 0.5. While the above two examples are provided, it should be understood that various weighting factors and percentages can be used to correspond to various classes of vehicles, and the weighting factors for special classes of vehicles are not intended to be limited to the examples provided. In practice, many different ways of characterizing the driving style can be used for special classes of vehicles.
[0061] Additionally or alternatively, a given vehicle characteristic can be classified as a driving behavior for a particular maneuver. For example, a given vehicle characteristic for an SUV might include data indicating that they tend to travel at higher speeds than other vehicles in bad weather, but travel at average speeds in normal weather, prefer the center lane, and change lanes at a slower rate than other vehicles. Similarly, a given vehicle characteristic for a convertible might include data indicating that they tend to travel at lower speeds than other vehicles in bad weather, but travel at higher speeds in normal weather, prefer the high-speed lane, and change lanes at a faster rate than other vehicles. While the above two examples have been provided, it should be noted that this disclosure is not limited to these two examples. In practice, any method of characterizing various types of vehicles can be used in the given vehicle characteristic database 44. Additionally, different levels of characterization can be incorporated, including characterization based on any other level of characterization that can be used to distinguish driving styles based on vehicle classification (e.g., sedan, SUV, convertible, pickup truck, etc.), vehicle brand (e.g., Toyota, Lexus, Honda, Ford, Dodge, etc.), vehicle model (e.g., Toyota Camry, Lexus IS, Honda Accord, Ford Flex, Dodge Charger, etc.), and / or a specific type of vehicle.
[0062] The host vehicle 100 may include various subsystems (e.g., trajectory predictors). These subsystems can be represented as various modules 46. Examples of subsystems include trajectory predictors such as the safety trajectory determination module 60 or the surrounding vehicle trajectory prediction module 52. Other subsystems may be directed towards driver actions, fail-safe mechanisms, database updates, etc. The various modules 46 perform various tasks in the host vehicle 100. When executed by the processor 18, a module 46 can be implemented as computer-readable program code that implements one or more of the various processes described herein. Such computer-readable program code can be stored in memory 19. A module 46 may be a component of the processor 18, or it may run on and / or be distributed among other processing systems to which the processor 18 is functionally connected. A module 46 may include instructions (e.g., program logic) that can be executed by the processor 18. Additionally or alternatively, memory 19 may contain such instructions. Various modules 46 can be functionally connected to the processor 18, the database 20, and / or the sensor system 22. Various examples of modules 46 that the host vehicle 100 may include are described here.
[0063] In one or more configurations, the host vehicle 100 may include a driver behavior module 48. The driver behavior module 48 can receive predetermined vehicle characteristics from a predetermined vehicle characteristics database 44 in order to determine the actions of the drivers of identified surrounding vehicles 14.
[0064] Additionally or alternatively, the driver behavior module 48 can receive data from the sensor system 22. The data received from the sensor system 22 can indicate the behavior of the driver of an identified surrounding vehicle 14. In such an example, the data received from the sensor system 22 can be used both to classify objects located in the external environment and to determine the behavior of the driver of the surrounding vehicle 14. The driver behavior module 48 can use both predetermined vehicle characteristics received from a predetermined vehicle characteristics database 44 and the data received from the sensor system 22 to determine the driver's behavior. Such data received from the sensor system 22 can determine, or can be used to determine, the current driving characteristics of the surrounding vehicle 14. The current driving characteristics can indicate one or more driving behaviors. For example, the current driving characteristics may be a numerical value on a scale, a percentage of aggression, a weighting factor compared to a standard sedan, or any other way that characterizes one or more driving behaviors toward the surrounding vehicle 14 based on data obtained from the sensor system 22. Examples of such data include changes in direction, speed, acceleration, lane position, and / or any other type of data that may be useful in determining the behavior of drivers of surrounding vehicles 14.
[0065] In one or more configurations, the current driving characteristics may be substantially the same as predetermined vehicle characteristics for surrounding vehicles 14. In such cases, surrounding vehicles 14 exhibit one or more driving behaviors that match the type of surrounding vehicle. In one or more configurations, the current driving characteristics may differ from the predetermined vehicle characteristics. In such cases, surrounding vehicles 14 do not exhibit one or more driving behaviors that match the type of surrounding vehicle. The driver behavior module 48 can use both predetermined vehicle characteristics received from the predetermined vehicle characteristics database 44 and data received from the sensor system 22 to determine the driver's behavior.
[0066] The host vehicle 100 may further include an identification vehicle database 50. The host vehicle 100 can record identifiers for surrounding vehicles 14. Such identifiers may include, for example, license plate numbers, vehicle identification numbers, or any other type of identifier commonly used to distinguish individual vehicles. Identifiers can be stored in the identification vehicle database 50. Additionally or alternatively, predetermined vehicle characteristics for surrounding vehicles 14 can also be stored in the identification vehicle database 50. In such an example, the identification vehicle characteristics may be associated with identifiers, and both the identifiers and the associated predetermined vehicle characteristics can be stored in the identification vehicle database 50.
[0067] In one or more configurations, the host vehicle 100 may include a surrounding vehicle trajectory prediction module 52. The surrounding vehicle trajectory prediction module 52 can predict the trajectory of the surrounding vehicle 14 based on either predetermined vehicle characteristics for the surrounding vehicle 14 or current driving characteristics based on data received from the sensor system 22. Referring here to Figures 1B, 1C, and 2, the surrounding vehicle 14 is shown in the left lane relative to the host vehicle 100. The host vehicle 100 can determine the type of the surrounding vehicle 14, for example, that it is a convertible. The host vehicle 100 can also use data received from the sensor system 22 to detect that the surrounding vehicle 14 has accelerated relative to the host vehicle 100, and that the surrounding vehicle 14 has moved its position laterally toward a lane marking directly adjacent to the host vehicle 100 and between the host vehicle 100 and the surrounding vehicle 14. The sensor system may also include lane marking sensors (not shown) used to monitor and sense vehicles or objects (e.g., other vehicles) within the lane, or other sensors that respond to the presence of road signs, traffic signals, signs, or emergency vehicles. Such sensor data can be used to establish lane paths in a world model and therefore for predictive path calculations.
[0068] Additionally or alternatively, the host vehicle 100 can access map data from the navigation unit 54 of the host vehicle 100 indicating that the left lane ends ahead. The host vehicle 100 can use a predetermined vehicle characteristics database 44, data received from any one of the sensor systems 22, and the map data from the navigation unit 54 to predict the trajectory 16 of surrounding vehicles 14. The navigation unit 54 can be integrated into the head unit (not shown) of the host vehicle 100, or the navigation unit 54 can be a standalone or aftermarket component. The navigation unit 54 can include map data stored therein (for example, stored in a database included in the navigation unit 54), or the host vehicle 100 can further include a map database 56. Additionally or alternatively, the navigation unit 54 can be integrated into a passenger's mobile device and can store map data therein. In such an example, the host vehicle 100 can access the passenger's mobile device and retrieve the map data stored therein.
[0069] Additionally, each of the received data types may have a corresponding confidence level. Referring here to Figure 2, the surrounding vehicle trajectory prediction module 52 can evaluate the confidence levels of given vehicle characteristics, current driving characteristics, and map data. Such confidence levels can be used to optimize the predicted trajectory 16 for the surrounding vehicles 14. For example, if some of the sensors in the sensor system 22 are not functioning optimally or are in a faulty state, the current driving characteristics may have a lower confidence level. The confidence level can be a numerical value on a scale, a confidence percentage, a weighted factor, or any other way that characterizes the confidence level in the data.
[0070] The surrounding vehicle trajectory prediction module 52 can also evaluate the confidence level of any of the surrounding vehicles 14, including predetermined vehicle characteristics, current driving characteristics, and map data, and can predict the predicted trajectory 16 for the surrounding vehicles 14. In one or more configurations, the predicted trajectory 16 may also have a corresponding confidence level. Very similar to the confidence levels described above, the confidence level for the predicted trajectory 16 may be a numerical value on a scale, a percentage of confidence, a weighted factor, or any other way that characterizes the confidence level in the data.
[0071] In one or more configurations, the host vehicle 100 may include a database update module 58. The database update module 58 can be configured to update any of the databases 20. Such updates can be performed based on various factors. In one or more configurations, the database update module 58 may receive prompts for updates. Such prompts can be received by similar methods typically used to update software and databases commonly known in this art. For example, prompts can be received at a car dealership or wirelessly.
[0072] In one or more configurations, the database update module 58 can determine that an update is necessary. Such a decision can be made for a specific type of vehicle or for a particularly identified vehicle. Thus, in such cases, the database update module 58 can update either the predetermined vehicle characteristics database 44 or the identified vehicle database 50. The database update module 58 can compare predetermined vehicle characteristics for surrounding vehicles 14 with the identified driving characteristics of the surrounding vehicles 14. If the predetermined vehicle characteristics and the identified driving characteristics differ, the database update module 58 can determine that an update is necessary for either the predetermined vehicle characteristics database 44 or the identified vehicle database 50. For example, the database update module 58 can update the identified vehicle database 50 in response to the fact that the identified driving characteristics differ from the predetermined vehicle characteristics, and that the identified driving characteristics are not, for example, within one standard deviation of the predetermined vehicle characteristics, within two standard deviations of the predetermined vehicle characteristics, or within three standard deviations of the predetermined vehicle characteristics. The database update module 58 can update the predetermined vehicle characteristics stored in the identified vehicle database 50 that correspond to the identifier corresponding to the surrounding vehicle 14. Additionally or alternatively, the database update module 58 can update the predetermined vehicle characteristics database 44 in response to the fact that the identified driving characteristics differ from predetermined vehicle characteristics, but for example, within one standard deviation of the predetermined vehicle characteristics.
[0073] The host vehicle 100 may include a safety trajectory determination module 60. The safety trajectory determination module 60 can determine a safety trajectory that the host vehicle 100 will follow. The safety trajectory may be, for example, a trajectory that does not cross the predicted trajectory 16 for surrounding vehicles 14. In some examples, such a safety trajectory may be following a previously planned trajectory for the vehicle when the previously planned trajectory for the host vehicle 100 is determined to be safe. The safety trajectory can be determined using any of the following: predicted trajectories from the surrounding vehicle trajectory prediction module 52, current driving characteristics from the sensor system 22, predetermined vehicle characteristics from a predetermined vehicle characteristics database 44 based on the type of surrounding vehicles 14, map data from the navigation unit 54, and any other data available to determine the safety trajectory of surrounding vehicles. The safety trajectory may include a confidence level. The confidence level for the safety trajectory may be a numerical value on a scale, a confidence level as a percentage, a weighted factor, or any other way of characterizing the confidence level in the data.
[0074] The host vehicle 100 can follow a safe trajectory in response to one or more sensors in the sensor system 22 changing from an active state to a faulty state. An active state can be the active, consistent, periodic, and / or predictable reception of reliable data from the sensor. A faulty state can be the inconsistent reception of data, the reception of unreliable or extremely out-of-bounds data, the sensor being in a state of being shut down, the sensor being in a passive state, or the sensor being in sleep mode. An example of a sensor being in a faulty state is when it is snowing outside the vehicle and some sensors (e.g., camera 36, radar sensor 30) may receive unreliable data. In these examples, the sensors receiving unreliable data can be considered to be in a faulty state. In response to one or more sensors in the sensor system 22 being in a faulty state, the host vehicle 100 can follow a safe trajectory.
[0075] Additionally or alternatively, the host vehicle 100 can follow a safe trajectory in response to a failure in one of the sensor categories (e.g., active sensor 24, passive sensor 26, vehicle motion sensor 28). In one or more configurations, the host vehicle 100 can use the safe trajectory and the active sensors when one or more of the categories are in a failure state. For example, when the active sensor 24 is in a failure state and the passive sensor 26 and vehicle motion sensor 28 are in an active state, the safe trajectory determination module 60 can determine a safe trajectory based on a predicted trajectory such as that determined by the surrounding vehicle trajectory prediction module 52, the passive sensor 26, and the vehicle motion sensor 28. Also, when the passive sensor 26 is in a failure state and the active sensor 24 and vehicle motion sensor 26 are in an active state, the safe trajectory determination module 60 can determine a safe trajectory based on a predicted trajectory such as that determined by the surrounding vehicle trajectory prediction module 52, the active sensor 24, and the vehicle motion sensor 28. Furthermore, when both the active sensor 24 and the passive sensor 26 are in a malfunction state, and the vehicle motion sensor 28 is in an active state, the safety trajectory determination module 60 can determine a safety trajectory based on a predicted trajectory determined by the surrounding vehicle trajectory prediction module 52 and the vehicle motion sensor 28.
[0076] The host vehicle 100 may include a fail-safe module 62. In an example where the safe track includes a confidence level, the fail-safe module 62 can compare the confidence level for the safe track to a threshold. The threshold can be, for example, a numerical threshold on a scale, a confidence threshold for a percentage, a weighting factor threshold, or any other way of setting a threshold for the confidence level in the data. If the confidence level for the safe track is, for example, less than the confidence level threshold, the host vehicle 100 can perform a fail-safe. Such a threshold can be the minimum level of confidence; otherwise, the host vehicle 100 performs a fail-safe. Alternatively, the maximum level of unreliability (e.g., the inverse of confidence) can be used as a threshold, and if the confidence level exceeds the maximum level of the unreliability threshold, the host vehicle 100 performs a fail-safe. Fail-safes can include, for example, stopping the host vehicle 100, moving the host vehicle 100 to the side of the road, slowing down the host vehicle 100, idling the host vehicle 100, or any other fail-safe for the host vehicle 100 to safely disable, stop, and / or prohibit the host vehicle 100 from moving.
[0077] The vehicle's sensor system 22 may also include ambient conditions sensors 64 that provide sensor signals related to the surrounding conditions, such as external temperature, precipitation (rain, snow, etc.), and road conditions (e.g., road roughness, or the presence of water or ice). Ambient conditions sensor data can be used to determine the reliability of other sensor values and to modify the vehicle model or the behavior of other objects. In some embodiments, the ambient conditions are provided to the vehicle via a network.
[0078] The sensor system 22 can also be used to monitor the vehicle itself to sense vehicle parameters such as, for example, vehicle speed, acceleration (in one or more dimensions), throttle position, engine speed, braking operation (pedal position and / or ABS operation), steering input, roll rate, wheel slip, occupancy and weight of occupants (e.g., to modify the vehicle behavior model), other engine inputs, driving component configuration (such as road wheel angle for an automobile), orientation and arrangement of control surfaces for an aircraft (such as ailerons or rudders), rudder orientation for a boat, nozzle configuration for a spacecraft, or other parameters. The sensor system 22 may also include GPS or other position sensors for measuring vehicle position, speed, and altitude. Vehicle speed can be determined as a time derivative of position, and acceleration can be determined as a time derivative of velocity. The sensor system 22 can also monitor the vehicle operator, for example, by using eye or gaze tracking, or by monitoring bio-parameters such as fatigue levels and related bio-parameters.
[0079] The sensor system 22 can also receive data from a remote source, for example, via a wireless communication link, or obtain data from other sources. For example, it can receive weather, road, traffic, radar, or other data. The sensors can be located away from the vehicle, for example, on the roadside, or embedded in the road, and the sensor data can be transmitted to the vehicle wirelessly. The sensors can be oriented forward, backward, or to the side of the vehicle, or any combination thereof, and include omnidirectional sensors.
[0080] For example, a sensor fusion element can be used to combine image data from multiple image sensors of the same or different types to represent the vehicle environment. A MIMO processor can be used as a sensor fusion element.
[0081] The safety trajectory determination module 60 or the surrounding vehicle trajectory prediction module 52 may trust input from sensor systems 22 that it should not trust, or it may not trust input from some sensors that it should trust. This trust in these sensors may be undesirable for the reliability of the safety trajectory determination module 60 or the surrounding vehicle trajectory prediction module 52.
[0082] Accordingly, aspects of this disclosure are directed towards an ego vehicle (e.g., ego or host vehicle 100) configured to determine the reliability of sensor values or data obtained from the sensor system 22 based on the surrounding conditions determined by the ego vehicle 100. For the outputs of a subsystem generated based on the obtained sensor values and / or other obtained values (e.g., outputs from the database 20), the ego vehicle 100 can determine the credibility of the outputs generated by the subsystem, based (at least in part) on the reliability of the obtained sensor values. The credibility of the subsystem outputs in terms of accuracy can be separate from the credibility of the model in terms of fidelity (e.g., separate from the credibility of a given prediction generated by a trajectory predictor).
[0083] As the surrounding conditions change, the ego vehicle 100 can update the reliability of the sensor values from the surrounding conditions sensor 64, and thus update the reliability of the output accuracy of a given subsystem. In one embodiment, the ego vehicle 100 obtains sensor values from the sensor system 22 and / or database values from the database 20. For example, the ego vehicle 100 can be equipped with a variety of sensors and databases, and the ego vehicle 100 can obtain sensor and / or database values from one or more of these sensors and databases. Among many other possibilities, examples of sensors include a lidar sensor 32, a radar sensor 30, and a camera 36.
[0084] The Ego Vehicle 100 determines one or more surrounding conditions using the surrounding condition sensors 64 and / or surrounding condition information received wirelessly from a remote device. For example, the processor 18 of the Ego Vehicle 100 can determine, based on sensor values and other information, whether the Ego Vehicle is on a highway, in a congested area, inside a tunnel, at an intersection, or about to change lanes. The Ego Vehicle 100 can determine cloud cover, the current time of day, and whether it is daytime or nighttime, based on sensor values from the surrounding sensors. The Ego Vehicle 100 can determine any combination of these or other surrounding conditions.
[0085] The Ego Vehicle 100 determines the reliability of the obtained sensor values from the active sensor 24, passive sensor 26, vehicle motion sensor 28, and / or database 20 based on the surrounding conditions determined from the surrounding conditions sensor 64. For example, if the Ego Vehicle 100 is in a congested or crowded area, the sensor values obtained from the radar sensor 30 are relatively unreliable because, for example, the radar sensor 30 may be interfered with and / or the sensor values may contain an unacceptable amount of noise. On the other hand, in such a situation, the sensor values obtained from the LiDAR sensor 32 mounted on the roof rack of the Ego Vehicle 100 are relatively reliable. However, if the Ego Vehicle 100 is traveling on a highway, the sensor values obtained from the radar sensor 30 are relatively reliable.
[0086] As another example, the sensor values obtained from camera 36 are relatively unreliable in low-light conditions (resulting in a black image) or high-light conditions (resulting in an unclear white image).
[0087] Reliability can be determined by a reliability estimator adjusted based on previously collected driving data. For example, the driving data may show the predicted trajectory for a vehicle and the actual trajectory taken by the vehicle. The trajectory can be predicted based on sensor values from one or more sensors that are the same as (or similar to) those of the ego vehicle 100, obtained under the same (or similar) surrounding conditions. The reliability estimator can be adjusted based on the calculated difference between the predicted trajectory and the actual trajectory. Based on this adjustment, the reliability estimator can determine the reliability of the sensor values obtained from one or more sensors under the current conditions. In one embodiment, the reliability estimator may be part of the processor 18. In another embodiment, the reliability estimator may be part of the module 46.
[0088] In one embodiment, the credibility estimator generates a credibility level for the output of a subsystem of the ego vehicle 10. The credibility level indicates the credibility in accuracy of the output of the subsystem (e.g., the surrounding vehicle trajectory prediction module 52 or the safe trajectory determination module 60). The subsystem output is based on obtained sensor values, and the credibility level is generated based (at least in part) on the reliability of the obtained sensor values. In one embodiment, the credibility estimator may be part of the processor 18. In other embodiments, the credibility estimator may be part of module 46.
[0089] For example, the ego vehicle 100 can provide sensor values to a trajectory predictor (e.g., surrounding vehicle trajectory prediction module 52 or safety trajectory determination module 60), and the trajectory predictor generates a predicted trajectory for the ego vehicle 100 (or another vehicle) based on the sensor values. The ego vehicle 100 may determine that the obtained sensor values are relatively unreliable, and therefore the credibility estimator may generate a low credibility level for the predicted trajectory. Thus, even if the trajectory predictor predicts that the ego vehicle 100 is highly likely to take a given trajectory, the ego vehicle 100 may decide not to pass this predicted trajectory to a different subsystem due to the low credibility of the highly likely trajectory.
[0090] Figure 3 shows an example of a hardware implementation of the dynamic and variable learning system 300 according to an aspect of the present disclosure. The dynamic and variable learning system 300 can be a component of a vehicle, a robotic device, or other device. For example, as shown in Figure 3, the dynamic and variable learning system 300 is a component of an autonomous vehicle 328. The dynamic and variable learning system 300 is located at the rear of the autonomous vehicle 328, but it can be located anywhere on the vehicle (e.g., at the front of the vehicle). The aspects of the present disclosure are not limited to the autonomous vehicle 328, as other devices such as buses, boats, drones, or robots are also conceivable for the use of the dynamic and variable learning system 300. The autonomous vehicle 328 may be autonomous or semi-autonomous.
[0091] The dynamic and variable learning system 300 can be implemented in a bus architecture, typically represented by a bus 350. The bus 350 may include any number of interconnection buses and bridges, depending on the specific application of the dynamic and variable learning system 300 and the constraints on the overall design. The bus 350 links together various circuits, including one or more processors and / or hardware modules, such as the processor 320, communication module 322, position module 318, sensor module 302, motion module 326, navigation module 324, computer-readable medium 314, and the dynamic and variable learning module 308. The dynamic and variable learning module 308 includes a reliability predictor 308a and a reliability estimator 308b. In some embodiments, the dynamic and variable learning module 308, including the reliability predictor 308a and the reliability estimator 308b, is part of the processor 320. Bus 350 can also link various other circuits, such as timing sources, peripherals, voltage regulators, and power management circuits, which are well known in this technology and will not be described further.
[0092] The dynamic and variable learning system 300 includes a transceiver 316 coupled to a processor 320, a sensor module 302, a dynamic and variable learning module 308, a communication module 322, a position module 318, a movement module 326, a navigation module 324, and a computer-readable medium 314. The transceiver 316 is coupled to an antenna 344. The transceiver 316 communicates with various other devices via the transmitting medium. For example, the transceiver 316 can receive commands via transmission from a user or remote device. As another example, the transceiver 316 can transmit driving statistics and information, environmental information, and other desirable information from the dynamic and variable learning module 308 to a server (not shown), and vice versa.
[0093] The dynamic and variable learning system 300 includes a processor 320 coupled to a computer-readable medium 314. The processor 320 performs processing, including the execution of software stored in the computer-readable medium 314, and provides the functions disclosed. When executed by the processor 320, the software causes the dynamic and variable learning system 300 to perform various functions described for an autonomous vehicle 328 or any of the modules 302, 314, 316, 318, 320, 322, 324, or 326. The computer-readable medium 314 can also be used to store data manipulated by the processor 320 when the processor 320 executes the software.
[0094] The sensor module 302 can obtain measured values or environmental information through different sensors, such as the first sensor 306 and the second sensor 304. For example, the sensors can determine ambient information and sensor values and provide them to the dynamic and variable learning module 308. The first sensor 306 may be an ambient sensor. The second sensor may be a distance sensor, such as a light detection and distance (lidar) sensor or a wireless detection and distance (radar) sensor. Of course, aspects of this disclosure are not limited to the sensors described above, as other types of sensors, such as thermal, ultrasonic, and / or laser sensors, can also be considered for either sensor 304 or 306.
[0095] The measurements from the first sensor 306 and the second sensor 304 can be processed by one or more of the following in conjunction with a computer-readable medium 314 to implement the functions described herein: the processor 320, the sensor module 302, the communication module 322, the position module 318, the dynamic and variable learning module 308, the movement module 326, and the navigation module 324. In one configuration, the data obtained by the first sensor 306 and the second sensor 304 can be transmitted to an external device via the transceiver 316. The first sensor 306 and the second sensor 304 can be coupled to or communicate with the autonomous vehicle 328.
[0096] The position module 318 can determine the position of the autonomous vehicle 328. For example, the position module 318 can use the Global Positioning System (GPS) to determine the position of the autonomous vehicle 328. The communication module 322 can facilitate communication via the transceiver 316. For example, the communication module 322 can be configured to provide communication functionality via different radio protocols such as WiFi, Long-Term Evolution (LTE), 4G, and 5G. The communication module 322 can also be used to communicate with other components of the autonomous vehicle 328 that are not modules of the dynamic and variable learning system 300.
[0097] The mobility module 326 can facilitate the movement of the autonomous vehicle 328. For example, the mobility module 326 can control the movement of the wheels. Another example is that the mobility module 326 can communicate with a power source of the autonomous vehicle 328, such as an engine or battery. Of course, aspects of this disclosure are not limited to providing movement via wheels, but can be considered for other types of components for providing movement, such as propellers, treads, fins, and / or jet engines.
[0098] The dynamic and variable learning system 300 also includes a navigation module 324 via a movement module 326 for planning routes or controlling the movement of the autonomous vehicle 328. The navigation module 324 can communicate with the dynamic and variable learning module 308, the sensor module 302, the transceiver 316, the processor 320, the communication module 322, the position module 318, the movement module 326, and the computer-readable medium 314.
[0099] A module may be software that operates on the processor 320 and resides / is stored on a computer-readable medium 314, one or more hardware modules coupled to the processor 320, or any combination thereof.
[0100] According to aspects of the present disclosure, the dynamic and variable learning system 300 includes a navigation module 324, a sensor module 322, a transceiver 316, a processor 320, a communication module 322, a position module 318, a movement module 326, and a dynamic and variable learning module 308 that communicates with a computer-readable medium 314.
[0101] In one configuration, the dynamic and variable learning module 308 determines reliability parameters for sensor values obtained from one or more sensors (e.g., sensors 304, 306) of the autonomous vehicle 328, based on the surrounding conditions determined by the ego vehicle 100. For example, as shown in Figure 2, the sensor system 22 provides surrounding condition values and other sensor values used to determine the credibility of subsystems, which are based on the reliability of the sensor values. For example, the credibility in the accuracy of the subsystem's output is determined based on the reliability of the sensor values.
[0102] For example, reliability can be determined by a reliability estimator 308a adjusted based on previously collected driving data. For example, the driving data can show the predicted trajectory for the vehicle and the actual trajectory taken by the vehicle. The trajectory can be predicted based on sensor values from the same (or similar) sensors as the autonomous vehicle 328, obtained under the same (or similar) surrounding conditions. The reliability estimator 308a can be adjusted based on the calculated difference between the predicted trajectory and the actual trajectory. Based on this adjustment, the reliability estimator 308a can determine the reliability of the sensor values obtained from one or more sensors under the current conditions.
[0103] In one embodiment, the credibility estimator 308b generates a credibility level for the output of the subsystem of the autonomous vehicle 328. The credibility level indicates the credibility of the subsystem's output in terms of accuracy. The subsystem's output is based on obtained sensor values, and the credibility level is generated based (at least in part) on the reliability of the obtained sensor values.
[0104] Figure 4 illustrates a method 400 for providing dynamic and variable learning of an ego vehicle by determining and using the most reliable input, according to an aspect of the present disclosure. As shown in Figure 4, in block 402, the ego vehicle determines the reliability level of sensor values obtained from one or more ego vehicle sensors based on the surrounding environment of the ego vehicle. In block 404, the ego vehicle determines the reliability level in accuracy of the output of an ego vehicle subsystem. The subsystem output is based on sensor values, and the reliability level is based on the reliability level of the sensor values.
[0105] Based on the teachings, those skilled in the art should recognize that the scope of this disclosure is intended to include any aspect of this disclosure, whether implemented independently or in combination with any other aspect of this disclosure. For example, an apparatus can be implemented using any number of the described aspects, and a method can be practiced using any number of the described aspects. In addition, the scope of this disclosure is intended to include apparatuses or methods practiced using other structures, functions, or structures and functions, in addition to or other than the various aspects of this disclosure described. It should be understood that any aspect of this disclosure can be embodied by one or more elements of the claims.
[0106] The term "as an example" is used here to mean "provided as an example, instance, or illustration." Any aspect described herein as "as an example" should not necessarily be interpreted as being preferable or superior to any other aspect.
[0107] While specific embodiments are described herein, many variations and substitutions of these embodiments are within the scope of this disclosure. Although some benefits and advantages of preferred embodiments are mentioned, the scope of this disclosure is not intended to be limited to specific benefits, uses, and purposes. Rather, embodiments of this disclosure are intended to be broadly applicable to different technologies, system configurations, networks, and protocols, some of which are illustrated in the figures of preferred embodiments and in the description below. The detailed description and figures are illustrative and not intended to limit the scope of this disclosure as defined by the accompanying claims and their equivalents.
[0108] The term "decide" as used here encompasses a wide range of actions. For example, "decide" can include calculating, operating, processing, deriving, investigating, examining (e.g., examining a table, database, or other data structure), and confirming. Additionally, "decide" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and furthermore, "decide" can include resolving, selecting, choosing, and establishing.
[0109] The phrase "at least one" in a list of items, as used here, refers to any combination of those items that includes a single component. For example, "at least one of a, b, or c" is intended to include a, b, c, a and b, a and c, b and c, and a, b and c.
[0110] The various examples of logic blocks, modules, and circuits described in connection with this disclosure can be implemented or run on a processor configured to perform the functions considered herein. The processor may be a neural network processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device (PLD), isolated gate or transistor logic, isolated hardware components, or any combination thereof designed to perform the functions described herein. The processor may be a microprocessor, a controller, a microcontroller, or a state machine specially configured as described herein. The processor may also be implemented as a combination of arithmetic units, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors working with a DSP core, or other special configurations as described herein.
[0111] Steps of methods or algorithms described in connection with this disclosure can be embodied directly in hardware, in software modules executed by a processor, or a combination of both. A software module can reside on storage or machine-readable media, including random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, CD-ROMs or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other media accessible by a computer that can be used to transport or store desired program code in the form of instructions or data structures. A software module may comprise a single instruction or a number of instructions and may be distributed across several different code segments, across different programs, and across a number of storage media. A storage medium can be coupled to a processor so that the processor can read information from and write information to the storage medium. Alternatively, the storage medium can be integrated into the processor.
[0112] The methods disclosed herein comprise one or more steps or actions to achieve the described method. The steps and / or actions of the method can be replaced without departing from the scope of the claims. In other words, unless a particular order of steps or actions is specified, the order and / or use of any particular steps and / or actions can be modified without departing from the scope of the claims.
[0113] The functions described can be implemented in hardware, software, firmware, or any combination thereof. When implemented in hardware, an example hardware configuration may include a processing system in the device. The processing system can be implemented in a bus architecture. The bus may include any number of interconnection buses and bridges, depending on the specific application of the processing system and the constraints of the overall design. The bus can link together various circuits, including processors, machine-readable media, and bus interfaces. The bus interface can connect network adapters, in particular, to the processing system via the bus. Network adapters can implement signal processing functions. In some embodiments, user interfaces (e.g., keypads, displays, mice, joysticks, etc.) can also be connected to the bus. The bus can also link various other circuits, such as timing sources, peripherals, voltage regulators, and power management circuits, which are well known in this art and will not be described further.
[0114] The processor can manage buses and processing, including the execution of software stored on machine-readable media. Whether referred to as software, firmware, middleware, microcode, hardware description language, etc., software should be interpreted as meaning instructions, data, or any combination thereof.
[0115] In hardware implementations, machine-readable media can be part of a processing system separate from the processor. However, as those skilled in the art will readily recognize, machine-readable media, or any part thereof, can be outside the processing system. For example, machine-readable media can include transmission lines, data-modulated carriers, and / or computer products separate from the device, all of which are accessible to the processor via a bus interface. Alternatively or additionally, machine-readable media, or any part thereof, can be integrated into the processor, as in the case of caches and / or specialized register files. The various components under consideration can be described as having a specific location, like local components, but they can also be configured in various ways, like some components configured as part of a distributed computing system.
[0116] The processing system may consist of one or more microprocessors providing the functionality of a processor, and external memory providing at least a portion of a machine-readable medium, all of which are linked to other supporting circuits via an external bus architecture. Alternatively, the processing system may comprise one or more neuromorphic processors to implement the neuron model and neural system model described herein. As yet another alternative, the processing system may be implemented in an application-specific integrated circuit (ASIC) having the processor, bus interface, user interface, supporting circuits, and at least a portion of a machine-readable medium integrated on a single chip, or by one or more field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, isolated hardware components, or any other suitable circuits, or any combination of circuits capable of performing the various functions described throughout this disclosure. Those skilled in the art will recognize how the functions described for the processing system are best implemented, depending on the specific application and the overall design constraints imposed on the system as a whole.
[0117] A machine-readable medium can contain a number of software modules. These software modules may include transmit and receive modules. Each software module may reside in a single storage device or be distributed across multiple storage devices. For example, a software module may be loaded from a hard drive into RAM when a triggered event occurs. While a software module is running, the processor may load some of the instructions into a cache to increase access speed. One or more cache lines may then be loaded into a special-purpose register file for execution by the processor. When the functionality of a software module is referred to below, it will be understood that such functionality is realized by the processor when executing instructions from that software module. Furthermore, it should be recognized that aspects of this disclosure result in improvements to the functionality of processors, computers, machines, or other systems that implement such aspects.
[0118] When implemented in software, a function can be stored or transmitted as one or more commands or codes on a computer-readable medium. Computer-readable medium includes both computer storage devices and communication media, which include any storage medium that facilitates the transfer of computer programs from one location to another. Additionally, any connection can appropriately be called a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared (IR), radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disks and disc-shaped discs as used herein include compact discs (CDs), laserdiscs (registered trademark), optical discs, digital multipurpose discs (DVDs), floppy disks (registered trademark), and Blu-ray (registered trademark) discs, where discs typically reproduce data magnetically, while disc-shaped discs reproduce data optically using a laser. Therefore, in some embodiments, the computer-readable medium may include a non-temporary computer-readable medium (e.g., a physical medium). Additionally, in other embodiments, the computer-readable medium may include a temporary computer-readable medium (e.g., a signal). The above combinations should also be included within the scope of computer-readable mediums.
[0119] Therefore, in some embodiments, a computer program product may be provided to perform the operations presented herein. For example, such a computer program product may include a computer-readable medium having instructions stored (and / or encoded therein) thereon, which are executable by one or more processors to perform the operations described herein. In some embodiments, the computer program product may include packaging material.
[0120] Furthermore, it should be recognized that modules and / or other suitable means for performing the methods and techniques described herein can be obtained by download and / or by user terminals and / or base stations, where applicable. For example, such devices can be coupled to a server to facilitate the transfer of means for performing the methods described herein. Alternatively, the various methods described herein can be provided via storage means so that user terminals and / or base stations can obtain the various methods by coupling or providing storage means to the device. Furthermore, any other suitable techniques for providing the methods and techniques described herein to the device can be utilized.
[0121] It should be understood that the claims are not limited to the exact configurations and components described above. Various modifications, changes, and variations in the arrangement, operation, and details of the above methods and apparatus can be made without departing from the scope of the claims.
Claims
1. 1. A method for controlling an ego vehicle by determining and using a most reliable input, comprising: determining a confidence level of sensor values from one or more ego vehicle sensors based on environmental conditions of the ego vehicle; determining a level of confidence in the accuracy of an output of the ego vehicle's sensor value based on a confidence level of the sensor value; The reliability level is determined by a reliability estimator trained based on previously collected driving data, and the reliability level is determined based on the reliability estimator; predicting trajectories for surrounding vehicles based on the assessment of the confidence levels; determining a safe trajectory for the ego vehicle to follow that does not cross a predicted trajectory for the surrounding vehicles; if the confidence level in the safe trajectory is less than a confidence level threshold, the ego vehicle executes a failsafe; The method comprises:
2. updating a confidence level of the sensor value when the ambient conditions change; updating the confidence level in the accuracy of outputs of the subsystems of the ego vehicle based on updated sensor values; The method of claim 1 further comprising:
3. The method of claim 1 , wherein the ambient conditions are determined by ambient condition sensors on the ego vehicle and / or are provided to the ego vehicle from a remote device.
4. The method of claim 1 , further comprising: reducing a confidence level of the sensor value from a radar sensor when the ego vehicle is in a congested area.
5. The method of claim 1 , further comprising increasing a confidence level of the sensor value of a radar sensor when the ego vehicle is traveling on a highway.
6. 2. The method of claim 1, further comprising: reducing a confidence level of the sensor values from the ego vehicle's camera when the ego vehicle is in low light conditions resulting in a black image or high light conditions resulting in a fuzzy white image.
7. 1. A system for controlling an ego vehicle by determining and using a most reliable input, comprising: Memory, At least one processor, determining a confidence level of sensor values from one or more ego vehicle sensors based on ambient conditions of the ego vehicle's environment; determining a level of confidence in the accuracy of an output of the ego vehicle's sensor value based on a confidence level of the sensor value; The reliability level is determined by a reliability estimator trained on previously collected driving data, and the reliability level is determined based on the reliability estimator; predicting trajectories for surrounding vehicles based on the assessment of the confidence levels; determining a safe trajectory for the ego vehicle to follow that does not intersect predicted trajectories for the surrounding vehicles; a processor configured to cause the ego vehicle to execute a failsafe if the confidence level in the safe trajectory is below a threshold confidence level; The system comprises:
8. The at least one processor updating a confidence level of the sensor value when the ambient conditions change; updating the confidence level in the accuracy of the outputs of the subsystems of the ego vehicle based on updated sensor values. The system of claim 7 further configured to:
9. The system of claim 7 , wherein the ambient conditions are determined by ambient condition sensors on the ego vehicle and / or are provided to the ego vehicle from a remote device.
10. 8. The system of claim 7, wherein the at least one processor is further configured to reduce a confidence level of the sensor value from a radar sensor when the ego vehicle is in a congested area.
11. 8. The system of claim 7, wherein the at least one processor is further configured to increase a confidence level of the sensor value of a radar sensor when the ego vehicle is traveling on a highway.
12. 8. The system of claim 7, wherein the at least one processor is further configured to reduce a confidence level of the sensor values from the ego vehicle's camera when the ego vehicle is in low light conditions resulting in a black image or high light conditions resulting in a fuzzy white image.
13. A non-transitory computer readable medium having program code executed by a processor for controlling an ego vehicle by determining and using a most reliable input, the non-transitory computer readable medium comprising: program code for determining a confidence level of sensor values from one or more ego vehicle sensors based on ambient conditions of the ego vehicle's environment; program code for determining a level of confidence in the accuracy of an output of the ego vehicle's sensor value based on the confidence level of the sensor value; program code for determining the reliability level with a reliability estimator trained based on previously collected driving data, and determining the reliability level based on the reliability estimator; program code for predicting a trajectory for surrounding vehicles based on the assessment of the confidence level; program code for determining a safe trajectory for the ego vehicle to follow that does not cross the predicted trajectories for the surrounding vehicles; program code for the ego vehicle to execute a failsafe if the confidence level in the safe trajectory is below a threshold confidence level; A non-transitory computer readable medium comprising:
14. updating a confidence level of the sensor value when the ambient conditions change; 14. The non-transitory computer readable medium of claim 13, further comprising program code for updating the level of confidence in the accuracy of outputs of the subsystems of the ego vehicle based on updated sensor values.
15. 14. The non-transitory computer-readable medium of claim 13, wherein the ambient conditions are determined by ambient condition sensors on the ego vehicle and / or are provided to the ego vehicle from a remote device.
16. 14. The non-transitory computer readable medium of claim 13, further comprising program code for reducing a confidence level of the sensor value from a radar sensor when the ego vehicle is in a congested area.
