Dynamic and variable learning by determining and using most trustworthy inputs

Abstract

To provide a method of providing dynamic and variable learning for an ego vehicle by determining and using most trustworthy inputs.SOLUTION: The method includes determining, based on ambient conditions of an environment of the ego vehicle, a level of trustworthiness of sensor values obtained from one or more ego vehicle sensors. The method also includes determining a level of confidence in the accuracy of an output of a subsystem of the ego vehicle. The output of the subsystem is based on the sensor values. The level of confidence is based on the level of trustworthiness of the sensor values.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] Certain aspects of the present disclosure relate generally to vehicle notifications, and more particularly to systems and methods for dynamic and variable learning by determining and using the most reliable inputs. [Background technology]

[0002] A system / subsystem predictor (e.g., a trajectory predictor) may trust inputs from devices (e.g., sensors and / or cameras) that the predictor should not trust, or may not trust inputs from some devices that the predictor should trust. For example, in a well-known vehicle crash, the trajectory predictor (or perhaps another subsystem) trusted sensor inputs from a camera and ignored sensor inputs from LIDAR, even though LIDAR would have provided the most reliable input in that situation. This undesired trust or lack of trust reduces the reliability of the warning system. Summary of the Invention

[0003] A method is described for providing dynamic and variable learning for an ego vehicle by determining and using most reliable inputs. The method includes determining a confidence level for sensor values ​​obtained from one or more ego vehicle sensors based on ambient conditions in the ego vehicle's environment. The method also includes determining a confidence level for the accuracy of outputs of subsystems of the ego vehicle. The subsystem outputs are based on the sensor values. The confidence level is based on the confidence level for the sensor values.

[0004] A system is described that provides dynamic and variable learning for an ego vehicle by determining and using the most reliable inputs. The system includes a memory and one or more processors coupled to the memory. The processor is configured to determine a confidence level for sensor values ​​obtained from one or more ego vehicle sensors based on ambient conditions in the ego vehicle's environment. The processor is also configured to determine a confidence level for the accuracy of outputs of subsystems of the ego vehicle. The subsystem outputs are based on the sensor values. The confidence level is based on the confidence level for the sensor values.

[0005] A non-transitory computer-readable medium having non-transitory program code recorded thereon is described. The program code provides dynamic and variable learning for an ego vehicle by determining and using most reliable inputs. The program code, when executed by a processor, includes program code for determining a confidence level for sensor values ​​obtained from one or more ego vehicle sensors based on ambient conditions in the ego vehicle's environment. The program code also includes program code for determining a confidence level in the accuracy of outputs of subsystems of the ego vehicle, the outputs of the subsystems being based on the sensor values. The confidence level is based on the confidence level for the sensor values.

[0006] This has outlined broadly the features and technical advantages of the present disclosure so that the detailed description that follows may be better understood. Additional features and advantages of the present disclosure are described below. Those skilled in the art should recognize that they may readily utilize this disclosure as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. Those skilled in the art should also recognize that such equivalent constructions do not depart from the teachings of the present disclosure as set forth in the appended claims. The novel features believed characteristic of the present disclosure, both as to its structure and method of operation, together with further objects and advantages, will be better understood from the following description when considered in conjunction with the accompanying drawings. It is to be expressly understood, however, that each of the features is provided for the purpose of illustration and description only and is not intended as a definition of the limits of the present disclosure. [Brief explanation of the drawings]

[0007] The features, nature, and advantages of the present disclosure will become more apparent from the detailed description set forth below when taken in conjunction with the drawings in which like reference characters identify correspondingly throughout.

[0008] [Figure 1A] 1 illustrates an example of a vehicle in an environment, according to aspects of the present disclosure. [Figure 1B] 1 is a top view of an example road including a host vehicle with an external environment and surrounding vehicles located therein, according to one embodiment of the disclosure. FIG. [Figure 1C] 1 is a top view of an example road including a host vehicle with an external environment and surrounding vehicles located therein, according to one embodiment of the disclosure. FIG. [Figure 2] FIG. 1D is a schematic block diagram of the example host vehicle of FIGS. 1B and 1C configured to predict the trajectory of a surrounding vehicle or host vehicle, according to one embodiment of the disclosure. [Figure 3] FIG. 1 illustrates an example hardware implementation for a dynamic and variable learning system, according to aspects of the present disclosure. [Figure 4]1 illustrates a method for providing dynamic and variable learning of an ego vehicle by determining and using the most reliable inputs, according to an aspect of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0009] The detailed description set forth below in connection with the accompanying drawings is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0010] In recent years, various attempts have been made to realize the automatic operation of moving objects such as four-wheeled vehicles. To realize the automatic operation of moving objects, it is important to accurately detect objects such as vehicles, pedestrians, and obstacles around the moving object and to avoid danger while traveling based on the detection results. Of these two factors, object detection technology using various sensors and radar is known as a technology that accurately detects surrounding objects.

[0011] A system / subsystem predictor (e.g., a trajectory predictor) may trust inputs (e.g., sensor inputs) from devices (e.g., a sensor system including sensors and / or cameras) that the predictor should not trust, or may distrust inputs from some devices that the predictor should trust. For example, in well-known vehicle crashes, the trajectory predictor (or perhaps another subsystem) trusted sensor inputs from cameras and ignored sensor inputs from lidar, even though lidar would have provided the most reliable input in that situation. This undesired trust or lack of trust reduces the reliability of the warning system.

[0012] Aspects of the present disclosure are directed to an ego vehicle configured to determine the reliability of sensor values ​​obtained from one or more sensors, cameras, or other devices based on ambient conditions determined by the ego vehicle. For subsystem outputs generated based on the obtained sensor values, the ego vehicle can determine confidence in the accuracy of the outputs generated by the subsystem, where this confidence is based (at least in part) on the reliability of the obtained sensor values. Confidence in the accuracy of the subsystem outputs can be separate from confidence in the fidelity of the model (e.g., separate from confidence in the accuracy of a given prediction generated by a trajectory predictor).

[0013] As surrounding conditions change, the ego vehicle can update the reliability of sensor values, and therefore its confidence in the accuracy of a given subsystem's output.

[0014] In one aspect, the ego vehicle obtains sensor values ​​from one or more sensors. For example, the ego vehicle can be equipped with a variety of sensors, and the ego vehicle can obtain sensor values ​​from one or more of these sensors. The sensors can include, for example, lidar, radar, cameras, photoelectric sensors, or microphones, among many other possibilities.

[0015] The ego vehicle determines one or more surrounding conditions. For example, the ego vehicle can determine whether the ego vehicle is on a highway, in a congested area, in a tunnel, at an intersection, or about to change lanes. The ego vehicle can determine cloud cover, the current time of day, and whether it is daytime or nighttime. The ego vehicle can determine any combination of these or other surrounding conditions.

[0016] The vehicle's sensor system may also include ambient condition sensors that provide sensor signals or ambient condition data associated with ambient conditions such as external temperature, precipitation (such as rain or snow), road conditions (e.g., road roughness or the presence of water or ice), etc.

[0017] The ego vehicle determines the reliability of the obtained sensor values ​​based on the determined surrounding conditions. For example, if the ego vehicle is in a congested or crowded area, the sensor values ​​obtained from the radar may be relatively unreliable because, for example, the radar may be jammed and / or the sensor values ​​may contain an unacceptable amount of noise. On the other hand, in such a situation, the sensor values ​​obtained from the lidar mounted on the roof rack may be relatively reliable. However, if the ego vehicle is traveling on a highway, the sensor values ​​obtained from the radar may be relatively reliable.

[0018] As another example, sensor values ​​obtained from cameras are relatively unreliable in low light conditions (resulting in black images) or high light conditions (resulting in washed-out white images).

[0019] The reliability can be determined by a trust estimator trained based on previously collected driving data. For example, the driving data can indicate a predicted trajectory for the vehicle and an actual trajectory taken by the vehicle. The trajectory can be predicted based on sensor values ​​from one or more sensors that are the same (or similar) as the ego vehicle and obtained in the same (or similar) surroundings. The reliability estimator can be trained based on the calculated difference between the predicted trajectory and the actual trajectory. Based on this training, the reliability estimator can determine the reliability of the sensor values ​​obtained from one or more sensors in the current situation. For example, the reliability can be the difference between a predicted occurrence and an actual occurrence in all modes, or the likelihood of an event occurring given the prediction. The modes can be trajectory position, other vehicle positions, etc.

[0020] In one aspect, the trust estimator generates a trust level for the output of a subsystem of the ego vehicle, the trust level indicating confidence in the accuracy of the output of the subsystem, the output of the subsystem being based on obtained sensor values, and the trust level being generated based (at least in part) on the reliability of the obtained sensor values.

[0021] For example, the ego vehicle can provide sensor values ​​to a trajectory predictor, which generates a predicted trajectory for the ego vehicle (or other vehicles) based on the sensor values. The ego vehicle can determine that the obtained sensor values ​​are relatively unreliable, and therefore the confidence estimator can generate a low confidence level for the predicted trajectory. Thus, even if the trajectory predictor predicts that the ego vehicle is likely to take a given trajectory, the ego vehicle can decide not to pass this predicted trajectory to different subsystems due to the low confidence in the likely trajectory.

[0022] In one aspect, the trajectory predictor generates a confidence level that is separate from the confidence level generated by the confidence estimator. Similar to the confidence level generated by the confidence estimator, the confidence level generated by the trajectory predictor indicates confidence in the accuracy of the predicted trajectory (i.e., output) generated by the trajectory predictor (i.e., subsystem). However, unlike the confidence level generated by the confidence estimator, the confidence level generated by the trajectory predictor does not separately consider the reliability of the obtained sensor values. By considering the surrounding circumstances, the ego vehicle determines not only the fidelity of the model (e.g., trajectory predictor) but also the reliability of the overall subsystem output.

[0023] FIG. 1A illustrates an example of a vehicle 100 (ego or host vehicle) in an environment 150 according to an embodiment of the present disclosure. In this example, the vehicle 100 is an autonomous vehicle. As shown in FIG. 1A, the vehicle 100 can travel along a road 110. A first vehicle 104 can be ahead of the vehicle 100, and a second vehicle 116 can be adjacent to the ego vehicle 100. In this example, the vehicle 100 can include a 2D camera 108, such as a 2D RGB camera, and a lidar sensor 106. Other sensors, such as radar and / or ultrasonic, are also contemplated. Additionally or alternatively, the vehicle 100 can include one or more additional 2D cameras and / or lidar sensors. For example, the additional sensors can be side-facing and / or rear-facing sensors.

[0024] In one configuration, the 2D camera 108 captures a 2D image including objects in a field of view 114 of the 2D camera 108. The lidar sensor 106 can generate one or more output streams. A first output stream can include a 3D point cloud of objects in a first field of view, such as a 360° field of view 112 (e.g., a bird's eye view). A second output stream 124 can include a 3D point cloud of objects in a second field of view, such as a forward-facing field of view.

[0025] The 2D image captured by the 2D camera includes a 2D image of the first vehicle 104 because the first vehicle 104 is in the field of view 114 of the 2D camera 108. As known to those skilled in the art, the lidar sensor 106 uses laser light to sense the shape, size, and position of objects in the environment. The lidar sensor 106 can scan the environment vertically and horizontally. In this example, an artificial neural network (e.g., an autonomous driving system) of the vehicle 100 can extract height and / or depth features from the first output stream. The autonomous driving system of the vehicle 100 can also extract height and / or depth features from the second output stream.

[0026] Information obtained from sensors 106, 108 can be used to assess the driving environment. For example, information obtained from sensors 106, 108 can be used to identify objects that are not visible to vehicle 100. This information can be used to generate one or more location-specific notifications.

[0027] 1B and 1C, a top view of an example roadway including a host vehicle 100 having an external environment 12 and surrounding vehicles 14 located therein is shown. A system may include the host vehicle 100. The host vehicle 100 may be any suitable type of vehicle. The host vehicle 100 may be any form of at least partially motorized transportation. In one or more arrangements, the host vehicle 100 may be an automobile. While arrangements are described herein with respect to an automobile, it should be understood that examples and implementations are not limited to automobiles. In one or more arrangements, the host vehicle 100 may be a watercraft, aircraft, spacecraft, golf cart, motorcycle, and / or any other form of at least partially motorized transportation.

[0028] The host vehicle 100 may be located anywhere. For example, the host vehicle 100 may be traveling along a road. The host vehicle 100 may have an associated external environment 12. The external environment 12 may be an area surrounding the host vehicle 100 and / or any portion thereof. One or more objects may be located in the external environment 12 of the host vehicle 100. For example, the one or more objects may be vehicles surrounding the host vehicle 100 that are also traveling along a road. The external environment 12 of the host vehicle 100 may include any number of surrounding vehicles 14. While only one surrounding vehicle 14 is shown in FIGS. 1B and 1C, it will be understood that the arrangements described herein are not limited in this respect. In practice, more than one surrounding vehicle 14 may be located in the external environment 12 of the host vehicle 100. Additionally or alternatively, there may be no surrounding vehicles in the external environment 12 of the host vehicle 100.

[0029] As described herein, the host vehicle 100 can monitor the external environment 12 of the host vehicle 100. The host vehicle 100 can detect objects located in the external environment 12 of the host vehicle 100. The host vehicle 100 can classify the objects as surrounding vehicles 14. The host vehicle 100 can predict a trajectory 16 of the surrounding vehicles 14. The predicted trajectory 16 can be a series of actions, directions, orientations, and / or driving paths for the surrounding vehicles 14. As discussed below, the predicted trajectory 16 can be determined based on both one or more predetermined vehicle characteristics for the surrounding vehicles 14 and one or more current driving characteristics. The predetermined vehicle characteristics can be based on the type of surrounding vehicle 14. The current driving characteristics can be obtained from sensor data from a sensor system of the host vehicle 100.

[0030] Referring now to FIG. 2, a schematic diagram of an example host vehicle 100 configured to predict a trajectory (e.g., trajectory 16) of the host vehicle 100 or surrounding vehicles 14 shown in FIGS. 1B and 1C is shown. The host vehicle 100 may include various elements. Some of the possible elements of the host vehicle 100 are shown in FIG. 2 and described herein. However, it should be understood that the host vehicle 100 need not include all of the elements shown in FIG. 2 or described herein. The host vehicle 100 may include any combination of the various elements shown in FIG. 2. Furthermore, the host vehicle 100 may include elements in addition to those shown in FIG. 2. Furthermore, although various elements are shown in FIG. 2 as being located within the host vehicle 100, it should be understood that one or more of these elements may be located outside the host vehicle 100. Furthermore, the elements shown may be physically separated from one another by large distances.

[0031] The host vehicle 100 may include one or more processors 18. The processor 18 may be any component or group of components configured to execute any of the processes described herein or any type of instructions that cause such processes to be executed. The processor 18 may be implemented with one or more general-purpose and / or special-purpose processors. Examples of suitable processors 18 include microprocessors, microcontrollers, digital signal processors, and other circuits capable of executing software. Further examples of suitable processors 18 include, but are not limited to, central processing units (CPUs), array processors, vector processors, digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), application-specific integrated circuits (ASICs), programmable logic circuits, and controllers. The processor 18 may include at least one hardware circuit (e.g., an integrated circuit) configured to execute instructions contained in program code. In an arrangement with multiple processors 18, such processors may operate independently of each other, or one or more processors may operate in combination with each other. In one or more arrangements, processor 18 may be the main processor of host vehicle 100. For example, processor 18 may be an electronic control unit (ECU).

[0032] The host vehicle 100 may include a computer-readable medium. In one or more arrangements, the computer-readable medium may be memory 19. The memory 19 may include volatile / non-volatile memory. Examples of suitable memory 19 include RAM (random access memory), flash memory, ROM (read-only memory), PROM (programmable read-only memory), EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), registers, magnetic disk, optical disk, hard drive, or any other suitable storage medium, or any combination thereof. The memory 19 may include instructions in program code stored thereon. Such instructions may be executed by the processor 18 and / or one or more modules of the host vehicle 100. In one or more arrangements, the memory 19 may be a component of the processor 18. In one or more arrangements, the memory 19 may be operatively connected to the processor 18 and / or one or more modules of the host vehicle 100 for use by them. Functionally connected can include direct or indirect connections, including connections without direct physical contact.

[0033] The host vehicle 100 may include one or more databases 20 for storing one or more types of data. The databases 20 may be components of the memory 19, the processor 18, or the databases 20 may be operatively connected to the processor 18 and / or the memory 19 for use therewith.

[0034] The host vehicle 100 may also include a sensor system 22. The sensor system 22 may include one or more sensors. The one or more sensors may be configured to monitor something in the external environment 12 (shown in FIGS. 1B and 1C) of the host vehicle 100. In one or more arrangements, the one or more sensors may be configured to monitor in real time. Real time may be a level of processing response that a user or system perceives as being rapid enough for a particular process or decision to be made, allowing the processor to keep up with some external process. The sensor system 22 may be located anywhere in or on the host vehicle 100. The sensor system 22 may include, at least in part, existing systems of the host vehicle 100, such as backup sensors, lane keeping sensors, and / or front sensors, to name a few possibilities.

[0035] In deployments where sensor system 22 includes multiple sensors, the multiple sensors may be distributed in any suitable manner around host vehicle 100. The sensors may operate independently of one another or in combination with one another. In such cases, two or more sensors may form a sensor network.

[0036] The sensor system 22 may include one or more sensors configured to sense the external environment 12, or a portion thereof, of the host vehicle 100. For example, the sensor system 22 may be configured to acquire data of at least a portion in front of and / or at least a portion behind the external environment 12 of the host vehicle 100. For example, the sensor system 22 may monitor a portion in front of the host vehicle 100 along the longitudinal direction α and / or may monitor a portion behind the host vehicle 100 along the longitudinal direction α.

[0037] Additionally or alternatively, the sensor system 22 can be configured to acquire data of at least a lateral portion of the external environment 12 of the host vehicle 100. The lateral portion can be, for example, a portion of the external environment 12 located between a front portion and a rear portion of the host vehicle 100. For example, the sensor system 22 can be configured to monitor a left and / or right portion of the host vehicle 100 along the lateral direction β.

[0038] The sensor system 22 may be operatively connected to the processor 18, database 20, memory 19, and / or any other components and / or modules of the host vehicle 100. Various examples of possible sensors for the sensor system 22 are described herein. However, it will be understood that the disclosure is not limited to the particular sensors described. The sensor system 22 may include categories of sensors such as active sensors 24, passive sensors 26, and / or vehicle motion sensors 28.

[0039] The active sensor 24 can be any sensor or group of sensors configured to actively emit a signal for the purpose of determining the quantity or property of something. For example, the active sensor 24 can emit an electromagnetic signal, a radio signal, a laser signal, etc. Such a signal can be emitted into the external environment 12 of the host vehicle 100. Examples of such active sensors 24 can include a radar sensor 30, a lidar sensor 32, etc.

[0040] In one or more arrangements, the active sensor 24 may include one or more radar sensors 30. The radar sensor 30 may be any device, component, and / or system capable of detecting something, at least in part, using radio signals. The radar sensor 30 may be configured to detect the presence of one or more objects in the external environment 12 of the host vehicle 100, the position of each detected object relative to the host vehicle 100, the distance between each detected object and the host vehicle 100 in one or more directions (e.g., the longitudinal direction α, the lateral direction β, and / or other directions), the altitude of each detected object, the speed of each detected object, and / or the movement of each detected object. The radar sensor 30, or data obtained thereby, may determine or be used to determine the speed, position, and / or orientation of objects in the external environment 12 of the host vehicle 100. The radar sensor 30 may have three-dimensional coordinate data associated with the objects.

[0041] In one or more arrangements, the active sensor 24 may include a lidar sensor 32. The lidar sensor 32 may be any device, component, and / or system capable of detecting something, at least in part, using an electromagnetic signal. In one or more arrangements, the electromagnetic signal may be a laser signal. The lidar sensor 32 may include a laser source and / or a laser scanner configured to emit a laser signal and a detector configured to detect reflections of the laser signal. The lidar sensor 32 may be configured to operate in a coherent or non-coherent detection mode.

[0042] The lidar sensor 32 may be configured to detect the presence of one or more objects in an environment external to the host vehicle 100, the position of each detected object relative to the host vehicle 100, the distance in one or more directions between each detected object and the host vehicle 100, the altitude of each detected object, the velocity of each detected object, and / or the movement of each detected object. An example lidar sensor 32 may include, for example, a Velodyne® lidar system.

[0043] Passive sensor 26 can be any sensor or group of sensors configured to receive a signal broadcast from some entity independent of passive sensor 26. Such received signal can be broadcast from some entity located in external environment 12 of host vehicle 100. Additionally or alternatively, such received signal can be broadcast from any component or group of components of host vehicle 100 independent of passive sensor 26. For example, passive sensor 26 can receive electromagnetic signals, radio signals, laser signals, etc. Examples of such passive sensors 26 can include positioning sensors 34, cameras 36, IMU sensors 38, CAN sensors 40, etc.

[0044] In one or more arrangements, the passive sensors 26 may include a positioning sensor 34. The positioning sensor 34 may be one or more components or a group of components configured to determine the geographic location of the host vehicle 100.

[0045] The positioning sensor 34 may include a global positioning system, a local positioning system, and / or a geographic location system. The positioning sensor 34 may include a transceiver configured to estimate the position of the host vehicle 100 relative to the Earth. For example, the positioning sensor 34 may include a GPS transceiver for determining the vehicle's latitude, longitude, and / or altitude. The positioning sensor 34 may use other systems for determining the position of the host vehicle 100 (e.g., a laser-based location system, an inertial-aided GPS, and / or a camera-based location). It should be understood that many different systems or components may be substituted for or supplement the positioning sensor 34 to determine the position of the host vehicle 100 without departing from the scope of this disclosure.

[0046] In one or more arrangements, the passive sensors 26 may include one or more cameras. The cameras 36 may be any device, component, and / or system capable of capturing visual data. The visual data may include video and / or image information / data. The visual data may be in any suitable form. In one or more arrangements, the visual data may include a heat signature, thermal image, and / or thermal footage of portions of the external environment 12 of the host vehicle 100.

[0047] Camera 36 can be any suitable type of camera. For example, camera 36 can be a high-resolution camera, a high dynamic range (HDR) camera, an infrared camera, and / or a thermal imaging camera.

[0048] In one or more arrangements, the camera 36 may be positioned to capture visual data from at least a portion of the environment 12 external to the host vehicle 100. In one or more arrangements, the camera 36 may be positioned to capture visual data from at least a rear portion of the environment external to the host vehicle 100. As a further example, the camera 36 may be positioned to capture visual data from at least a left portion and / or a right portion of the environment external to the host vehicle 100.

[0049] The cameras 36 may be located in any suitable portion of the host vehicle 100. For example, the cameras 36 may be located within the host vehicle 100. One or more of the cameras 36 may be located on the exterior of the host vehicle 100. One or more of the cameras 36 may be located on the exterior of the host vehicle 100 or may be exposed to the exterior. Additionally or alternatively, one or more of the cameras 36 may be located on the side of the host vehicle 100. As another example, one or more of the cameras 36 may be located on the roof of the host vehicle 100.

[0050] In one or more arrangements, camera 36 can be one or more backup cameras. The backup cameras can be configured to capture visual data of the portion of external environment 12 rearward of host vehicle 100. In some arrangements, the one or more backup cameras can capture visual data of at least a portion of the portion of external environment rearward when host vehicle 100 is not in reverse gear mode and / or is not moving backward. Alternatively or additionally, camera 36 can include other cameras available for use in host vehicle 100.

[0051] In one or more arrangements, the passive sensors 26 may include an inertial measurement unit (IMU) sensor 38. The IMU sensor 38 may be operatively connected to an IMU (not shown) of the host vehicle 100. The IMU may be one or more mechanisms, devices, elements, components, systems, applications, and / or combinations thereof configured to determine inertial characteristics of the host vehicle 100. Such inertial characteristics may include, for example, velocity, orientation, gravity, etc. The IMU sensor 38 may determine the inertial characteristics of the host vehicle 100 using an accelerometer, a gyroscope, etc. The IMU sensor 38 may be configured to monitor data received from the IMU of the host vehicle 100.

[0052] In one or more arrangements, the passive sensor 26 may include a controller area network (CAN) sensor 40. The CAN sensor 40 may be operatively connected to a CAN bus (not shown) of the host vehicle 100. The CAN bus may be one or more components operatively connected to each other and communicating over a network in the host vehicle 100. The network may be a wired network and / or a wireless network. The CAN sensor 40 may be configured to monitor, access, and / or evaluate data transmitted on the CAN bus of the host vehicle 100.

[0053] Vehicle motion sensor 28 can be any component or group of components configured to detect the position, velocity (as a vector), speed (as a scalar), and / or acceleration of host vehicle 100. Vehicle motion sensor 28 can use a number of sensors already present on the vehicle to detect the position, velocity (as a vector), speed (as a scalar), and / or acceleration of host vehicle 100. Such sensors can include, for example, a tachometer, steering angle sensors, wheel angle sensors, wheel speed sensors, or any other sensor capable of detecting the position, velocity (as a vector), speed (as a scalar), and / or acceleration of host vehicle 100.

[0054] The sensor system 22 may be configured to monitor the environment 12 external to the host vehicle 100, including objects detected in the environment 12. In response to detecting an object in the environment 12 external to the host vehicle 100, the host vehicle 100 may classify the object.

[0055] In one or more arrangements, the host vehicle 100 can include an object classification database 42. The object classification database 42 can include data corresponding to classifications of various objects. For example, the object classification database 42 can include data corresponding to characteristics of various vehicles, on-road or off-road objects, and road attributes. Such characteristics can be the shapes of images stored in the object classification database 42, typical readings from one or more sensors in the sensor system 22 that indicate particular types of objects, and / or any other form of data useful for classifying objects. Examples of various vehicles, on-road or off-road objects, and road attributes include, for example, vehicles, motorcycles, trees, pedestrians, bicyclists, animals, road signs, obstacles, or any other objects typically found on or along roads.

[0056] The features stored in the object classification database 42 can be compared to data obtained by the sensor system 22. For example, images of vehicles stored in the object classification database 42 can be compared to images of objects in the external environment 12 obtained by the camera 36. Such comparisons can be performed by image processing software commonly known in the art. The image processing software is implemented on the processor 18. In response to a comparison that the vehicle is substantially similar to an object in the external environment 12, the host vehicle 100 can determine that the object in the external environment 12 is a surrounding vehicle 14.

[0057] Additionally or alternatively, object classification database 42 can store data readings that are typical for particular types of objects. For example, object classification database 42 can store data from typical lidar sensor readings indicating the presence of trees. The data from the typical lidar sensor readings can be compared to data received from lidar sensor 32. In response to the comparison of the typical lidar sensor readings being substantially similar to an object in external environment 12 detected by lidar sensor 32, host vehicle 100 can determine that the object in the external environment is a tree. Substantially similar can be, for example, within one standard deviation, within one-half a standard deviation, within one-quarter of a standard deviation, etc. Although the two foregoing examples are provided for purposes of clarity, any type of data can be stored in object classification database 42 for comparison with data obtained via sensor system 22. As a result of the comparison, the object detected in external environment 12 of host vehicle 100 can be classified.

[0058] The object classification database 42 may further include data indicating various types of vehicles. Examples of vehicle types may include sedans, sport utility vehicles (SUVs), convertibles, pickup trucks, semi-trucks, campers, motorcycles, tractors, a particular brand or model of a vehicle, and / or any other subclass of vehicle. Thus, in response to detecting that an object is a surrounding vehicle 14, the host vehicle 100 may determine the type of the surrounding vehicle 14. For example, the processor 18 may receive data from the sensor system 22. The data may indicate that an object located in the external environment 12 is a surrounding vehicle 14. In response to determining that the object is a surrounding vehicle 14, the processor 18 may compare the data to various types of vehicles stored in the object classification database 42. The processor 18 can determine that the surrounding vehicle 14 is, for example, a convertible based on a comparison of the data received from the sensor system 22 with data stored in the object classification database 42 that indicates typical sensor readings when a convertible is present in the external environment 12.

[0059] The host vehicle 100 may further include a predetermined vehicle characteristic database 44. The predetermined vehicle characteristic database 44 may include one or more predetermined vehicle characteristics for various types of vehicles. The predetermined vehicle characteristic may indicate a driving behavior for a type of vehicle. The predetermined vehicle characteristic may indicate at least one driving behavior associated with multiple vehicles of the same type. For example, a predetermined vehicle characteristic for a convertible may be more aggressive compared to a predetermined vehicle characteristic for a semi-truck.

[0060] The predetermined vehicle characteristic can be a number on a scale, a percentage of aggressiveness, a weighting factor compared to a standard sedan, or any other method of characterizing a driving style. For example, the predetermined vehicle characteristic for a standard sedan can be 50% aggressive, while the predetermined vehicle characteristic for a convertible can be 80% aggressive. Additionally or alternatively, the predetermined vehicle characteristic for a standard sedan can be a 1.0 aggressiveness factor, while the predetermined vehicle characteristic for a semi-truck can be a 0.5 aggressiveness factor. While two examples are provided above, it should be understood that various weighting factors and percentages can be used to accommodate different classes of vehicles, and the weighting factors for a particular class of vehicle are not intended to be limited to the examples provided. In practice, many different methods of characterizing driving style can be used for a particular class of vehicle.

[0061] Additionally or alternatively, predetermined vehicle characteristics can be classified as driving behaviors for particular maneuvers. For example, predetermined vehicle characteristics for SUVs can include data indicating that they tend to drive faster than other vehicles in inclement weather, but drive at an average speed in normal weather, prefer the center lane, and change lanes at a slower rate than other vehicles. Also, predetermined vehicle characteristics for convertibles can include data indicating that they tend to drive slower than other vehicles in inclement weather, but drive at a faster speed, prefer the fast lane, and change lanes at a faster rate than other vehicles in normal weather. It should be noted that while the above two examples are provided, the present disclosure is not limited to these two examples. In fact, any method for characterizing various types of vehicles can be used in the predetermined vehicle characteristics database 44. Additionally, different levels of characterization can be incorporated, including characterization based on vehicle classification (e.g., sedan, SUV, convertible, pickup truck, etc.), vehicle brand (e.g., Toyota, Lexus, Honda, Ford, Dodge, etc.), vehicle model (e.g., Toyota Camry, Lexus IS, Honda Accord, Ford Flex, Dodge Charger, etc.), and / or any other level of characterization that can be used to differentiate driving styles based on the particular type of vehicle.

[0062] The host vehicle 100 may include various subsystems (e.g., a trajectory predictor). The various subsystems may be represented as various modules 46. Examples of subsystems include a trajectory predictor, such as a safe trajectory determination module 60 or a surrounding vehicle trajectory prediction module 52. Other subsystems may be directed to driver actions, fail-safes, database updates, etc. The various modules 46 perform various tasks in the host vehicle 100. The modules 46 may be implemented as computer-readable program code that, when executed by the processor 18, implements one or more of the various processes described herein. Such computer-readable program code may be stored in the memory 19. The modules 46 may be components of the processor 18, or the modules 46 may be executed on and / or distributed among other processing systems to which the processor 18 is operatively connected. The modules 46 may include instructions (e.g., program logic) executable by the processor 18. Additionally or alternatively, the memory 19 may include such instructions. Various modules 46 may be operatively connected to the processor 18, the database 20, and / or the sensor system 22. Various examples of modules 46 that the host vehicle 100 may include are described herein.

[0063] In one or more arrangements, the host vehicle 100 may include a driver behavior module 48. The driver behavior module 48 may receive predetermined vehicle characteristics from the predetermined vehicle characteristics database 44 to determine the driver behavior of the identified surrounding vehicle 14.

[0064] Additionally or alternatively, the driver behavior module 48 can receive data from the sensor system 22. The data received from the sensor system 22 can indicate the behavior of the driver of the identified surrounding vehicle 14. In such an example, the data received from the sensor system 22 can be used to both classify objects located in the external environment and determine the behavior of the driver of the surrounding vehicle 14. The driver behavior module 48 can use both the predetermined vehicle characteristics received from the predetermined vehicle characteristics database 44 and the data received from the sensor system 22 to determine the driver behavior. Such data received from the sensor system 22 can determine or be used to determine the current driving characteristics of the surrounding vehicle 14. The current driving characteristics can indicate one or more driving behaviors. For example, the current driving characteristics can be a number on a scale, a percentage of aggressiveness, a weighting factor compared to a standard sedan, or any other method of characterizing one or more driving behaviors with respect to the surrounding vehicle 14 based on the data obtained from the sensor system 22. Examples of such data include direction, speed, acceleration, changes in lane position, and / or any other type of data that may be useful for determining the behavior of the driver of the surrounding vehicle 14.

[0065] In one or more configurations, the current driving characteristics can be substantially the same as the predetermined vehicle characteristics for the surrounding vehicle 14. In such an example, the surrounding vehicle 14 has one or more driving behaviors that are consistent with the type of surrounding vehicle. In one or more configurations, the current driving characteristics can differ from the predetermined vehicle characteristics. In such an example, the surrounding vehicle 14 does not exhibit one or more driving behaviors that are consistent with the type of surrounding vehicle. The driver behavior module 48 can use both the predetermined vehicle characteristics received from the predetermined vehicle characteristics database 44 and the data received from the sensor system 22 to determine the driver's behavior.

[0066] The host vehicle 100 may further include an identified vehicle database 50. The host vehicle 100 may record identifiers for the surrounding vehicles 14. Such identifiers may include, for example, license plate numbers, vehicle identification numbers, or any other type of identifier typically used to distinguish between individual vehicles. The identifiers may be stored in the identified vehicle database 50. Additionally or alternatively, predetermined vehicle characteristics for the surrounding vehicles 14 may also be stored in the identified vehicle database 50. In such an example, the identified vehicle characteristics may be associated with an identifier, and both the identifier and the associated predetermined vehicle characteristic may be stored in the identified vehicle database 50.

[0067] In one or more arrangements, the host vehicle 100 may include a surrounding vehicle trajectory prediction module 52. The surrounding vehicle trajectory prediction module 52 may predict a trajectory for the surrounding vehicle 14 based on one of predetermined vehicle characteristics for the surrounding vehicle 14 and current driving characteristics based on data received from the sensor system 22. Referring now to FIGS. 1B, 1C, and 2, the surrounding vehicle 14 is shown in a left lane relative to the host vehicle 100. The host vehicle 100 may determine the type of the surrounding vehicle 14, for example, as a convertible. The host vehicle 100 may also use data received from the sensor system 22 to detect that the surrounding vehicle 14 has accelerated relative to the host vehicle 100 and that the surrounding vehicle 14 has shifted its position laterally toward a lane marking directly adjacent to the host vehicle 100 and between the host vehicle 100 and the surrounding vehicle 14. The sensor system may also include lane marking sensors (not shown) used to monitor and sense vehicles or objects (e.g., other vehicles) within the lane, or other sensors responsive to road signs, traffic signals, road signs, or the presence of emergency vehicles. Such sensor data establishes the path of the lane within the world model and can therefore be used for predictive path calculation.

[0068] Additionally or alternatively, the host vehicle 100 can access map data from the navigation unit 54 of the host vehicle 100 indicating that the left lane ends ahead. The host vehicle 100 can use the predetermined vehicle characteristic database 44, data received from any one of the sensor systems 22, and map data from the navigation unit 54 to predict the trajectory 16 of the surrounding vehicle 14. The navigation unit 54 can be integrated into the head unit (not shown) of the host vehicle 100, or the navigation unit 54 can be a standalone or aftermarket component. The navigation unit 54 can include map data stored therein (e.g., stored in a database included in the navigation unit 54), or the host vehicle 100 can further include a map database 56. Additionally or alternatively, the navigation unit 54 can be integrated into the occupant's mobile device and can store map data therein. In such an example, the host vehicle 100 can access the occupant's mobile device and obtain the map data stored therein.

[0069] Additionally, each received data type may have a corresponding confidence level. Referring now to FIG. 2 , the surrounding vehicle trajectory prediction module 52 may evaluate confidence levels for predetermined vehicle characteristics, current driving characteristics, and map data. Such confidence levels may be used to optimize the predicted trajectory 16 for the surrounding vehicles 14. For example, if some of the sensors in the sensor system 22 are not functioning optimally or are in a fault state, the current driving characteristics may have a lower confidence level. The confidence level may be a number on a scale, a percentage confidence, a weighting factor, or any other method of characterizing the confidence level in the data.

[0070] The surrounding vehicle trajectory prediction module 52 can evaluate the confidence level for any of the predetermined vehicle characteristics, current driving characteristics, and map data for the surrounding vehicles 14 and predict a predicted trajectory 16 for the surrounding vehicles 14. In one or more configurations, the predicted trajectory 16 can also have a corresponding confidence level. Much like the confidence levels described above, the confidence level for the predicted trajectory 16 can be a number on a scale, a percentage confidence, a weighting factor, or any other way of characterizing the confidence level in the data.

[0071] In one or more arrangements, the host vehicle 100 may include a database update module 58. The database update module 58 may be configured to update any of the databases 20. Such updates may occur based on a variety of factors. In one or more arrangements, the database update module 58 may receive a prompt for an update. Such a prompt may be received in a manner similar to that typically used to update software and databases commonly known in the art. For example, the prompt may be received at a dealership or wirelessly.

[0072] In one or more arrangements, the database updating module 58 can determine that an update is needed. Such a determination can be made for a particular type of vehicle or for a specifically identified vehicle. Thus, in such an example, the database updating module 58 can update both the predetermined vehicle characteristic database 44 and the identified vehicle database 50. The database updating module 58 can compare predetermined vehicle characteristics for surrounding vehicles 14 with the identified driving characteristics of the surrounding vehicles 14. If the predetermined vehicle characteristics and the identified driving characteristics differ, the database updating module 58 can determine that an update is needed for either the predetermined vehicle characteristic database 44 or the identified vehicle database 50. For example, the database updating module 58 can update the identified vehicle database 50 in response to the identified driving characteristics differing from the predetermined vehicle characteristics and the identified driving characteristics not being, for example, within one standard deviation of the predetermined vehicle characteristic, within two standard deviations of the predetermined vehicle characteristic, within three standard deviations of the predetermined vehicle characteristic, etc. The database updating module 58 can update the predetermined vehicle characteristics stored in the identified vehicle database 50 corresponding to the identifiers corresponding to the surrounding vehicles 14. Additionally or alternatively, the database update module 58 may update the predetermined vehicle characteristic database 44 in response to the identified driving characteristic differing from the predetermined vehicle characteristic but being, for example, within one standard deviation of the predetermined vehicle characteristic.

[0073] The host vehicle 100 may include a safe trajectory determination module 60. The safe trajectory determination module 60 may determine a safe trajectory for the host vehicle 100 to follow. A safe trajectory may be, for example, a trajectory in which the host vehicle 100 does not cross the predicted trajectory 16 for the surrounding vehicle 14. In some examples, such a safe trajectory may be following a previously planned trajectory for the host vehicle 100 when the previously planned trajectory of the host vehicle 100 is determined to be safe. The safe trajectory may be determined using any of the predicted trajectories from the surrounding vehicle trajectory prediction module 52, current driving characteristics from the sensor system 22, predetermined vehicle characteristics from the predetermined vehicle characteristics database 44 based on the type of surrounding vehicle 14, map data from the navigation unit 54, and any other data usable to determine a safe trajectory for the surrounding vehicle. The safe trajectory may include a confidence level. The confidence level for a safe trajectory may be a number on a scale, a percentage confidence, a weighting factor, or any other method of characterizing the confidence level in the data.

[0074] The host vehicle 100 can follow a safe trajectory in response to one or more of the sensors in the sensor system 22 changing from an active state to a fault state. An active state can be actively, consistently, regularly, and / or predictably receiving reliable data from the sensors. A fault state can be inconsistently receiving data, receiving unreliable or extremely outlying data, a sensor deactivating, a sensor being in a passive state, a sensor being in a sleep mode, etc. An example of a sensor being in a fault state can be when it is snowing outside the vehicle and some sensors (e.g., camera 36, ​​radar sensor 30) may be receiving unreliable data. In these examples, the sensors receiving unreliable data can be considered to be in a fault state. In response to one or more of the sensors in the sensor system 22 being in a fault state, the host vehicle 100 can follow a safe trajectory.

[0075] Additionally or alternatively, host vehicle 100 can follow a safe trajectory in response to one of the sensor categories (e.g., active sensor 24, passive sensor 26, vehicle motion sensor 28) being in a failed state. In one or more arrangements, host vehicle 100 can use the safe trajectory and the sensors in an active state when one or more of the categories are in a failed state. For example, when active sensor 24 is in a failed state and passive sensor 26 and vehicle motion sensor 28 are in an active state, safe trajectory determination module 60 can determine a safe trajectory based on the predicted trajectory as determined by surrounding vehicle trajectory prediction module 52, passive sensor 26, and vehicle motion sensor 28. Also, when passive sensor 26 is in a failed state and active sensor 24 and vehicle motion sensor 26 are in an active state, safe trajectory determination module 60 can determine a safe trajectory based on the predicted trajectory as determined by surrounding vehicle trajectory prediction module 52, active sensor 24, and vehicle motion sensor 28. Additionally, when both the active sensor 24 and the passive sensor 26 are in a failed state and the vehicle motion sensor 28 is in an active state, the safe trajectory determination module 60 can determine a safe trajectory based on the predicted trajectory as determined by the surrounding vehicle trajectory prediction module 52 and the vehicle motion sensor 28.

[0076] The host vehicle 100 can include a failsafe module 62. In examples where the safe trajectory includes a confidence level, the failsafe module 62 can compare the confidence level for the safe trajectory to a threshold. The threshold can be, for example, a numeric threshold on a scale, a percentage confidence threshold, a weighting factor threshold, or any other method of setting a threshold for a confidence level in data. If the confidence level for the safe trajectory is below the confidence level threshold, for example, the host vehicle 100 can implement a failsafe. Such a threshold can be a minimum level of confidence; otherwise, the host vehicle 100 implements a failsafe. Alternatively, a maximum level of untrustworthiness (e.g., the inverse of trustworthiness) can be used as a threshold, and if the confidence level exceeds the maximum level of untrustworthiness threshold, the host vehicle 100 implements a failsafe. The failsafe can be, for example, stopping the host vehicle 100, moving the host vehicle 100 to the side of the road, slowing the host vehicle 100, idling the host vehicle 100, or any other failsafe for the host vehicle 100 to safely disable, stop, and / or prohibit the host vehicle 100 from moving.

[0077] The vehicle's sensor system 22 may also include ambient condition sensors 64 that provide sensor signals related to ambient conditions, such as external temperature, precipitation (such as rain, snow, etc.), road conditions (e.g., road roughness or the presence of water or ice), etc. The ambient condition sensor data may be used to determine the reliability of other sensor values ​​and may be used to modify the vehicle's model or the behavior of other objects. In some aspects, ambient conditions are provided to the vehicle over a network.

[0078] The sensor system 22 can also be used to monitor the vehicle itself to sense vehicle parameters such as, for example, vehicle speed, acceleration (in one or more dimensions), throttle position, engine RPM, braking operation (pedal position and / or ABS operation), steering input, yaw rate, wheel slip, passenger occupancy and weight (e.g., to modify vehicle behavior models), other engine inputs, driving component configuration (such as road wheel angle for an automobile), control surface orientation and configuration (such as ailerons or rudder) for an aircraft, rudder orientation for a boat, nozzle configuration for a spacecraft, or other parameters. The sensor system 22 can also include GPS or other position sensors for vehicle position, speed, and altitude measurement. Vehicle speed can be determined as the time derivative of position, and acceleration can be determined as the time derivative of velocity. The sensor system 22 can also monitor the vehicle operator, for example, using eye or gaze tracking, or by monitoring biometric parameters, such as biometric parameters related to fatigue levels.

[0079] The sensor system 22 can also receive data from remote sources, for example, via wireless communication links, or can obtain data from other sources. For example, it can receive weather, road, traffic, radar, or other data. The sensors can be located remotely from the vehicle, such as at the roadside, or embedded in the road and can transmit sensor data wirelessly to the vehicle. The sensors can point in the direction forward, rearward, or to the sides of the vehicle, or any combination thereof, and include omnidirectional sensors.

[0080] For example, a sensor fusion component can be used to combine image data from multiple image sensors of the same or different types into a representation of the vehicle environment. A MIMO processor can be used as a sensor fusion component.

[0081] The safe trajectory determination module 60 or the surrounding vehicle trajectory prediction module 52 may trust inputs from sensor systems 22 that the safe trajectory determination module 60 or the surrounding vehicle trajectory prediction module 52 should not trust, or may not trust inputs from some sensors that it should trust. This reliance on these sensors may be detrimental to the reliability of the safe trajectory determination module 60 or the surrounding vehicle trajectory prediction module 52.

[0082] Accordingly, aspects of the present disclosure are directed to an ego vehicle (e.g., ego or host vehicle 100) configured to determine the reliability of sensor values ​​or data obtained from sensor system 22 based on ambient conditions determined by ego vehicle 100. For subsystem outputs generated based on the obtained sensor values ​​and / or other obtained values ​​(e.g., output from database 20), ego vehicle 100 can determine a confidence in the accuracy of the output generated by the subsystem that is based (at least in part) on the reliability of the obtained sensor values. Confidence in the accuracy of the subsystem output can be separate from confidence in the fidelity of the model (e.g., separate from confidence in the accuracy of a given prediction generated by a trajectory predictor).

[0083] As ambient conditions change, ego vehicle 100 can update the reliability of sensor values ​​from ambient condition sensors 64, and therefore, can update its confidence in the accuracy of the output of a given subsystem. In one embodiment, ego vehicle 100 obtains sensor values ​​from sensor system 22 and / or database values ​​from database 20. For example, ego vehicle 100 can be equipped with a variety of sensors and databases, and ego vehicle 100 can obtain sensor and / or database values ​​from one or more of these sensors and databases. Sensors can include, for example, lidar sensors 32, radar sensors 30, and cameras 36, among many other possibilities.

[0084] Ego vehicle 100 determines one or more ambient conditions using ambient condition sensors 64 and / or ambient condition information received wirelessly from a remote device. For example, processor 18 of ego vehicle 100 can determine whether the ego vehicle is on a highway, in a congested area, in a tunnel, at an intersection, or about to change lanes based on sensor values ​​and other information. Ego vehicle 100 can determine cloud cover, the current time of day, and whether it is daytime or nighttime based on sensor values ​​from the ambient sensors. Ego vehicle 100 can determine any combination of these or other ambient conditions.

[0085] Ego vehicle 100 determines the reliability of the active sensors 24, passive sensors 26, vehicle motion sensors 28, and / or the obtained sensor values ​​of database 20 based on the determined ambient conditions from ambient condition sensors 64. For example, if ego vehicle 100 is in a congested or crowded area, the sensor values ​​obtained from radar sensor 30 may be relatively unreliable because, for example, radar sensor 30 may be obstructed and / or the sensor values ​​may contain an unacceptable amount of noise. On the other hand, in such a situation, the sensor values ​​obtained from lidar sensor 32 mounted on the roof rack of ego vehicle 100 may be relatively reliable. However, if ego vehicle 100 is traveling on a highway, the sensor values ​​obtained from radar sensor 30 may be relatively reliable.

[0086] As another example, the sensor values ​​obtained from camera 36 are relatively unreliable in low light conditions (resulting in a black image) or high light conditions (resulting in an indistinct white image).

[0087] The reliability can be determined by a reliability estimator trained based on previously collected driving data. For example, the driving data can indicate a predicted trajectory for the vehicle and an actual trajectory taken by the vehicle. The trajectory can be predicted based on sensor values ​​from one or more sensors that are the same (or similar) as ego vehicle 100 and obtained in the same (or similar) surroundings. The reliability estimator can be trained based on calculated differences between the predicted trajectory and the actual trajectory. Based on this training, the reliability estimator can determine the reliability of the sensor values ​​obtained from one or more sensors in the current situation. In one embodiment, the reliability estimator can be part of processor 18. In another embodiment, the reliability estimator can be part of module 46.

[0088] In one embodiment, the trust estimator generates a trust level for the output of a subsystem of ego vehicle 10. The trust level indicates the confidence in the accuracy of the output of the subsystem (e.g., surrounding vehicle trajectory prediction module 52 or safe trajectory determination module 60). The output of the subsystem is based on obtained sensor values, and the trust level is generated based (at least in part) on the reliability of the obtained sensor values. In one embodiment, the trust estimator can be part of processor 18. In another embodiment, the trust estimator can be part of module 46.

[0089] For example, ego vehicle 100 can provide sensor values ​​to a trajectory predictor (e.g., surrounding vehicle trajectory prediction module 52 or safe trajectory determination module 60), which generates a predicted trajectory for ego vehicle 100 (or another vehicle) based on the sensor values. Ego vehicle 100 can determine that the obtained sensor values ​​are relatively unreliable, and therefore the confidence estimator can generate a low confidence level for the predicted trajectory. Thus, even if the trajectory predictor predicts that ego vehicle 100 is likely to take a given trajectory, ego vehicle 100 can decide not to pass this predicted trajectory to different subsystems due to the low confidence in the highly likely trajectory.

[0090] FIG. 3 illustrates an example hardware implementation for a dynamic and variable learning system 300 according to an embodiment of the present disclosure. The dynamic and variable learning system 300 can be a component of a vehicle, a robotic device, or other device. For example, as shown in FIG. 3, the dynamic and variable learning system 300 is a component of an autonomous vehicle 328. Although the dynamic and variable learning system 300 is located at the rear of the autonomous vehicle 328, the dynamic and variable learning system 300 can be located anywhere in the vehicle (e.g., at the front of the vehicle). Aspects of the present disclosure are not limited to the autonomous vehicle 328, as other devices, such as buses, boats, drones, or robots, are also contemplated for use with the dynamic and variable learning system 300. The autonomous vehicle 328 may be autonomous or semi-autonomous.

[0091] Dynamic and variable learning system 300 can be implemented with a bus architecture, generally represented by bus 350. Bus 350 can include any number of interconnected buses and bridges, depending on the particular application of dynamic and variable learning system 300 and the overall design constraints. Bus 350 links together various circuits, including one or more processors and / or hardware modules, such as processor 320, communications module 322, position module 318, sensor module 302, movement module 326, navigation module 324, computer-readable medium 314, and dynamic and variable learning module 308. Dynamic and variable learning module 308 includes reliability predictor 308a and trustworthiness estimator 308b. In some aspects, dynamic and variable learning module 308, including reliability predictor 308a and trustworthiness estimator 308b, is part of processor 320. Bus 350 may also link various other circuits, such as timing supplies, peripherals, voltage regulators, and power management circuits, which are well known in the art and will not be described further.

[0092] The dynamic and variable learning system 300 includes a transceiver 316 coupled to a processor 320, a sensor module 302, a dynamic and variable learning module 308, a communication module 322, a position module 318, a movement module 326, a navigation module 324, and a computer-readable medium 314. The transceiver 316 is coupled to an antenna 344. The transceiver 316 communicates with various other devices via a transmission medium. For example, the transceiver 316 can receive commands via transmission from a user or a remote device. As another example, the transceiver 316 can transmit driving statistics and information, environmental information, and other desired information from the dynamic and variable learning module 308 to a server (not shown), or vice versa.

[0093] The dynamic and variable learning system 300 includes a processor 320 coupled to a computer-readable medium 314. The processor 320 performs processes, including executing software stored on the computer-readable medium 314, to provide the disclosed functionality. The software, when executed by the processor 320, causes the dynamic and variable learning system 300 to perform various functions described for a particular device, such as an autonomous vehicle 328 or any of the modules 302, 314, 316, 318, 320, 322, 324, and 326. The computer-readable medium 314 can also be used to store data that is manipulated by the processor 320 when it executes the software.

[0094] The sensor module 302 can obtain measurements or environmental information via different sensors, such as a first sensor 306 and a second sensor 304. For example, the sensors can determine ambient condition information and sensor values ​​and provide them to the dynamic and variable learning module 308. The first sensor 306 can be an ambient condition sensor. The second sensor can be a ranging sensor, such as a light detection and ranging (lidar) sensor or a radio detection and ranging (radar) sensor. Of course, aspects of the present disclosure are not limited to the aforementioned sensors, as other types of sensors, such as, for example, thermal, ultrasonic, and / or laser, are also contemplated for either of the sensors 304, 306.

[0095] Measurements from the first sensor 306 and the second sensor 304 may be processed by one or more of the processor 320, the sensor module 302, the communication module 322, the position module 318, the dynamic and variable learning module 308, the movement module 326, and the navigation module 324, in conjunction with the computer-readable medium 314, to implement the functionality described herein. In one configuration, data obtained by the first sensor 306 and the second sensor 304 may be transmitted to an external device via the transceiver 316. The first sensor 306 and the second sensor 304 may be coupled to or in communication with an autonomous vehicle 328.

[0096] The location module 318 can determine the location of the autonomous vehicle 328. For example, the location module 318 can use a global positioning system (GPS) to determine the location of the autonomous vehicle 328. The communication module 322 can facilitate communication via the transceiver 316. For example, the communication module 322 can be configured to provide communication capabilities via different wireless protocols, such as WiFi, Long Term Evolution (LTE), 4G, 5G, etc. The communication module 322 can also be used to communicate with other components of the autonomous vehicle 328 that are not modules of the dynamic and variable learning system 300.

[0097] The locomotion module 326 can facilitate locomotion of the autonomous vehicle 328. As an example, the locomotion module 326 can control the movement of wheels. As another example, the locomotion module 326 can be in communication with a power source for the autonomous vehicle 328, such as an engine or a battery. Of course, aspects of the present disclosure are not limited to providing locomotion via wheels, and other types of components for providing locomotion are contemplated, such as propellers, treads, fins, and / or jet engines.

[0098] The dynamic and variable learning system 300 also includes a navigation module 324 for planning a path or controlling the movement of the autonomous vehicle 328 via a movement module 326. The navigation module 324 can be in communication with the dynamic and variable learning module 308, the sensor module 302, the transceiver 316, the processor 320, the communication module 322, the position module 318, the movement module 326, and the computer-readable medium 314.

[0099] The modules may be software running on the processor 320 and residing / stored on the computer-readable medium 314, one or more hardware modules coupled to the processor 320, or some combination thereof.

[0100] According to aspects of the present disclosure, the dynamic and variable learning system 300 includes a navigation module 324, a sensor module 322, a transceiver 316, a processor 320, a communication module 322, a position module 318, a movement module 326, and a dynamic and variable learning module 308 in communication with a computer-readable medium 314.

[0101] In one configuration, dynamic and variable learning module 308 determines reliability parameters for sensor values ​​obtained from one or more sensors (e.g., sensors 304, 306) of autonomous vehicle 328 based on the ambient conditions determined by ego vehicle 100. For example, as shown in Figure 2, sensor system 22 provides ambient conditions values ​​and other sensor values ​​that are used to determine the trustworthiness of a subsystem based on the reliability of the sensor values. For example, confidence in the accuracy of a subsystem's output is determined based on the reliability of the sensor values.

[0102] For example, the reliability can be determined by a reliability estimator 308a trained based on previously collected driving data. For example, the driving data can indicate a predicted trajectory for the vehicle and an actual trajectory taken by the vehicle. The trajectory can be predicted based on sensor values ​​from one or more sensors that are the same (or similar) as the autonomous vehicle 328 and obtained in the same (or similar) surroundings. The reliability estimator 308a can be trained based on calculated differences between the predicted trajectory and the actual trajectory. Based on this training, the reliability estimator 308a can determine the reliability of the sensor values ​​obtained from the one or more sensors in the current situation.

[0103] In one aspect, the trust estimator 308b generates a trust level for the outputs of the subsystems of the autonomous vehicle 328. The trust level indicates the confidence in the accuracy of the outputs of the subsystems. The outputs of the subsystems are based on obtained sensor values, and the trust level is generated based (at least in part) on the reliability of the obtained sensor values.

[0104] 4 illustrates a method 400 for providing dynamic and variable learning of an ego vehicle by determining and using the most reliable inputs according to an embodiment of the present disclosure. As shown in FIG. 4, in block 402, the ego vehicle determines a confidence level of sensor values ​​obtained from one or more ego vehicle sensors based on ambient conditions in the ego vehicle's environment. In block 404, the ego vehicle determines a confidence level in the accuracy of outputs of subsystems of the ego vehicle. The subsystem outputs are based on the sensor values, and the confidence level is based on the confidence level of the sensor values.

[0105] Based on the teachings, one skilled in the art should recognize that the scope of the present disclosure is intended to include any aspect of the present disclosure, whether implemented independently or in combination with any other aspect of the present disclosure. For example, an apparatus can be implemented using any number of the described aspects, and a method can be practiced using any number of the described aspects. In addition, the scope of the present disclosure is intended to include apparatuses or methods practiced using other structures, functions, or structures and functions in addition to or other than the various aspects of the present disclosure described. It should be understood that any aspect of the present disclosure can be embodied by one or more elements of a claim.

[0106] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0107] While particular embodiments are described herein, many variations and permutations of these embodiments are within the scope of the present disclosure. Although certain benefits and advantages of the preferred embodiments are mentioned, the scope of the present disclosure is not intended to be limited to particular benefits, uses, and objectives. Rather, the embodiments of the present disclosure are intended to be broadly applicable to different technologies, system configurations, networks, and protocols, some of which are illustrated by way of example in the drawings of the preferred embodiments and in the following description. The detailed description and figures are merely examples of the present disclosure and do not limit the scope of the present disclosure, which is defined by the appended claims and their equivalents.

[0108] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" can include calculating, computing, processing, deriving, investigating, examining (e.g., examining a table, database, or other data structure), ascertaining, and the like. Additionally, "determining" can include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Further, "determining" can include resolving, selecting, choosing, establishing, and the like.

[0109] As used herein, a phrase referring to "at least one" of a list of items refers to any combination of those items, including single members. By way of example, "at least one of a, b, or c" is intended to include a, b, c, a and b, a and c, b and c, and a, b, and c.

[0110] Various example logic blocks, modules, and circuits described in connection with this disclosure may be implemented or executed by a processor configured to perform the functions discussed in this disclosure. The processor may be a neural network processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. The processor may be a microprocessor, a controller, a microcontroller, or a state machine specially configured as described herein. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or other special configurations as described herein.

[0111] The steps of a method or algorithm described in connection with this disclosure can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module can reside in storage or machine-readable media, including random access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, removable disk, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium accessible by a computer that can be used to carry or store desired program code in the form of instructions or data structures. A software module can comprise a single instruction, or many instructions, and can be distributed over several different code segments, or among different programs, and across multiple storage media. A storage medium can be coupled to the processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium can be integral to the processor.

[0112] The methods disclosed herein comprise one or more steps or actions for achieving the described method. Method steps and / or actions may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0113] The described functions can be implemented in hardware, software, firmware, or any combination thereof. When implemented in hardware, an example hardware configuration can comprise a processing system in an apparatus. The processing system can be implemented in a bus architecture. The bus can include any number of interconnected buses and bridges, depending on the particular application and overall design constraints of the processing system. The bus can link various circuits together, including a processor, a machine-readable medium, and a bus interface. The bus interface can connect a network adapter, among other things, to the processing system via the bus. The network adapter can perform signal processing functions. For certain embodiments, a user interface (e.g., a keypad, a display, a mouse, a joystick, etc.) can also be connected to the bus. The bus can also link various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, etc., which are well known in the art and will not be described further.

[0114] The processor may be responsible for managing the bus and processes, including the execution of software stored on a machine-readable medium. Software, whether referred to as software, firmware, middleware, microcode, hardware description language, or the like, shall be construed to mean instructions, data, or any combination thereof.

[0115] In a hardware implementation, the machine-readable medium may be part of a processing system separate from the processor. However, as those skilled in the art will readily recognize, the machine-readable medium, or any portion thereof, may be external to the processing system. By way of example, the machine-readable medium may include a transmission line, a carrier wave modulated with data, and / or a computer product separate from the device, all of which are accessible to the processor via a bus interface. Alternatively or additionally, the machine-readable medium, or any portion thereof, may be integrated into the processor, such as in the case of a cache and / or specialized register file. While the various components discussed may be described as having a particular location, such as local components, they may also be configured in various ways, such as certain components configured as part of a distributed computing system.

[0116] A processing system can be configured with one or more microprocessors providing processor functionality and external memory providing at least a portion of the machine-readable medium, all linked to other support circuitry via an external bus architecture. Alternatively, the processing system can include one or more neuromorphic processors to implement the neuron models and neural system models described herein. As another alternative, the processing system can be implemented with an application-specific integrated circuit (ASIC) having the processor, bus interface, user interface, support circuitry, and at least a portion of the machine-readable medium integrated on a single chip, or with one or more field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), controllers, state machines, gate logic, discrete hardware components, or any other suitable circuitry or combination of circuitry capable of performing the various functions described throughout this disclosure. Those skilled in the art will recognize how to best implement the described functionality for a processing system depending on the particular application and the overall design constraints imposed on the overall system.

[0117] The machine-readable medium may comprise multiple software modules. The software modules may include a transmitting module and a receiving module. Each software module may reside on a single storage device or may be distributed across multiple storage devices. As an example, a software module may be loaded into RAM from a hard drive when a triggering event occurs. During execution of a software module, the processor may load some of the instructions into a cache to increase access speed. One or more cache lines may then be loaded into a special-purpose register file for execution by the processor. When referring below to functionality of a software module, it will be understood that such functionality is implemented by the processor when executing instructions from that software module. Furthermore, it should be recognized that aspects of the present disclosure may result in improvements in the functionality of a processor, computer, machine, or other system implementing such aspects.

[0118] If implemented in software, the functions can be stored on or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage devices and communication media, including any storage medium that facilitates transfer of a computer program from one place to another. Additionally, any connection can be properly referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared (IR), radio, and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. As used herein, disks and platters include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and Blu-ray disks. Disks typically reproduce data magnetically, while platters reproduce data optically with lasers. Thus, for some aspects computer-readable medium may comprise non-transitory computer-readable medium (e.g., tangible media). Additionally, for other aspects computer-readable medium may comprise transitory computer-readable medium (e.g., a signal). Combinations of the above should also be included within the scope of computer-readable media.

[0119] As such, certain aspects may comprise a computer program product for performing the operations presented herein. For example, such a computer program product may comprise a computer-readable medium having instructions stored (and / or encoded) thereon, the instructions being executable by one or more processors to perform the operations described herein. For certain aspects, the computer program product may include packaging materials.

[0120] Furthermore, it should be appreciated that modules and / or other suitable means for performing the methods and techniques described herein can be downloaded and / or obtained by a user terminal and / or a base station, if applicable. For example, such a device can be coupled to a server to facilitate the transfer of means for performing the methods described herein. Alternatively, the various methods described herein can be provided via storage means, such that the user terminal and / or base station can obtain the various methods upon coupling or providing the storage means to the device. Furthermore, any other suitable technique for providing the methods and techniques described herein to a device can be utilized.

[0121] It should be understood that the claims are not limited to the precise configuration and components described above. Various modifications, changes and variations in the arrangement, operation and details of the methods and apparatus described above can be made without departing from the scope of the claims.

Claims

1. 1. A method for providing dynamic and variable learning for an ego vehicle by determining and using a most reliable input, comprising: determining a confidence level for sensor values ​​obtained from one or more ego vehicle sensors based on environmental conditions of the ego vehicle; determining a level of confidence in the accuracy of the output of the ego vehicle's subsystem based on the reliability level of the sensor values; The method has the following features.

2. updating a confidence level of the sensor value as the ambient conditions change; updating the confidence level in the accuracy of the outputs of the subsystems of the ego vehicle based on updated sensor values; The method of claim 1 further comprising:

3. The method of claim 1 , wherein the ambient conditions are determined by ambient condition sensors on the ego vehicle and / or are provided to the ego vehicle from a remote device.

4. The method of claim 1 , further comprising reducing a confidence level of the sensor values ​​from a radar sensor when the ego vehicle is in a congested area.

5. The method of claim 1 , further comprising increasing a confidence level of the sensor value of a radar sensor when the ego vehicle is traveling on a highway.

6. 10. The method of claim 1, further comprising: reducing a confidence level of the sensor values ​​from the ego vehicle's camera when the ego vehicle is in low light conditions that result in a black image or high light conditions that result in a fuzzy white image.

7. The method of claim 1 , wherein the confidence level is determined by a confidence estimator trained on previously collected driving data.

8. The method of claim 1 , wherein the trust level is determined based on a trust estimator.

9. 1. A system for providing dynamic and variable learning for an ego vehicle by determining and using a most reliable input, comprising: Memory and at least one processor, determining a confidence level for sensor values ​​obtained from one or more ego vehicle sensors based on ambient conditions of the ego vehicle's environment; a processor configured to determine a level of confidence in the accuracy of the output of the ego vehicle's sensor value-based subsystem based on a confidence level of the sensor value; A system comprising:

10. The at least one processor updating the confidence level of the sensor value as the ambient conditions change; updating the confidence level in the accuracy of the outputs of the subsystems of the ego vehicle based on updated sensor values; The system of claim 9 further configured to:

11. 10. The system of claim 9, wherein the ambient conditions are determined by ambient condition sensors on the ego vehicle and / or are provided to the ego vehicle from a remote device.

12. 10. The system of claim 9, wherein the at least one processor is further configured to reduce a confidence level of the sensor values ​​from a radar sensor when the ego vehicle is in a congested area.

13. 10. The system of claim 9, wherein the at least one processor is further configured to increase a confidence level of the sensor value of a radar sensor when the ego vehicle is traveling on a highway.

14. 10. The system of claim 9, wherein the at least one processor is further configured to reduce a confidence level of the sensor values ​​from the ego vehicle's camera when the ego vehicle is in low light conditions that result in a black image or high light conditions that result in a fuzzy white image.

15. The system of claim 9 , wherein the confidence level is determined by a confidence estimator trained based on previously collected driving data.

16. The system of claim 9 , wherein the trust level is determined based on a trust estimator.

17. 1. A non-transitory computer readable medium having program code executed by a processor for providing dynamic and variable learning for an ego vehicle by determining and using a most reliable input; program code for determining a confidence level of sensor values ​​obtained from one or more ego vehicle sensors based on ambient conditions of the ego vehicle's environment; program code for determining a level of confidence in the accuracy of an output of the ego vehicle's subsystem based on the sensor value, based on the confidence level of the sensor value; 1. A non-transitory computer-readable medium comprising:

18. updating the confidence level of the sensor value as the ambient conditions change; 20. The non-transitory computer-readable medium of claim 17, further comprising program code for updating the confidence level in the accuracy of the outputs of the subsystems of the ego vehicle based on updated sensor values.

19. 20. The non-transitory computer-readable medium of claim 17, wherein the ambient conditions are determined by ambient condition sensors on the ego vehicle and / or are provided to the ego vehicle from a remote device.

20. 20. The non-transitory computer readable medium of claim 17, further comprising program code for reducing a confidence level of the sensor values ​​from a radar sensor when the ego vehicle is in a congested area.

Images