Redundant environmental perception tracking for automated driving systems

Abstract

Redundant environmental perception tracking for automated driving systems. One embodiment provides an automated driving system for a vehicle, wherein the system includes multiple sensors, a memory, and an electronic processor.The electronic processor is designed to receive, from multiple processors, environmental information from a common field of view; to generate, based on the environmental information, multiple hypotheses regarding an object within the common field of view, wherein the multiple hypotheses include at least one set of hypotheses excluding the environmental information from at least one sensor of a first sensor type; to determine, based on a subset of the multiple hypotheses, an object state of the object, wherein the subset includes the at least one set of hypotheses excluding the environmental information from the at least one sensor; and to execute a vehicle maneuver based on the determined object state.

Description

CROSS-REFERENCE TO RELATED REGISTRATION

[0001] This application claims the priority and benefit of the preliminary U.S. application No. 62 / 854,804, filed on May 30, 2019, the entire contents of which are hereby incorporated by reference. BACKGROUND OF THE INVENTION

[0002] Driver assistance and automated driving systems are becoming increasingly popular. Such systems rely on environmental information from a variety of different sensors to detect the vehicle's surroundings. By detecting objects (and their movements) in the vehicle's environment, the system is able to drive the vehicle according to the detected objects and their movements. DE 10 2008 013 366 A1 relates to a method for providing information to driver assistance systems. EP 2 604 478 B1 relates to a method for detecting malfunctions of one or more sensors in a multi-sensor arrangement of a vehicle assistance system used to monitor a vehicle's environment. BRIEF DESCRIPTION OF THE MULTIPLE VIEWS OF THE DRAWINGS

[0003] The accompanying drawings, in which identical reference numerals refer to identical and functionally similar elements throughout the separate views, together with the detailed description below, are included in the specification and form a part of it, and serve to further illustrate embodiments of concepts and to explain various principles and advantages of these embodiments. Fig. Figure 1 is a block diagram of an automated driving system according to some embodiments. Fig. 2 is a process model that guides you through the system of Fig. 1. The implemented process for determining an object based on environmental information is illustrated, according to some embodiments. Fig. 3 is a flowchart that shows a route through the system of Fig. Figure 1 illustrates an implemented method for processing redundant detected environmental information, according to some embodiments. Fig. Figure 4 is a diagram depicting an environment surrounding a vehicle with an automated driving system. Fig. 1 illustrated according to some embodiments. Fig. 5 is a graph containing several according to the procedure of Fig. Two hypotheses are illustrated, according to some embodiments.

[0004] Experts will recognize that elements in the figures are illustrated for simplification and clarity and are not necessarily drawn to scale. For example, the dimensions of some elements in the figures may be exaggerated relative to other elements to improve the understanding of embodiments of the present invention.

[0005] The equipment and process components have, where appropriate, been represented by conventional symbols in the drawings, which show only those specific details relevant to understanding the embodiments of the present invention, so that the disclosure is not obscured by details which are readily apparent to persons skilled in the art with knowledge of the description herein. DETAILED DESCRIPTION OF THE INVENTION

[0006] As noted above, driver assistance and automated driving (hereinafter collectively referred to as automated driving) depend on a variety of sensors with numerous overlapping fields of view to detect objects (cars, pedestrians, lane markings, curbs, etc.) within an area surrounding the vehicle (hereinafter referred to as the vehicle's surrounding environment). Using the sensor information, the automated driving system determines the object's position and movement. The system then controls (drives) the vehicle based on this determined position and / or movement. Accurate determination of the object's position and movement is crucial so that the vehicle is steered correctly to avoid a collision.

[0007] Although the present disclosure focuses on automated driving, it is not limited to automated driving. For example, it can also be used for object detection in surveillance applications (e.g., multiple sensors mounted on buildings and vehicles to track pedestrians, vehicles, or other suitable objects) or any other application that determines the state of an object. The surveillance aspect of this disclosure is set out below specifically with regard to numbered Examples 21-40.

[0008] The invention relates to an automated driving system for a vehicle with the features according to claim 1, a method for operating an automated driving system with the features according to claim 7, and a non-volatile computer-readable medium with the features according to claim 13.

[0009] According to the invention, an automated driving system is provided for a vehicle, wherein the system includes several sensors, a memory and an electronic processor that is communicatively coupled with the memory and the several sensors.The electronic processor is designed to receive, from multiple processors, environmental information from a common field of view; to generate, based on the environmental information, multiple hypotheses regarding an object within the common field of view, wherein the multiple hypotheses include at least one set of hypotheses excluding the environmental information from at least one sensor of a first sensor type; to determine, based on a subset of the multiple hypotheses, an object state of the object, wherein the subset includes the at least one set of hypotheses excluding the environmental information from the at least one sensor of the first sensor type; and to execute a vehicle maneuver based on the determined object state.

[0010] According to the invention, a method for operating an automated driving system is provided. The method includes receiving, with an electronic processor, environmental information from a common field of view from multiple sensors. The method includes generating, with the electronic processor, multiple hypotheses regarding an object within the common field of view based on the environmental information, wherein the multiple hypotheses include at least one set of hypotheses based exclusively on the environmental information from at least one sensor of a first sensor type. The method includes detecting, with the electronic processor, a fault associated with the at least one sensor of the first sensor type.The method involves determining, using the electronic processor, an object state based on a subset of several hypotheses, where the subset includes at least one set of hypotheses excluding environmental information from at least one sensor of the first sensor type. The method also involves executing, using the electronic processor, a vehicle maneuver based on the determined object state.

[0011] According to the invention, a non-volatile, computer-readable medium provides instructions which, when executed by an electronic processor, cause the electronic processor to perform a set of operations. The set of operations includes receiving environmental information from a common field of view from multiple sensors. The set of operations includes generating multiple hypotheses about an object within the common field of view based on the environmental information, wherein the multiple hypotheses include at least one set of hypotheses based exclusively on the environmental information from at least one sensor of a first sensor type. The set of operations includes detecting a fault associated with the at least one sensor of the first sensor type.The set of operations includes determining the object's state based on a subset of several hypotheses, where the subset comprises at least one set of hypotheses excluding environmental information from at least one sensor of the first sensor type. The set of operations also includes performing a vehicle maneuver based on the determined object state.

[0012] Before any embodiments of the invention are explained in detail, it is understood that the invention is not limited in its application to the design details and arrangement of components set forth in the following description or illustrated in the following drawings. The invention is capable of other embodiments and of being exercised or carried out in various ways. Although the examples described herein relate, for instance, to automated driving systems, the methods described herein can be applied to other embodiments of driver assistance systems. It should be understood that the term "vehicle" refers to any type of transportation system, including, but not limited to, cars, motorcycles, drones, boats, and the like.

[0013] To simplify the description, some or all of the exemplary systems presented herein are illustrated with a single instance of each of its components. Some examples may not describe or illustrate all components of the systems. Other embodiments may include more or less of each of the illustrated components, may combine some components, or may include additional or alternative components.

[0014] Fig. Figure 1 is an exemplary automated driving system 100 according to some embodiments. The system 100 includes a vehicle 405 comprising an electronic processor 105, a memory 110, an input / output interface 115, several sensors 120, and a transceiver 125. In some embodiments, the system 100 also includes several sensors 130 mounted on infrastructure located external to or near the vehicle 405. As explained above, in alternative embodiments, the system 100 includes a monitoring device instead of the vehicle 405, wherein the monitoring device comprises an electronic processor, a memory, an input / output interface, several sensors, and a transceiver similar to the electronic processor 105, the memory 110, the input / output interface 115, the several sensors 120, and the transceiver 125, respectively, as described herein.

[0015] The illustrated components, together with other various modules and components, can be coupled to one another by means of or via one or more connections, including, for example, control or data buses that enable communication between them. The use of control and data buses for the connection between the various modules and components and the exchange of information between them would be apparent to a person skilled in the art with regard to the description provided herein. It should be understood that some components of the automated driving system 100 within a vehicle (for example, the vehicle 405 of Fig. 4) can be incorporated / integrated, for example the vehicle's communication system and / or the vehicle's electrical system (not shown).

[0016] The electronic processor 105 receives and provides information (for example, from memory 110, from the input / output interface 115, and / or from the multiple sensors 120) and processes the information by executing one or more software instructions or modules, which may be stored, for example, in a random access memory (“RAM”) area of ​​memory 110, a read-only memory (“ROM”) of memory 110, or another non-volatile, computer-readable medium (not shown). The software may include firmware, one or more instructions, program data, filters, rules, one or more program modules, and other executable instructions. The electronic processor 105 is designed to retrieve and execute software relating to the control processes and procedures described herein from memory 110.

[0017] Memory 110 can contain one or more non-volatile, computer-readable media and includes a program storage area and a data storage area. The program storage area and the data storage area can contain combinations of different storage types, as described herein. Memory 110 can take the form of any non-volatile, computer-readable medium.

[0018] The input / output interface 115 is designed to receive input via one or more user input devices or interfaces and to provide system output via one or more user output devices. The input / output interface 115 receives information and signals (for example, via one or more wired and / or wireless connections) from devices both internal and external to the vehicle 405 and provides information and signals to them. The input / output interface 115 is coupled to the multiple sensors 120. The multiple sensors 120 provide input to the input / output interface 115. The multiple sensors 120 include different types of sensors located throughout the vehicle 405. The multiple sensors 130 are mounted on infrastructure located externally and near the vehicle 405.The multiple sensors 120 and the multiple sensors 130 can include, among other things, radar sensors, lidar sensors, image sensors, ultrasonic sensors, or other suitable sensors. It should be understood that each of the multiple sensors 120 and 130 can contain more than one sensor of the same type.

[0019] To simplify the description, the multiple sensors 120 are described as having an overlapping field of view (where the field of view is an area of ​​the vehicle 405's surroundings monitored by the respective sensor). To further simplify the description and avoid redundancy, only the multiple sensors 120 are described below. However, the description below is not limited to the multiple sensors 120 and is equally applicable to a combination of the multiple sensors 120 and the multiple sensors 130, where the multiple sensors 120 and the multiple sensors 130 are combined to form a larger plurality of sensors.

[0020] As described in more detail below, this overlapping area, in which the fields of view of each of the multiple sensors 120 coincide, is referred to as the "common field of view." The system 100 may also include additional sensors, separate from (but similar to) the multiple sensors 120 described herein, which share a different common field of view. The extent of a sensor's field of view depends on the sensor's configuration. Thus, the multiple sensors 120 may have fields of view of varying sizes (and depths).

[0021] In some embodiments, the transceiver 125 is configured for wireless coupling with wireless networks. Alternatively or additionally, the vehicle 405 may include a connector or port (not shown) for receiving a connection to a wired network (for example, Ethernet). The electronic processor 105 is designed to operate the transceiver 125 to receive an input and provide a system output, or a combination of both. The transceiver 125 receives information and signals (for example, via one or more wired and / or wireless connections) from devices (including, in some embodiments, from the communication controller) both internal and external to the vehicle 405 and provides information and signals to them.

[0022] In some embodiments, the transceiver 125 can receive information and signals from the multiple sensors 130 mounted on the infrastructure near or external to the vehicle 405 and provide this information and these signals to the vehicle. In these embodiments, the transceiver 125 is designed to communicate with a wireless vehicle-to-everything (V2X) network.

[0023] In other embodiments, the transceiver 125 can receive information and signals from the multiple sensors 130 mounted on other vehicles in close proximity to or external to the vehicle 405 and provide them with information and signals. In these embodiments, the transceiver 125 is designed to communicate with a wireless V2X network and / or a wireless vehicle-to-vehicle (V2V) network.

[0024] Fig. Figure 2 illustrates a process model 200 implemented by an automated driving system, according to some embodiments. To simplify the description, the model 200 is to be defined with respect to the system 100. Fig. 1, in particular the electronic processor 105, can be described. As described above, the automated driving system 100 receives information from the multiple sensors 120 (sensor measurement 205). The object(s) within the environment surrounding the vehicle 405 can be determined by the automated driving system 100, which is formed by fusion (block 210) of the data from the multiple sensors 120.

[0025] Determining an object can involve not only determining its position relative to the vehicle 405, but also its speed and / or direction of movement relative to the vehicle 405, and identifying the type of object (object classification) (for example, determining whether the object is a car, a motorcycle, a person, or a bicycle). The determination of one or more of the object's properties described above is collectively referred to here as an "object state." Based on the object state, a function executed by the electronic processor 105 determines whether there should be a response to the object and what that response should be (output 220).

[0026] The fusion of sensor information (Block 210) involves one or more predictive models, artificial intelligence, deep learning, and / or object tracking processes (Block 215), which are performed by the electronic processor 105. In object tracking processes, if information from multiple sensors / sources is used, more than one possible object can be determined by the electronic processor 105. Each possible state (determined here based on information from any number and type of sensors 120) is referred to as an association hypothesis. An object state is then determined by the electronic processor 105 based on the association hypotheses.The association hypotheses (and thus the specific object state) are monitored by the electronic processor 105 and updated periodically (with each period being called a cycle) based on new information received from the sensors 120.

[0027] Some approaches to using and managing multiple association hypotheses over time include, for example, MHT (Multiple Hypothesis Tracking) and RFS (Random Finite Set) processes, including LMB (Labeled Multi-Bernoulli) filters. While multiple association hypotheses provide robustness in tracking, errors (especially in the case of faulty but plausible measurements) may not be detected immediately. This can be particularly true for systematic errors in a single sensor (for example, due to incorrect calibration, angle reading errors due to reflection from the detected surface, incorrect measurement of a specific object by deep learning, etc.). Such errors can lead to an incorrect update for the specific object state and a loss of tracking. Thus, the system might perform an incorrect vehicle maneuver due to the update.

[0028] The approaches mentioned above are capable of weeding out erroneous hypotheses based on subsequent measurements over time. Specifically, tracking performance is improved by discarding (deleting) hypotheses that contain an erroneous tracking-to-measurement association (random errors) while retaining several other hypotheses. A procedure for keeping the number of hypotheses manageable involves grouping similar hypotheses while discarding improbable ones. However, in the case of systematic errors (such as those described above), the aforementioned approaches will detect such affected hypotheses.

[0029] As described herein, an “error” can be a problem with the sensor (usually a sensor defect), such as misalignment, or the “error” can be a false detection (which might be missing, shifted from the true value, or similar). False detections are expected and can be difficult to distinguish from a sensor error. A sensor error would typically be reported if a sensor exhibits errors for most traces or if the sensor is missing for most traces over an error maturation period.

[0030] Fig. Figure 3 is a flowchart illustrating an exemplary method 300 for determining an object in an environment surrounding a vehicle according to some embodiments. The method 300 is robust against both types of the aforementioned “errors” because the method 300 generates different hypotheses even in the presence of a single undetected sensor error.

[0031] As an example, method 300 is described as being carried out by system 100 and, in particular, by the electronic processor 105. However, it should be understood that in some embodiments, parts of method 300 can be carried out by other devices.

[0032] To simplify the description, procedure 300 is also described in conjunction with Fig. 4 described. Fig. Figure 4 illustrates the vehicle 405 and an object 415 (in the illustrated embodiment, a car). The object 415 is located near (within the environment, in the surroundings) the vehicle 405. As mentioned above, each of the multiple sensors 120 has a field of view in which it monitors. In the illustrated embodiment, for example, the field of view 410A corresponds to a first sensor 120A of the multiple sensors 120, and the field of view 410B corresponds to a second sensor 120B of the multiple sensors 120. The area where the two intersect is referred to as the common field of view 410C. For the sake of simplicity, the common field of view 410C is described with respect to the first and second sensors 120A and 120B. It should be understood that in further embodiments, more sensors may be associated with the common field of view 410C based on their respective fields of view.

[0033] Again with reference to Fig. 3. The electronic processor 105 at block 305 receives environmental information from a common field of view (for example, the common field of view 410C) from the multiple sensors 120. As mentioned above, the multiple sensors 120 include sensors of various types, including, but not limited to, radar sensors, image sensors, lidar sensors, and other suitable sensors. Thus, the environmental information received by the multiple sensors 120 can be acquired in different media (for example, image-based, ultrasound-based, etc.).

[0034] In block 310, the electronic processor 105 generates several hypotheses regarding an object (for example, object 415 of) based on the environmental information. Fig. 4) within the common field of view (for example, the common field of view 410C). The multiple hypotheses include at least one set of hypotheses determined using the received environmental information, excluding environmental information from at least one sensor. In other words, one hypothesis is generated using environmental information from all of the multiple sensors 120, except for at least one of the multiple sensors 120. In the event that a fault associated with the at least one sensor is present or detected, there is at least one set of hypotheses that is not affected by a faulty / inaccurate measurement taken by the at least one sensor.

[0035] For example, to detect the fault associated with at least one sensor of the first sensor type, the electronic processor 105 can be designed to detect a divergence of a second hypothesis of the multiple hypotheses from a first hypothesis of the multiple hypotheses, while the multiple hypotheses are updated over time based on the additional environmental information received by the multiple hypotheses. This fault detection is described below with reference to Fig. 5 described in more detail.

[0036] In block 315, the electronic processor 105 determines an object state of object 415 based on a subset of several hypotheses, where the subset includes the hypothesis determined using the received environmental information, excluding environmental information from at least one sensor. In other words, the electronic processor 105 is able to determine an object state without dependence on a faulty measurement.

[0037] In block 320, the electronic processor 105 performs a vehicle maneuver based on the specified object state. The vehicle maneuver involves steering and / or influencing the speed (accelerating / braking) of the vehicle 405 based on the specified object state. Fig. 4. The specified object state of object 415 could, for example, be that object 415 is a vehicle in the left lane traveling straight ahead at x miles per hour. Accordingly, the electronic processor 105 can control vehicle 405 to remain in the right lane and only merge into the left lane after object 415 has passed it. The determination of the vehicle maneuver by the electronic processor 105 based on the specified object state can be carried out using one or more automated driving techniques, which are not discussed here for the sake of brevity.

[0038] In some embodiments, an additional hypothesis is determined for each sensor, each excluding information received from that sensor. In other embodiments, an additional hypothesis is calculated that excludes information from one or more sensors of the same type among the multiple sensors 120. For example, the sensor types of the multiple sensors 120 may include a combination of radar, video, and lidar. Thus, the electronic processor 105 would determine a "no radar information" hypothesis, a "no video information" hypothesis, and a "no lidar information" hypothesis, along with hypotheses that include measurements from all sensors of the multiple sensors 120.

[0039] Additionally or alternatively, in some examples, the multiple sensors 120 may include a combination of sensor types or a single sensor type (e.g., only lidar, only radar, or only video). In these examples, the electronic processor 105 may determine a "No-Lidar1" hypothesis, a "No-Lidar2" hypothesis, and a "No-Lidar3" hypothesis, along with hypotheses that incorporate measurements from all sensors of the multiple sensors 120. In response to the detection of a fault associated with any sensor (or type), the hypothesis (or hypotheses) that was not determined based on information from the faulty sensor(s) may be used in determining the object state (while one or more of the hypotheses that are impaired / based on such information may be excluded from determining the object state).Simply put, the electronic processor 105 can determine an object's state based on hypotheses that exclude environmental information from a sensor type, or at a more granular level, based on hypotheses that exclude environmental information from a specific defective sensor.

[0040] Within a mixed representation of an object's tracking (for example, in the case of an LMB filter), the method 300 can be applied by storing the sensor types involved in each component of the mixed distribution and avoiding the deletion and merging of specific components regardless of their weight and distance to other state components. To save memory and computational load, the electronic processor 105, in some embodiments, assigns flags (for example, "consistent with no radar") to components for which the associated sensor measurement (here, radar) had a very high likelihood. In the case of Gaussian-distributed states, this corresponds to a very small Mahalanobis distance, reflecting the fact that the expected measurement and the actual measurement are almost identical.In some embodiments, the electronic processor 105 is further designed to determine sensor degradation based on a detected fault. In particular, the distance between the state estimates of the different state components of an LMB (which can be reduced to a Bernoulli distribution if the components are reduced to a single object for each Gaussian mixture distribution) can be used as a feature for sensor degradation detection. Possible ways to detect this include time series analysis or the permanent deviation of more than two sensors.

[0041] Fig.Figure 5 illustrates a mixture distribution over time 500 (increasing from left to right) using a first sensor type and a second sensor type (for example, radar and lidar, respectively). At time step A, the measurements from the two sensor types are consistent, and thus their respective multiple hypotheses 501A and 501B coincide (illustrated as an overlapping circle). At time step B, a measurement 502A is received from the first sensor type (indicated as a white star). The measurement 502A does not contradict the hypotheses 501B of the second sensor type, and thus the hypotheses of both sensor types continue to coincide at time step B. At time step C, a measurement from the second sensor type (illustrated as a black star 503A) causes the hypotheses 501B of the second sensor type to differ from the hypothesis 501A of the first sensor type.Since some of the hypotheses from the first sensor type exclude information from the second sensor type, the overlapping hypotheses may not be merged at time step C, and the highly improbable hypotheses 501A (whose probability decreased due to a second received measurement from the second sensor type, labeled as star 503B) at time step D are not discarded. At step E, a new measurement from the first sensor (star 502B) is incorporated, and thus the hypotheses 501A of the first sensor type are not affected by the systematic error occurring in the second sensor type.

[0042] The following are numbered examples of systems, methods, and non-volatile computer-readable media according to various aspects of the present disclosure.

[0043] Example 1: An automated driving system for a vehicle, wherein the system comprises: multiple sensors; a memory;and an electronic processor that is communicatively coupled to the memory and the multiple sensors, wherein the electronic processor is designed to receive environmental information from the multiple processors of a common field of view, generate, based on the environmental information, multiple hypotheses regarding an object within the common field of view, wherein the multiple hypotheses include at least one set of hypotheses excluding the environmental information from at least one sensor of a first sensor type, determine, based on a subset of the multiple hypotheses, an object state of the object, wherein the subset includes the at least one set of hypotheses excluding the environmental information from the at least one sensor of the first sensor type, and execute a vehicle maneuver based on the determined object state.

[0044] Example 2: The automated driving system of Example 1, wherein the electronic processor is further designed to update the multiple hypotheses over time based on additional environmental information received from the multiple sensors.

[0045] Example 3: The automated driving system of Example 2, wherein the electronic processor is further designed to detect a fault associated with the at least one sensor of the first sensor type, and wherein the determination, based on the subset of multiple hypotheses, of the object state of the object takes place in response to the detection of the fault associated with the at least one sensor of the first sensor type.

[0046] Example 4: The automated driving system of one of Examples 1 to 3, wherein the multiple sensors include two or more sensors selected from a group consisting of: one or more radar sensors, one or more lidar sensors, one or more image sensors and one or more ultrasonic sensors.

[0047] Example 5: The automated driving system of one of the examples 1 to 4, wherein a first sensor of the multiple sensors is a radar sensor, and wherein a second sensor of the multiple sensors is a lidar sensor.

[0048] Example 6: The automated driving system of Example 5, wherein for generating, based on environmental information, the multiple hypotheses regarding the object within the common field of view, wherein the multiple hypotheses include at least one set of hypotheses based exclusively on environmental information from at least one sensor of a first sensor type, the electronic processor is further configured to generate a first set of hypotheses of the multiple hypotheses at least partially based on environmental information received from the radar sensor and the lidar sensor, and to generate a second set of hypotheses of the multiple hypotheses at least partially based on environmental information received from the radar sensor and not based on environmental information received from the lidar sensor.and generating a third set of hypotheses from the multiple hypotheses at least partially based on the environmental information received by the lidar sensor, and not based on the environmental information received by the radar sensor.

[0049] Example 7: The automated driving system of Example 6, wherein for determining, based on the subset of several hypotheses, the object state of the object, wherein the subset includes the at least one set of hypotheses excluding the environmental information from the at least one sensor of a first sensor type, the electronic processor is further designed to determine the object state of the object based on the third set of hypotheses.

[0050] Example 8: The automated driving system of one of the examples 1 to 7, wherein the subset includes other hypotheses including environmental information from other sensors of the first sensor type.

[0051] Example 9: A method for operating an automated driving system, wherein the method comprises: receiving, with an electronic processor, environmental information of a common field of view from multiple sensors; generating, with the electronic processor, multiple hypotheses about an object within the common field of view based on the environmental information, wherein the multiple hypotheses include at least one set of hypotheses excluding the environmental information from at least one sensor of a first sensor type; determining, with the electronic processor, an object state of the object based on a subset of the multiple hypotheses, wherein the subset includes the at least one set of hypotheses excluding the environmental information from the at least one sensor of the first sensor type; and executing, with the electronic processor, a vehicle maneuver based on the determined object state.

[0052] Example 10: The procedure of Example 9, further comprising: updating, with the electronic processor, several hypotheses over time based on additional environmental information received from the multiple sensors.

[0053] Example 11: The procedure of Example 10, further comprising: detecting, with the electronic processor, a fault associated with the at least one sensor of the first sensor type, wherein determining the object state of the object based on the subset of multiple hypotheses takes place in response to the detection of the fault associated with the at least one sensor of the first sensor type.

[0054] Example 12: The method of one of Examples 9 to 11, wherein the multiple sensors include two or more sensors selected from a group consisting of: one or more radar sensors, one or more lidar sensors, one or more image sensors, and one or more ultrasonic sensors.

[0055] Example 13: The method of one of Examples 9 to 12, wherein a first sensor of the multiple sensors is a radar sensor, and wherein a second sensor of the multiple sensors is a lidar sensor.

[0056] Example 14: The procedure of Example 13, wherein generating the multiple hypotheses regarding the object within the common field of view based on the environmental information further includes: generating a first set of hypotheses of the multiple hypotheses at least partially based on the environmental information received by the radar sensor and the lidar sensor; generating a second set of hypotheses of the multiple hypotheses at least partially based on the environmental information received by the radar sensor and not based on the environmental information received by the lidar sensor; and generating a third set of hypotheses of the multiple hypotheses at least partially based on the environmental information received by the lidar sensor and not based on the environmental information received by the radar sensor.

[0057] Example 15: The procedure of Example 14, wherein determining the object state of the object based on the subset of multiple hypotheses, wherein the subset includes at least one set of hypotheses excluding the environmental information from at least one sensor of the first sensor type, furthermore includes determining the object state of the object based on the third set of hypotheses.

[0058] Example 16: The procedure of one of Examples 9 to 15, wherein the subset includes other hypotheses including environmental information from other sensors of the first sensor type.

[0059] Example 17: A non-volatile, computer-readable medium comprising instructions which, when executed by an electronic processor, cause the electronic processor to perform a set of operations comprising: receiving environmental information of a common field of view from multiple sensors; generating multiple hypotheses about an object within the common field of view based on the environmental information, wherein the multiple hypotheses include at least one set of hypotheses excluding the environmental information from at least one sensor of a first sensor type; determining an object state of the object based on a subset of the multiple hypotheses, wherein the subset includes the at least one set of hypotheses excluding the environmental information from the at least one sensor of the first sensor type; and performing a vehicle maneuver based on the determined object state.

[0060] Example 18: The non-volatile, computer-readable medium of Example 17, wherein the set of operations further includes updating the multiple hypotheses over time based on additional environmental information received from the multiple sensors.

[0061] Example 19: The non-volatile computer-readable medium of Example 18, wherein the set of operations further includes detecting a fault associated with the at least one sensor of the first sensor type, wherein determining the object state of the object based on the subset of multiple hypotheses takes place in response to the detection of the fault associated with the at least one sensor of the first sensor type.

[0062] Example 20: The non-volatile, computer-readable medium of one of Examples 17 to 19, wherein the subset includes other hypotheses including environmental information from other sensors of the first sensor type.

[0063] Example 21: A monitoring system, wherein the system comprises: several sensors; a memory;and an electronic processor that is communicatively coupled to the memory and the multiple sensors, the electronic processor being configured to receive environmental information from the multiple processors of a common field of view, generate, based on the environmental information, multiple hypotheses regarding an object within the common field of view, wherein the multiple hypotheses include at least one set of hypotheses excluding the environmental information from at least one sensor of a first sensor type, determine, based on a subset of the multiple hypotheses, an object state of the object, wherein the subset includes the at least one set of hypotheses excluding the environmental information from the at least one sensor of the first sensor type, and track the object based on the determined object state.

[0064] Example 22: The monitoring system of Example 21, wherein the electronic processor is further designed to update the multiple hypotheses over time based on additional environmental information received from the multiple sensors.

[0065] Example 23: The monitoring system of Example 22, wherein the electronic processor is further designed to detect a fault associated with the at least one sensor of the first sensor type, and wherein the determination, based on the subset of multiple hypotheses, of the object state of the object takes place in response to the detection of the fault associated with the at least one sensor of the first sensor type.

[0066] Example 24: The monitoring system of one of Examples 21 to 23, wherein the multiple sensors include two or more sensors selected from a group consisting of: one or more radar sensors, one or more lidar sensors, one or more image sensors and one or more ultrasonic sensors.

[0067] Example 25: The monitoring system of one of Examples 21 to 24, wherein a first sensor of the multiple sensors is a radar sensor, and wherein a second sensor of the multiple sensors is a lidar sensor.

[0068] Example 26: The monitoring system of Example 25, wherein for generating, based on the environmental information, the multiple hypotheses regarding the object within the common field of view, wherein the multiple hypotheses include at least one set of hypotheses based exclusively on the environmental information from at least one sensor of a first sensor type, the electronic processor is further configured to generate a first set of hypotheses of the multiple hypotheses at least partially based on the environmental information received from the radar sensor and the lidar sensor, and to generate a second set of hypotheses of the multiple hypotheses at least partially based on the environmental information received from the radar sensor and not based on the environmental information received from the lidar sensor.and generating a third set of hypotheses from the multiple hypotheses at least partially based on the environmental information received by the lidar sensor, and not based on the environmental information received by the radar sensor.

[0069] Example 27: The monitoring system of Example 26, wherein for determining, based on the subset of several hypotheses, the object state of the object, wherein the subset includes the at least one set of hypotheses excluding the environmental information from the at least one sensor of a first sensor type, the electronic processor is further designed to determine the object state of the object based on the third set of hypotheses.

[0070] Example 28: The monitoring system of one of the examples 21 to 27, wherein the subset includes other hypotheses including environmental information from other sensors of the first sensor type.

[0071] Example 29: A method for operating a monitoring system, wherein the method comprises: receiving, with an electronic processor, environmental information of a common field of view from multiple sensors; generating, with the electronic processor, multiple hypotheses about an object within the common field of view based on the environmental information, wherein the multiple hypotheses include at least one set of hypotheses excluding the environmental information from at least one sensor of a first sensor type; determining, with the electronic processor, an object state of the object based on a subset of the multiple hypotheses, wherein the subset includes the at least one set of hypotheses excluding the environmental information from the at least one sensor of the first sensor type; and tracking, with the electronic processor, the object based on the determined object state.

[0072] Example 30: The procedure of Example 29, further comprising: updating, with the electronic processor, several hypotheses over time based on additional environmental information received from the multiple sensors.

[0073] Example 31: The procedure of Example 30, further comprising: detecting, with the electronic processor, a fault associated with the at least one sensor of the first sensor type, wherein determining the object state of the object based on the subset of multiple hypotheses takes place in response to the detection of the fault associated with the at least one sensor of the first sensor type.

[0074] Example 32: The method of one of Examples 29 to 31, wherein the multiple sensors include two or more sensors selected from a group consisting of: one or more radar sensors, one or more lidar sensors, one or more image sensors and one or more ultrasonic sensors.

[0075] Example 33: The method of one of Examples 29 to 32, wherein a first sensor of the multiple sensors is a radar sensor, and wherein a second sensor of the multiple sensors is a lidar sensor.

[0076] Example 34: The procedure of Example 33, wherein generating the multiple hypotheses regarding the object within the common field of view based on the environmental information further includes: generating a first set of hypotheses of the multiple hypotheses at least partially based on the environmental information received by the radar sensor and the lidar sensor; generating a second set of hypotheses of the multiple hypotheses at least partially based on the environmental information received by the radar sensor and not based on the environmental information received by the lidar sensor; and generating a third set of hypotheses of the multiple hypotheses at least partially based on the environmental information received by the lidar sensor and not based on the environmental information received by the radar sensor.

[0077] Example 35: The procedure of Example 34, wherein determining the object state of the object based on the subset of several hypotheses, wherein the subset includes at least one set of hypotheses excluding the environmental information from at least one sensor of the first sensor type, furthermore includes determining the object state of the object based on the third set of hypotheses.

[0078] Example 36: The procedure of one of Examples 29 to 35, wherein the subset includes other hypotheses including environmental information from other sensors of the first sensor type.

[0079] Example 37: A non-volatile, computer-readable medium comprising instructions which, when executed by an electronic processor, cause the electronic processor to perform a set of operations comprising: receiving environmental information of a common field of view from multiple sensors; generating multiple hypotheses about an object within the common field of view based on the environmental information, wherein the multiple hypotheses include at least one set of hypotheses excluding the environmental information from at least one sensor of a first sensor type; determining an object state of the object based on a subset of the multiple hypotheses, wherein the subset includes the at least one set of hypotheses excluding the environmental information from the at least one sensor of the first sensor type; and tracking the object based on the determined object state.

[0080] Example 38: The non-volatile, computer-readable medium of Example 37, wherein the set of operations further includes updating the multiple hypotheses over time based on additional environmental information received from the multiple sensors.

[0081] Example 39: The non-volatile computer-readable medium of Example 38, wherein the set of operations further includes detecting a fault associated with the at least one sensor of the first sensor type, wherein determining the object state of the object based on the subset of multiple hypotheses takes place in response to the detection of the fault associated with the at least one sensor of the first sensor type.

[0082] Example 40: The non-volatile, computer-readable medium of one of Examples 37 to 39, wherein the subset includes other hypotheses including environmental information from other sensors of the first sensor type.

[0083] Specific embodiments have been described in the foregoing specification. However, a person skilled in the art will recognize that various modifications and alterations can be made without deviating from the scope of protection of the invention as set forth in the claims below. Accordingly, the specification and the figures should be regarded as illustrative rather than limiting, and all such modifications are to be included within the scope of protection of the present teachings.

[0084] The benefits, advantages, solutions to problems, and any one or more elements that may cause any benefit, advantage, or solution to occur or be more pronounced, shall not be considered critical, necessary, or essential features or elements of any or all claims. The invention is defined solely by the attached claims, including any amendments made during the pendency of this application and all equivalents of these claims in published form.

[0085] Furthermore, in this document, relational expressions such as first and second, above and below, and the like may be used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any such actual relationship or order between such entities or actions.The expressions “includes,” “comprising,” “includes,” “indicating,” “contains,” “including,” “contains,” “containing,” or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, procedure, article, or facility that includes, exhibits, incorporates, or contains a list of elements may not only include those elements but may also include other elements not expressly listed or inherent in such process, procedure, article, or facility. An element preceded by “includes…,” “exhibits…,” “includes…,” or “contains…” does not, without further limitations, preclude the presence of additional identical elements in the process, procedure, article, or facility that includes, exhibits, incorporates, or contains the element.The terms "one" and "a" are defined as one or more, unless expressly stated otherwise herein. The terms "essentially," "substantially," "approximately," "about," or any other version thereof are defined as "obvious" as would be understood by a person skilled in the art, and in one non-limiting embodiment, the term is defined as being within 10%, in another embodiment within 5%, in another embodiment within 1%, and in yet another embodiment within 0.5%. The term "coupled," as used herein, is defined as connected, although not necessarily directly and not necessarily mechanically. A device or structure that is "configured" in a certain way is configured at least in that way, but may also be configured in ways not listed.

[0086] It is understood that some embodiments may consist of one or more generic or specialized processors (or “processing devices”), such as microprocessors, digital signal processors, custom processors, and field-programmable gate arrays (FPGAs), and unique stored program instructions (including both software and firmware) controlling the one or more processors to implement, in conjunction with certain non-processor circuitry, some, most, or all of the functions of the method and / or device described herein. Alternatively, some or all of the functions could be implemented by a state machine that does not have any stored program instructions, or in one or more application-specific integrated circuits (ASICs) in which each function, or some combinations of certain functions, are implemented as custom logic.Of course, a combination of the two approaches could be used.

[0087] Furthermore, one embodiment can be implemented as a computer-readable storage medium containing computer-readable code for programming a computer (which, for example, includes a processor) to perform a method as described and claimed herein. Examples of such computer-readable storage media include, but are not limited to, a hard disk, a CD-ROM, an optical storage device, a magnetic storage device, a ROM (read-only memory), a PROM (programmable read-only memory), an EPROM (erasable programmable read-only memory), an EEPROM (electrically erasable programmable read-only memory), and flash memory.Furthermore, it is expected that an average professional, regardless of potentially considerable effort and many design possibilities motivated, for example, by available time, current technology, and economic considerations, when guided by the concepts and principles disclosed herein, will simply be able to generate such software instructions, programs, and ICs with minimal experimentation.

[0088] The summary of the disclosure is provided to enable the reader to quickly ascertain the essence of the technical disclosure. It is submitted with the understanding that it is not intended to be used for interpreting or limiting the scope of protection or the meaning of the claims. It can be seen in the foregoing detailed description that various features of different embodiments are grouped together for the purpose of optimizing the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly set forth in each claim. Instead, the subject matter of the invention, as reflected in the following claims, lies in fewer than all the features of any single disclosed embodiment.Therefore, the following claims are hereby included in the detailed description, each claim being a separate claimed subject matter.

Claims

[1] Automated driving system for a vehicle, the system comprising: multiple sensors; a storage facility; and an electronic processor that is communicatively coupled with the memory and the multiple sensors, wherein the electronic processor is designed to: Received from the multiple processors, from environmental information of a common field of view, Generating, based on environmental information, multiple hypotheses regarding an object within the common field of view, wherein the multiple hypotheses include at least one set of hypotheses based exclusively on environmental information from at least one sensor of a first sensor type, Determine, based on a subset of several hypotheses, an object state of the object, wherein the subset includes the at least one set of hypotheses excluding the environmental information from the at least one sensor of the first sensor type, and Performing a vehicle maneuver based on the specified object state; wherein a first sensor of the multiple sensors is a radar sensor, and wherein a second sensor of the multiple sensors is a lidar sensor; wherein, for generating, based on the environmental information, the multiple hypotheses regarding the object within the common field of view, wherein the multiple hypotheses include at least one set of hypotheses excluding the environmental information from the at least one sensor of the first sensor type, the electronic processor is further designed to: Generating a first set of hypotheses from among several hypotheses, at least partially based on the environmental information received from the radar sensor and the lidar sensor, Generating a second set of hypotheses from the multiple hypotheses at least partially based on the environmental information received by the radar sensor, and not based on the environmental information received by the lidar sensor, and Generating a third set of hypotheses from the multiple hypotheses at least partially based on the environmental information received by the lidar sensor, and not based on the environmental information received by the radar sensor. [2] Automated driving system according to claim 1, wherein the electronic processor is further configured to update the multiple hypotheses over time based on additional environmental information received from the multiple sensors. [3] Automated driving system according to claim 2, wherein the electronic processor is further configured to detect a fault associated with the at least one sensor of the first sensor type, and wherein the determination, based on the subset of several hypotheses, of the object state of the object takes place in response to the detection of the fault associated with the at least one sensor of the first sensor type. [4] Automated driving system according to claim 1, wherein the multiple sensors comprise two or more sensors selected from a group consisting of the following: one or more radar sensors, one or more lidar sensors, one or more image sensors, and one or more ultrasonic sensors. [5] Automated driving system according to claim 1, wherein for determining, based on the subset of several hypotheses, the object state of the object, wherein the subset includes the at least one set of hypotheses excluding the environmental information from the at least one sensor of a first sensor type, the electronic processor is further configured to determine the object state of the object based on the third set of hypotheses. [6] Automated driving system according to claim 1, wherein the subset includes other hypotheses including environmental information from other sensors of the first sensor type. [7] Method for operating an automated driving system, the method comprising: Receiving, with an electronic processor, environmental information from a common field of view of multiple sensors; Generating, with the electronic processor, several hypotheses regarding an object within the common field of view based on the environmental information, wherein the several hypotheses include at least one set of hypotheses exclusively of the environmental information from at least one sensor of a first sensor type; Determine, using the electronic processor, an object state of the object based on a subset of several hypotheses, wherein the subset includes at least one set of hypotheses excluding environmental information from at least one sensor of the first sensor type; and Performing, with the electronic processor, a vehicle maneuver based on the specified object state; wherein a first sensor of the multiple sensors is a radar sensor, and wherein a second sensor of the multiple sensors is a lidar sensor; where generating multiple hypotheses regarding the object within the common field of view based on environmental information further includes: Generating a first set of hypotheses from among several hypotheses, at least partially based on the environmental information received from the radar sensor and the lidar sensor, Generating a second set of hypotheses from the multiple hypotheses at least partially based on the environmental information received by the radar sensor, and not based on the environmental information received by the lidar sensor, and Generating a third set of hypotheses from the multiple hypotheses at least partially based on the environmental information received by the lidar sensor, and not based on the environmental information received by the radar sensor. [8] The method of claim 7, further comprising: Updating, with the electronic processor, which generates multiple hypotheses over time based on additional environmental information received from the multiple sensors. [9] The method of claim 8, further comprising: Detect, with the electronic processor, a fault associated with at least one sensor of the first sensor type, where the determination of the object state of the object takes place based on the subset of multiple hypotheses in response to the detection of the fault associated with the at least one sensor of the first sensor type. [10] Method according to claim 7, wherein the multiple sensors comprise two or more sensors selected from a group consisting of the following: one or more radar sensors, one or more lidar sensors, one or more image sensors, and one or more ultrasonic sensors. [11] Method according to claim 7, wherein determining the object state of the object based on the subset of several hypotheses, wherein the subset includes the at least one set of hypotheses excluding the environmental information from the at least one sensor of the first sensor type, further includes determining the object state of the object based on the third set of hypotheses. [12] Method according to claim 7, wherein the subset includes other hypotheses including environmental information from other sensors of the first sensor type. [13] Non-volatile, computer-readable medium comprising instructions which, when executed by an electronic processor, cause the electronic processor to perform a set of operations comprising: Receiving environmental information from a shared field of view from multiple sensors; Generating multiple hypotheses about an object within the common field of view based on environmental information, wherein the multiple hypotheses include at least one set of hypotheses based exclusively on environmental information from at least one sensor of a first sensor type; Determining an object state of the object based on a subset of several hypotheses, wherein the subset includes at least one set of hypotheses excluding environmental information from at least one sensor of the first sensor type; wherein a first sensor of the multiple sensors is a radar sensor, and wherein a second sensor of the multiple sensors is a lidar sensor; where generating multiple hypotheses regarding the object within the common field of view based on environmental information further includes: Generating a first set of hypotheses from among several hypotheses, at least partially based on the environmental information received from the radar sensor and the lidar sensor, Generating a second set of hypotheses from the multiple hypotheses at least partially based on the environmental information received by the radar sensor, and not based on the environmental information received by the lidar sensor, and Generating a third set of hypotheses from the multiple hypotheses at least partially based on the environmental information received by the lidar sensor, and not based on the environmental information received by the radar sensor; and Performing a vehicle maneuver based on the specified object state. [14] Non-volatile computer-readable medium according to claim 13, wherein the set of operations further includes updating the multiple hypotheses over time based on additional environmental information received from the multiple sensors. [15] Non-volatile computer-readable medium according to claim 14, wherein the set of operations further includes detecting a fault associated with the at least one sensor of the first sensor type, wherein determining the object state of the object based on the subset of multiple hypotheses takes place in response to the detection of the fault associated with the at least one sensor of the first sensor type. [16] Non-volatile computer-readable medium according to claim 13, wherein the subset includes other hypotheses including environmental information from other sensors of the first sensor type.

Images