Multi-hypothesis object tracking for autonomous driving systems

Through the multi-assumption object tracking method, the electronic processor generates pseudo-measurement data and associated hypothesis sets, which solves the object tracking problem caused by wrong measurement models and preprocessing in the autonomous driving system, improves the accuracy and robustness of object tracking, and reduces the Collective risk.

CN112009468BActive Publication Date: 2024-11-26ROBERT BOSCH GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010482868.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-05-30
Filing Date
2020-06-01
Publication Date
2024-11-26
Estimated Expiration
2040-06-01

AI Technical Summary

Technical Problem

In the autonomous driving system, the wrong classification of object tracking is caused by incorrect measurement models and sensor data preprocessing, resulting in incorrect object status and vehicle control, increasing the risk of collision.

Method used

Using a multi-hypothesis object tracking method, the environment information is received through an electronic processor, pseudo-measurement data is generated, and the association hypothesis set is generated based on the combination of multiple measurement models and preprocessing steps, and the object state is determined, thereby controlling the vehicle and improving robustness. .

Benefits of technology

Improves the accuracy and robustness of object tracking, reduces incorrect vehicle control, and reduces collision risk.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112009468B_ABST
    Figure CN112009468B_ABST
Patent Text Reader

Abstract

A method and system for multi-hypothesis object tracking for an autonomous driving system. A system includes an electronic processor configured to receive environmental information and generate pseudo-measurement data associated with an object within an environment of a vehicle. The electronic processor is further configured to determine a set of associated hypotheses about the object based on the environmental information and the pseudo-measurement data. The electronic processor is further configured to determine an object state of the object based on the set of associated hypotheses. The electronic processor is further configured to control the vehicle based on the determined object state.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Related Applications

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 854,821, filed May 30, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] Various embodiments relate to multi-hypothesis object tracking for autonomous driving systems, and more particularly, to robustness against errors in object tracking utilizing a multi-hypothesis approach. Summary of the invention

[0004] Driver assistance and automated driving (collectively referred to herein as "automated driving") use various sensors with multiple overlapping fields of view to detect one or more objects within the area or environment surrounding the vehicle, such as another vehicle, a pedestrian, a lane marking, a curb, or the like. Using the sensor information, the automated driving system can determine one or more parameters associated with the detected object, such as a position, a direction of movement, or the like (a determined object state). The automated driving system then controls the vehicle based on the determined object state. Accurately determining the object state enables appropriate control of the vehicle (e.g., steering to avoid collision with the object).

[0005] However, errors often occur in object tracking, especially due to incorrect measurement models, errors in sensor data preprocessing, or a combination of the above two. Misclassification of detected objects may result in the use of incorrect measurement models. As an example, when an object is incorrectly classified as a cyclist when it is actually a vehicle, a measurement model dedicated to cyclists may be used to track the object. However, since the object is not actually a cyclist, the use of the cyclist measurement model may result in inappropriate control of the vehicle. As another example, during the preprocessing of sensor data, when there are actually two objects, the laser radar stixel aggregation to form an "L" shape may mistakenly detect an object. Such errors may result in incorrect object states, track ruptures, or a combination of the above two. As a result, the vehicle may be controlled in an inappropriate manner due to incorrect object states.

[0006] Therefore, there is a need to improve the robustness against incorrect assumptions in the measurement model, data preprocessing, or a combination of the above two, which ultimately improves the object tracking accuracy for use by the autonomous driving system. To address these and other problems, the embodiments described herein provide, among other things, methods and systems for multi-hypothesis object tracking, such that hypothesis formation is used to achieve robustness against errors in the measurement model, data preprocessing, and the like. The embodiments described herein provide multiple measurement models, preprocessing steps, or a combination of the above two that are systematically used in parallel with various assumptions. The assumptions can be about object class, object form, a diverse method for generating clusters from raw measurement results, and the like.

[0007] For example, one embodiment provides an autonomous driving system. The system includes an electronic processor configured to receive environmental information and generate pseudo-measurement data associated with an object within an environment of a vehicle. The electronic processor is further configured to determine a set of association hypotheses related to the object based on the environmental information and the pseudo-measurement data. The electronic processor is further configured to determine an object state of the object based on the set of association hypotheses. The electronic processor is further configured to control the vehicle based on the determined object state.

[0008] Another embodiment provides a method for multi-hypothesis object tracking. The method includes receiving environmental information and generating pseudo-measurement data using an electronic processor. The method also includes generating a set of association hypotheses for objects within a vehicle's surroundings based on the environmental information and the pseudo-measurement data. The method also includes determining, using the electronic processor, an object state of the object based on the set of association hypotheses. The method also includes controlling the vehicle based on the determined object state.

[0009] Yet another embodiment provides a non-transitory computer readable medium storing instructions that, when executed by an electronic processor, perform a set of functions. The set of functions includes receiving environmental information of a field of view from at least one sensor. The set of functions also includes generating pseudo-measurement data. The set of functions also includes generating a set of association hypotheses for objects within the field of view based on the environmental information and the pseudo-measurement data. The set of functions also includes determining an object state of the object based on the set of association hypotheses. The set of functions also includes determining a vehicle maneuver based on the determined object state. The set of functions also includes controlling a vehicle based on the vehicle maneuver.

[0010] Other aspects and embodiments will become apparent by consideration of the detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1A-1B Example objects within the vehicle's surroundings are illustrated.

[0012] Figure 2An example environment around a vehicle is illustrated.

[0013] Figure 3 A vehicle equipped with a system for multi-hypothesis object tracking in accordance with some embodiments is schematically illustrated.

[0014] Figure 4 Schematically illustrates a method according to some embodiments Figure 3 The controller of the system.

[0015] Figure 5 An example common field of view between two sensors is schematically illustrated in accordance with some embodiments.

[0016] Figure 6 is an illustration of a method according to some embodiments Figure 3 Flowchart of a method for multi-hypothesis object tracking performed by a system.

[0017] Figure 7 is an illustration of a method according to some embodiments Figure 3 The system implements a process model for determining an object based on environmental information. DETAILED DESCRIPTION

[0018] Before explaining any embodiment in detail, it should be understood that the embodiments are not limited in their application to the construction details and component arrangements set forth in the following description or illustrated in the accompanying drawings. Other embodiments are also possible, and the embodiments described and / or illustrated herein can be practiced or implemented in various ways. For example, although the examples described herein relate to an autonomous driving system, in other embodiments, the methods described herein may be applied to a driver assistance system, a traffic monitoring system, a traffic control system, a safety supervision system, or the like. It should also be understood that the term "vehicle" refers to a variety of vehicles, including, for example, passenger cars, trucks, boats, motorcycles, drones, and the like.

[0019] It should also be noted that multiple hardware- and software-based devices and multiple components of different structures can be used to implement the embodiments disclosed herein. In addition, each embodiment may include hardware, software, and electronic components or modules, which, for the purpose of discussion, may be illustrated and described as if most components are implemented only in hardware. However, those of ordinary skill in the art and based on reading this detailed description will recognize that, in at least one embodiment, the electronic-based aspects can be implemented in software that can be executed by one or more processors (e.g., stored on a non-transitory computer-readable medium). Therefore, it should be noted that multiple hardware- and software-based devices and multiple components of different structures can be used to implement various embodiments. It should also be understood that although some of the drawings illustrate hardware and software located in a particular device, these depictions are for illustrative purposes only. In some embodiments, the illustrated components may be combined or divided into separate software, firmware, and / or hardware. For example, instead of being located in a single electronic processor and executed by a single electronic processor, logic and processing may be distributed between multiple electronic processors. Regardless of how they are combined or divided, the hardware and software components may be located on the same computing device, or may be distributed between different computing devices connected by one or more networks or other suitable communication links.

[0020] In driver assistance and autonomous driving, objects in the vehicle's surroundings may be depicted in the form of an object list describing the objects. In this case, the objects may be formed by sensor data fusion, object tracking over time, or a combination of the two. Based on the object properties, a function determined by the vehicle's processor determines whether a reaction should be made to the object, and if so, what the reaction should be. In autonomous driving, especially in the field of sensor data fusion, high requirements are expected in terms of object quality, so as to avoid false negatives, so that objects that are actually present are not accidentally missed / ignored, as well as to avoid false positives, such as avoiding false triggering due to the wrong assumption of objects that do not exist in reality.

[0021] As described above, errors in object tracking may occur due to an erroneous measurement model, errors in sensor data preprocessing, or a combination of the above. An incorrect classification of an object may result in an erroneous measurement model. Errors in the measurement model may result in erroneous updates, which in turn may result in incorrectly determined object states (e.g., object classification, dynamic state, object extension, or a combination of the above), tracking breaks, or a combination of the above.

[0022] As an example, Figure 1A-1B The diagram shows the use of a radio radar sensor 60 to detect a secondary vehicle 50 and a cyclist 55, respectively. Figure 1AAs seen in FIG. 6 , the radio radar reflection (indicated by the symbol associated with reference numeral 62 ) is located at the nearest edge or corner of the secondary vehicle 50 relative to the radio radar sensor 60 . In contrast, as shown in FIG. Figure 1B As seen in FIG. 6 , the primary reflection (indicated by the symbol associated with reference numeral 65) is located in the middle of the cyclist 55 (e.g., a person sitting on a bicycle). Therefore, a misclassification of an object (such as classifying a secondary vehicle 50 as a cyclist 55 or vice versa) may ultimately result in utilizing an erroneous measurement model.

[0023] Errors in the pre-processing of the measurements (e.g., during aggregation) may also lead to similar undesirable results. As an example, Figure 2 An example environment around a vehicle 70 (equipped with an automated driving system) is illustrated. In the example shown, the environment around the vehicle 70 includes a first object 75A and a second object 75B (both depicted as secondary vehicles). Figure 2 As seen in FIG. 8 , the aggregation of LiDAR columns of pixels to form an “L” shape 80 may incorrectly determine that there is only one object instead of two objects (in this example, first object 75A and second object 75B) that exist in reality. This may result in spurious measurements (described in more detail below) caused by the aggregation in the sensor fusion process not being associated to form one object, and thus not updating the object with LiDAR data, which may result in tracking being broken, or this may result in updates occurring with partially incorrect measurements, which may result in incorrect object states. Due to such errors, the autonomous driving system may perform incorrect vehicle maneuvers or may not react appropriately to the object.

[0024] Measurement models and preprocessing steps of sensor data are increasingly used together with machine learning methods such as deep learning. These can be susceptible to errors in unpredictable situations. Such errors can have consequences in situations where low false negative rates are required. This can lead to erroneous updates of objects and, therefore, tracking breaks. As an example, errors can be caused by the object itself. For example, the object could be one that was not present in the training data of the deep learning / neural network, which could lead to false results. In the event of such errors, the assisted / autonomous driving system may no longer be able to react appropriately to the object, which could increase the risk of a collision.

[0025] To address these and other issues, embodiments described herein provide, among other things, methods and systems for multi-hypothesis object tracking such that hypothesis formation is used to achieve robustness against errors in measurement models, data preprocessing, and the like. Embodiments described herein provide multiple measurement models, preprocessing steps, or combinations thereof that are systematically used in parallel with various hypotheses such as object classification, object form, diverse methods for generating clusters from raw measurements, and the like.

[0026] Figure 3 A system 100 for multi-hypothesis object tracking of an autonomous driving system for a vehicle 105 is illustrated. Although illustrated as a four-wheeled vehicle, the vehicle 105 may encompass vehicles of various types and designs. For example, the vehicle 105 may be a car, motorcycle, truck, bus, semi-tractor, or other type of vehicle.

[0027] In the example shown, the system 100 includes a controller 110, a plurality of sensors 115 (collectively referred to herein as "sensors 115" and individually as "sensors 115"), a steering system 130, a braking system 135, and an acceleration control system 140. In some embodiments, the system 100 includes, in various configurations, Figure 3 , and may perform other functions in addition to the functions described herein. As an example, in some embodiments, system 100 includes Figure 3 The four sensors 115 shown in FIG. 1 may be a different number of sensors 115 , such as a single sensor 115 .

[0028] like Figure 4 As shown in FIG, the controller 110 includes an electronic processor 200 (e.g., a microprocessor, an application specific integrated circuit, or another suitable electronic device), a memory 205 (e.g., one or more non-transitory computer-readable storage media), and a communication interface 210. The electronic processor 200, the memory 205, and the communication interface 210 communicate via one or more data connections or buses, or a combination thereof. Figure 4 The controller 110 shown in FIG. 1 represents one example, and in some embodiments, the controller 110 includes, in different configurations, Figure 4 1. The controller 110 may be implemented in several independent controllers (e.g., programmable electronic controllers) that are each configured to perform a specific function or sub-function. Alternatively or additionally, the controller 110 may include a sub-module that includes an additional electronic processor, memory, or application-specific integrated circuit (ASIC) for handling input / output functions, signal processing, and application of the methods listed below. In addition, in some embodiments, the controller 110 performs functions other than those described herein.

[0029] The electronic processor 200 is configured to access and execute computer-readable instructions ("software") stored in the memory 205. The software may include firmware, one or more applications, program data, filters, rules, one or more program modules, and other executable instructions. For example, the software may include instructions and associated data for performing a set of functions (including the methods described herein). For example, in some embodiments, the electronic processor 200 executes instructions for controlling the steering system 130, the braking system 135, the acceleration control system 140, or another vehicle system to perform an action (e.g., a vehicle maneuver or vehicle behavior) in accordance with the autonomous driving system of the vehicle 105. For example, as Figure 4 As shown in , the memory 205 can store the autonomous driving system 220. The autonomous driving system 220 controls the vehicle 105 (e.g., the steering system 130, the braking system 135, the acceleration control system 140, another vehicle system, or a combination thereof) to perform vehicle maneuvers with limited or no input from the driver of the vehicle 105.

[0030] return Figure 4 , the communication interface 210 allows the controller 110 to communicate with (eg, receive input from and provide output to) a device external to the controller 110. For example, as in Figure 3 As seen in FIG. 1 , the controller 110 may be communicatively connected to one or more of the following: the sensor 115, the steering system 130, the braking system 135, and the acceleration control system 140. In some embodiments, the communication interface 210 includes a port for receiving a wired connection with a device external to the controller 110. Thus, in some embodiments, the controller 110 is directly coupled to one or more components of the system 100 via a dedicated wired connection. Alternatively or additionally, the communication interface 210 communicates with a communication bus (e.g., a controller area network (“CAN”)) to communicate indirectly with devices external to the controller 110. Thus, in other embodiments, the controller 110 is communicatively coupled to one or more components via a shared communication link (such as a vehicle communication network or bus (e.g., CAN bus, Ethernet, or FlexRay)) or a wireless connection (via a transceiver). Each component of the system 100 may communicate with the controller 110 using a variety of communication types and protocols.

[0031] Sensors 115 are configured to detect or collect measurements or data associated with the environment or surroundings of vehicle 105 ("environmental information"). Sensors 115 may include, for example, a radio radar sensor, a lidar sensor, an ultrasonic sensor, an image sensor (or camera), or the like. In some embodiments, system 100 includes more than one type of sensor (e.g., a radio radar sensor and an image sensor). Environmental information may be associated with one or more objects within the environment surrounding vehicle 105. Objects may include, for example, pedestrians, cyclists, another vehicle, or the like. Thus, environmental information may include, for example, the distance between vehicle 105 and one or more objects in the environment surrounding vehicle 105, the position of vehicle 105 relative to one or more objects in the environment surrounding vehicle 105, or a combination of the above.

[0032] The sensors 115 may be located at various locations or positions throughout the vehicle 105 (e.g., inside the vehicle 105, outside the vehicle 105, or a combination of the two). As an example, the sensor 115 may be mounted externally to a portion of the vehicle 105, such as on a side mirror, front, rear, or one or more sides of the vehicle 105. Alternatively or additionally, one or more sensors 115 may be outside the vehicle or away from the vehicle, such as on an infrastructure surrounding the vehicle 105. As an example, the sensor 115 may be mounted on an infrastructure surrounding the vehicle 105 so that the sensor 115 may deliver environmental information to the autonomous driving system 220 (e.g., transmitted via vehicle-to-infrastructure communications, delivered via a backend server, or the like). Thus, any sensor having a field of view capable of observing a detected object may be a provider of environmental information.

[0033] The sensors 115 are associated with corresponding fields of view of an area of ​​the vehicle 105's surroundings. One or more operating parameters, such as the range of the field of view of the sensor 115, may be based on the configuration of the sensor 115. Thus, the sensors 115 may have fields of view of different sizes (and depths). However, the sensors 115 may have fields of view of similar (or the same) sizes (and depths). Thus, in some embodiments, the sensors 115 have fields of view of the same range, different ranges, or a combination of both. In some embodiments, the sensors 115 are positioned such that the sensor 115 has an overlapping field of view with at least one additional sensor 115. The overlapping area that overlaps the field of view of each sensor 115 is referred to herein as a "common field of view." As an example, Figure 5 A vehicle 105 is illustrated having a first sensor 115A and a second sensor 115B. As seen in the illustrated example, the first sensor 115A has a first field of view 305A, and the second sensor 115B has a second field of view 305B. Figure 5Also illustrated is a common field of view 310 between the first sensor 115A and the second sensor 115B.

[0034] although Figure 5 The example shown includes two sensors (a first sensor 115A and a second sensor 115B) having a common field of view (common field of view 310), but the system 100 may include one or more additional sensors (separate from but similar to the sensors 115 described herein), wherein the one or more additional sensors share a common field of view (e.g., a second common field of view). As an example, the system 100 may include a first sensor 115A, a second sensor 115B, and a third sensor. The third sensor may have a common field of view (e.g., a second common field of view) with the second sensor 115B. In this example, the second sensor 115B has a first common field of view with the first sensor 115A and a second common field of view with the third sensor. As another example, the system 100 may include a first sensor 115A, a second sensor 115B, a third sensor, and a fourth sensor. In this example, the first sensor 115A and the second sensor 115 may have a first common field of view, and the third sensor and the fourth sensor may have a second common field of view. Thus, the system 100 may include any number of sensors 115 with any number of common fields of view shared between one or more sensors 115.

[0035] Although not shown, other components of system 100 may include similar components (electronic processors, memory, and communication interfaces) as controller 110. However, in some embodiments, other components of system 100 include more, fewer, or different components than controller 110 in different configurations.

[0036] As described above, the electronic processor 200 of the controller 110 executes instructions for multi-hypothesis object tracking of the system 100 in order to achieve robustness against errors in the measurement model, data pre-processing, and the like. The electronic processor 200 executes instructions to perform Figure 6 , method 600 is used for multi-hypothesis object tracking of the autonomous driving system 220. The method 600 is described as being performed by the system 100, and in particular by the autonomous driving system 220 (as performed by the electronic processor 200). However, as described above, the functions (or portions thereof) described with respect to the method 600 may be performed by other devices (such as another controller associated with the vehicle 105), or distributed among multiple devices (such as multiple controllers associated with the vehicle 105). As an example, in some embodiments, the functions (or portions thereof) described with respect to the method 600 may be performed by a device external to the vehicle 105 or a remote device, such as, for example, where the vehicle 105 is remotely controlled.

[0037] like Figure 6 As shown in , the method 600 includes receiving environmental information using the electronic processor 200 (at block 605). As described above, the environmental information may include, for example, the distance between the vehicle 105 and one or more objects in the environment surrounding the vehicle 105, the position of the vehicle 105 relative to one or more objects in the environment surrounding the vehicle 105, or a combination of the above two. In some embodiments, the environmental information includes one or more attributes about objects in the environment surrounding the vehicle 105. The attributes of the object may relate to, for example, the position of the object relative to the vehicle 105, the speed of the object, the direction in which the object is moving, the distance between the vehicle 105 and the object, etc.

[0038] The electronic processor 200 receives environmental information from one or more sensors 115 (via the communication interface 210). In some embodiments, the electronic processor 200 receives environmental information continuously (e.g., in real time or near real time). However, in other embodiments, the electronic processor 200 receives environmental information cyclically or periodically (e.g., ten or twenty times per second). In some embodiments, the environmental information is associated with a common field of view. For example, as described above with respect to Figure 5 As described, in some embodiments, two or more sensors 115 share a common field of view (e.g., Figure 5 ). Alternatively or additionally, in some embodiments, sensor 115 includes various types of sensors, such as radio radar sensors, image sensors, radio radar sensors, and the like. Thus, in some embodiments, the environmental information received by electronic processor 200 may have one or more different media (e.g., image-based, ultrasound-based, etc.). As an example, electronic processor 200 may receive a first set of environmental information from a radio radar sensor (e.g., as first sensor 115) and a second set of environmental information from a lidar sensor (e.g., as second sensor 115). According to this example, the first set of environmental information is radio radar-based and the second set of environmental information is lidar-based.

[0039] The electronic processor 200 also generates pseudo measurement data (at block 610). The pseudo measurement data may include, for example, pseudo measurement results, measurement model assumptions, or a combination of the two. The pseudo measurement result may be, for example, an abstraction of a set of raw sensor measurements (e.g., a description of a measurement cluster). Alternatively, the raw measurement result may be used directly as a pseudo measurement result. Alternatively or additionally, a pseudo measurement result may be generated as a confirmation of an estimate, an indication that actual environmental information does not exist, or a combination of the two. As an example, a pseudo measurement result may indicate that actual environmental information does not exist (e.g., negative information), which may confirm an estimate or known occlusion or deplausibilize an object if there is no occlusion to explain the missing measurement result. In some embodiments, the pseudo measurement result is associated with an object within the environment of the vehicle 105, such as, for example, a three-dimensional bounding box that models a cluster of lidar points associated with the object. The measurement model assumption may be based on one or more assumptions, such as an assumed object classification or type ("object classification assumption"). Alternatively or additionally, the measurement model assumption may be a general assumption without an object classification assumption. Therefore, in some embodiments, the electronic processor 200 generates pseudo measurement data by generating one or more pseudo measurement results. Alternatively or additionally, in some embodiments, the electronic processor 200 generates pseudo measurement data by generating one or more measurement model hypotheses, wherein each measurement model hypothesis is generated based on an object classification hypothesis, a no object classification hypothesis, or a combination of the two. For example, Figure 7 As seen in FIG. 1 , the electronic processor 200 can generate a first measurement model hypothesis 705 and a second measurement model hypothesis 710 (as pseudo measurement data). In the example shown, the first measurement model hypothesis 705 assumes that the object is a vehicle, while the second measurement model hypothesis 710 assumes that the object is a cyclist.

[0040] Thus, in some embodiments, the electronic processor 200 generates a set of pseudo measurement data for each sensor measurement result, such that for each measurement result, a plurality of alternative pseudo measurement data hypotheses are generated using different modeling assumptions, aggregation assumptions, or the like. In such embodiments, different alternative versions of the pseudo measurement data may be used for each sensor measurement result.

[0041] In some embodiments, whether to generate a pseudo-measurement result or a measurement model assumption (as pseudo-measurement data) is based on the type of sensor, and the measurement model or pseudo-measurement result is based on the type of the sensor. For example, for a lidar sensor, the electronic processor 200 can generate a cluster of lidar points (e.g., via a deep neural network), which models the cluster as a three-dimensional bounding box. The three-dimensional bounding box can then be used as a pseudo-measurement result. Alternatively or additionally, the cluster of lidar points can be approximated, for example, by an L-shape, a line segment, an ellipse, or the like. The correspondingly described parameters can then be used as pseudo-measurements. Alternatively or additionally, the electronic processor 200 can generate a pseudo-measurement result based on the aggregation of lidar columnar pixels, where the aggregation does not incorporate object classification assumptions. As another example, for a radio radar sensor, the reflection model (measurement model) of the radio radar may depend on the object type, and the object type affects the measurement model used for object updating. Therefore, the electronic processor 105 can generate multiple measurement model assumptions with different assumptions (including general assumptions or not containing object classification assumptions).

[0042] In some embodiments, one or more assumptions for the measurement model, pseudo-measurement data, or a combination of the two include general assumptions, do not include object classification assumptions, or a combination of the two. These general models create robustness against, for example, incorrect classification information, classification assumptions, or a combination of the two. The purpose of the general assumptions is to become robust against incorrect assumptions by allowing reduced model granularity. For situations where more detailed measurement models or pseudo-measurement assumptions are not applicable (for example, covering rare object shapes or measurement situations), these general assumptions can be considered as fallback strategies. Although the accuracy of the fallback strategy may not be sufficient to ensure a comfortable response of the vehicle, it can avoid object loss and therefore avoid dangerous situations.

[0043] As described above, based on various one or more assumptions, the pseudo-measurement data may include one or more pseudo-measurement results, one or more measurement model assumptions, or a combination of the two. As a result, when more information (from later measurement scans or from other sensors) is available, the decision on which measurement model assumption / pseudo-measurement result to ultimately use may be pushed further back in the process chain. Its implementation in sensor data fusion is performed, for example, by using a multi-hypothesis approach during multiple data scans using methods such as labeled multi-Bernoulli ("LMB") filters. In some embodiments, one or more of the following assumptions are excluded: in which alternative pseudo-measurement results (e.g., generated by various cluster assumptions) are used simultaneously for object updates, because as a result, information from the measurement data is used multiple times, even if the independence of the measurements is assumed.

[0044] like Figure 6As seen in FIG. 6 , the electronic processor 200 also generates a set of association hypotheses for objects within the field of view (at block 615). In some embodiments, the field of view is a common field of view associated with two or more sensors 115, as described in more detail above. However, in other embodiments, the field of view is associated with a single sensor, such as in a single sensor system that obtains measurement data (environmental information) from a single sensor 115 over time. An association hypothesis generally refers to an assumption about a possible object state of an object (such as an object within a common field of view of two or more sensors 115 or an object within the field of view of one sensor 115). In object tracking, when an autonomous driving system (such as the autonomous driving system 220) utilizes information from one or more sources (such as one or more sensors 115), utilizes one or more measurement models, or a combination of the above two, the autonomous driving system 220 can determine more than one possible object state. Each possible object state is referred to herein as an association hypothesis.

[0045] Therefore, in some embodiments, the electronic processor 200 uses sensor fusion of environmental information ( Figure 7 The fusion of sensor information (at block 712) may involve one or more possible object states (as association hypotheses) for objects within the environment surrounding the vehicle 105. The object state may be based on one or more attributes of the object. For example, the object state may be based on the position of the object relative to the vehicle 105, the speed of the object, the direction of movement of the object, an object classification of the object that identifies what the object is (object classification) (e.g., whether the object is another vehicle, a motorcycle, a person, a bicycle, or the like), and the like. Thus, in some embodiments, determining an association hypothesis (or possible object state) includes determining one or more attributes associated with the object. The fusion of sensor information (at block 712) involves one or more measurement models, motion models, object tracking processes, or a combination of the foregoing (e.g., Figure 7 The first measurement model assumes 705 and the second measurement model assumes 710).

[0046] Thus, in some embodiments, a set of association hypotheses is associated with an object that is within a common field of view associated with two or more sensors 115 or within the field of view of a single sensor 115. The electronic processor 200 may generate the set of association hypotheses based on environmental information received from one or more sensors 115, pseudo-measurement data, or a combination thereof. The set of association hypotheses may be associated with or correspond to the pseudo-measurement data. For example, the set of association hypotheses may include an association hypothesis for each pseudo-measurement result, a measurement model hypothesis, or a combination thereof. Return to Figure 7, the electronic processor 200 may generate a first association hypothesis 715A and a second association hypothesis 715B (as a set of association hypotheses). The first association hypothesis 715B is associated with the second measurement model hypothesis 710, and the second association hypothesis 715A is associated with the first measurement model hypothesis 705. As an example, the environmental information received from the one or more sensors 115 may indicate the size of the object, the speed of the object, and the direction of movement of the object. Based on the environmental information in this example, the electronic processor 200 may generate an association hypothesis that the possible object state of the object is a vehicle type, where the object is traveling at a faster speed than the vehicle 105 and in the same direction as the vehicle 105.

[0047] In some embodiments, the electronic processor 200 updates, merges, and / or prunes one or more association hypotheses over time (e.g., each time cycle) as additional environmental information is received from the one or more sensors 115. In such embodiments, the association hypotheses (and thus the determined object states) are monitored and updated periodically (each period is referred to as a cycle) based on new information (such as new environmental information received from the one or more sensors 115). In some embodiments, the electronic processor 200 utilizes and manages multiple association hypotheses over time using, for example, a multiple hypothesis tracking (“MHT”) and random finite set (“RFS”) process including a labeled multi-Bernoulli (“LMB”) filter.

[0048] After generating the set of association hypotheses (at block 615), the electronic processor 200 then determines the object state of the object (at block 620). As an example, the determined object state may include an object class of a vehicle, where the object is (a) in the left lane, (b) in the same direction as the vehicle 105, and (c) traveling at a speed of 50 mph. The electronic processor 200 may determine the object state of the object based on the set of association hypotheses. Thus, in some embodiments, the determined object state is one of the association hypotheses included in the set of association hypotheses (i.e., one of the possible object states of the object). In some embodiments, the electronic processor 200 determines the object state of the object based on the probability associated with each association hypothesis included in the set of association hypotheses. For example, the electronic processor 200 may determine that the object state of the object is the association hypothesis (i.e., possible object state) with the highest probability (e.g., the association hypothesis that is most likely to be the correct object state of the object). Over time, and as more environmental information is received from the one or more sensors 115, the electronic processor 200 may combine or merge similar association hypotheses while removing association hypotheses that are determined to be less likely. Thus, the more hypotheses that are merged or combined, the higher the probability associated with the merged or combined associated hypotheses. As another example, the probability of an associated hypothesis can increase based on the number of possible hypotheses remaining. In other words, as more unlikely associated hypotheses are removed, the probability that the remaining associated hypotheses are correct (or true) becomes higher.

[0049] Alternatively or additionally, in some embodiments, the electronic processor 200 forwards one or more associated hypotheses and their corresponding probabilities to another component (e.g., for consideration of which possible object state is most likely to be the actual object state of the detected object). In some embodiments, for example, to limit computation time, constrain hypothesis growth, or a combination of both, a preliminary selection of one or more associated hypotheses is performed prior to sensor fusion (e.g., Figure 7 712).

[0050] like Figure 6As shown in , the electronic processor 200 may then control the vehicle 105 based on the determined object state of the object (at block 625). In some embodiments, the electronic processor 200 controls the vehicle 105 by determining whether to perform a vehicle maneuver based on the determined object state. Thus, in some embodiments, controlling the vehicle 105 based on the determined object state of the object includes: not performing a vehicle maneuver or performing a vehicle maneuver. Alternatively or additionally, in some embodiments, the electronic processor 200 controls the vehicle 105 by determining what vehicle maneuver (or reaction) to perform, the timing of the performance of the vehicle maneuver (e.g., when to perform the vehicle maneuver), etc. based on the determined object state. The determination of the vehicle maneuver (based on the determined object state) may be performed using one or more autonomous driving technologies, which are not discussed in detail herein for the sake of brevity.

[0051] The electronic processor 200 may perform a vehicle maneuver by controlling the steering of the vehicle 105, affecting the speed of the vehicle 105 (e.g., accelerating or braking the vehicle 105), etc. In some embodiments, the electronic processor 200 performs a vehicle maneuver by generating and transmitting one or more control signals to one or more components of the vehicle 105, such as the steering system 130, the braking system 135, the acceleration control system 140, and the like. In response to receiving the control signal, one or more components of the vehicle 105 may be controlled according to its corresponding control signal.

[0052] Thus, embodiments described herein can achieve robustness against errors in measurement models, data preprocessing, and the like, for example, by sorting out erroneous hypotheses based on subsequent measurements over time. For example, object tracking performance can be improved by acquiring and removing (or pruning) hypotheses that contain erroneous tracking-measurement associations (random errors) over time, while retaining several other hypotheses. Methods for maintaining a modest number of hypotheses can include, for example, merging similar hypotheses while removing (or pruning) unlikely hypotheses.

[0053] In the field of extended object tracking, various hypotheses can also be formed by another technique (aggregating sensor data). In these aggregation techniques, the impact of errors in the aggregation of measurement data is minimized because the hypothesis with the most consistent aggregation over time (indicating that the hypothesis is true or correct) accounts for the majority. However, the motivation of these aggregation techniques is to improve the performance of object tracking by improving the association and aggregation of measurement data, rather than to improve the robustness of the system against incorrect assumptions in measurement models and data preprocessing. However, with respect to the embodiments described herein, specific assumptions are determined to, for example, provide robustness against errors in measurement models and data preprocessing steps (e.g., sensor data aggregation, object classification, etc.) applied during the fusion of sensor data.

[0054] As described above, although the examples described herein relate to an autonomous driving system, in other embodiments, the methods and systems described herein may be applied to other systems, such as, for example, a driving assistance system, a traffic monitoring system, a traffic control system, a safety supervision system, etc. As an example, the sensor 115 may be mounted on an infrastructure. According to this example, the sensor 115 may deliver environmental information to the autonomous driving system (e.g., transmitted via vehicle-to-infrastructure communication, delivered via a backend server, or the like). Alternatively or additionally, the sensor 115 may collect environmental information related to traffic monitoring and control. As an example, the sensor 115 may detect environmental information related to the location of the observed traffic. According to this example, the electronic processor 200 may determine and control traffic monitoring and control actions, such as opening a shared lane, controlling traffic light timing, determining the price of premium traffic, etc. As another example, the sensor 115 may detect environmental information for a supervisory system (such as for monitoring the location of a vehicle, a pedestrian, or the like on a house). According to this example, the electronic processor 200 may determine and control supervisory system actions, such as whether an alarm should be triggered, based on the location of the vehicle or pedestrian. Thus, in some examples, the electronic processor 200 is not incorporated into or associated with a particular vehicle, such as the vehicle 100. Instead, the electronic processor 200 may be, for example, part of a traffic monitoring system or a supervision system, whether local (e.g., on a local traffic monitoring system or a supervision system) or remote (e.g., a cloud-based application server or a remote server).

[0055] Thus, various embodiments provide, among other things, methods and systems for multi-hypothesis object tracking for autonomous driving systems, thereby achieving robustness against errors in measurement models, data pre-processing, and the like. Various features and advantages of certain embodiments are set forth in the appended claims.

Claims

1. An automatic driving system for a vehicle, the system comprising: An electronic processor configured to: Receive environmental information, generating pseudo-measurement data associated with an object within the environment of the vehicle, comprising: generating a plurality of measurement model hypotheses as the pseudo-measurement data for each measurement result included in the environment information, wherein each of the plurality of measurement model hypotheses is based on a different assumption and comprises a first measurement model hypothesis based on a first assumption that the object is a first object type and a second measurement model hypothesis based on a second assumption that the object is a second object type, determining a set of association hypotheses related to the object based on the environmental information and the pseudo-measurement data, determining an object state of the object based on the set of association hypotheses, and The vehicle is controlled based on the determined state of the object.

2. The system according to claim 1, wherein: At least one of the plurality of measurement model hypotheses is a universal measurement model hypothesis.

3. The system according to claim 1, wherein: The electronic processor is configured to update the set of association hypotheses in response to receiving new environmental information.

4. The system according to claim 1, wherein: Each association hypothesis included in the set of association hypotheses is associated with a possible object state of the object.

5. The system according to claim 1, wherein: The electronic processor is configured to receive the environmental information from at least two sensors, the environmental information being associated with a common field of view of the at least two sensors.

6. The system according to claim 5, wherein: The pseudo measurement data includes negative information associated with the common field of view.

7. The system according to claim 1, wherein: The electronic processor is configured to determine an object state of the object as one of the association hypotheses included in the set of association hypotheses.

8. The system according to claim 7, wherein: Determining the object state as one of the association hypotheses is based on one of the association hypotheses having a higher probability than other association hypotheses included in the set of association hypotheses.

9. A method for multi-hypothesis object tracking, the method comprising: Receive environmental information; generating, with an electronic processor, pseudo measurement data by: generating a first measurement model hypothesis based on a first assumption that the object is of a first object type and generating a second measurement model hypothesis based on a second assumption that the object is of a second object type different from the first object type; generating a set of association hypotheses for objects in the vehicle's surroundings based on the environment information and the pseudo-measurement data; determining, with the electronic processor, an object state of the object based on the set of association hypotheses; and The vehicle is controlled based on the determined state of the object.

10. The method according to claim 9, further comprising: Receive new environmental information; and The set of association hypotheses is updated based on the new environmental information.

11. The method of claim 10, wherein updating the set of association hypotheses comprises: At least one association hypothesis is removed from the set of association hypotheses.

12. The method according to claim 9, wherein: Receiving the environmental information includes receiving the environmental information from at least two sensors, wherein the environmental information is associated with a common field of view of the at least two sensors.

13. The method according to claim 12, wherein: Generating the pseudo measurement data includes generating negative information associated with the common field of view.

14. The method according to claim 9, wherein: Generating the pseudo measurement data includes generating a measurement model hypothesis.

15. The method according to claim 9, wherein: Determining the object state includes determining the object state as one of the association hypotheses included in the set of association hypotheses.

16. The method according to claim 15, wherein: Determining the object state as one of the relevance hypotheses includes determining one of the relevance hypotheses to have a higher probability than other relevance hypotheses included in the set of relevance hypotheses.

17. A non-transitory computer readable medium storing instructions that, when executed by an electronic processor, perform a set of functions, the set of functions comprising: receiving environmental information of the field of view from at least one sensor; generating pseudo measurement data by: generating a first measurement model hypothesis based on a first assumption that the object is a first object type and generating a second measurement model hypothesis based on a second assumption that the object is a second object type different from the first object type; generating a set of association hypotheses for objects within the field of view based on the environmental information and the pseudo-measurement data; determining an object state of the object based on the set of association hypotheses; determining a vehicle maneuver based on the determined state of the object, and A vehicle is controlled based on the vehicle maneuver.

18. A computer program product comprising instructions which, when executed by an electronic processor, cause the electronic processor to perform the method according to any one of claims 9 to 16.