Methods and systems for multi-hypothesis-based fusion of sensor data

Abstract

This document describes a multi-hypothesis-based data fusion tracker. Each hypothesis corresponds to a different pseudo-measurement type. The fusion tracker automatically determines which pseudo-measurement type is more likely to be accurate in the current situation using a predefined error covariance associated with radar. The fusion tracker can rely on either a combination of radar and vision computations, or it can ignore vision-based pseudo-measurements and rely solely on radar pseudo-measurements. By selecting between three different bounding boxes (view-angle-based, view-lateral-position-based, or radar-based only), the fusion tracker can balance accuracy and speed when drawing, repositioning, or resizing bounding boxes, even in congested traffic or other high-volume conditions.

Description

Background Technology

[0001] In some vehicles, sensor fusion systems, or so-called "fusion trackers," combine information from multiple sensors to depict bounding boxes around objects that may obstruct driving. The combined sensor data provides a better estimate of the position of each object within the field of view (FOV) under various conditions. Resizing or repositioning these bounding boxes typically involves using expensive hardware that correlates low-level tracking and fuses sensor data at a speed sufficient to support computer-based decision-making for autonomous or semi-autonomous control. Some fusion trackers weight the low-level tracking based on error covariance. This error covariance is typically obtained from vision cameras and may not be reliable in all driving scenarios. The complexity of data fusion can be reduced using an extended Kalman filter (EKF). To maximize performance, an EKF can focus on the most linear sensor data; however, an EKF may miss correlations or become less accurate, discarding significant amounts of speed-related sensor data. Summary of the Invention

[0002] This document describes a multi-hypothesis-based fusion of sensor data. In one example, a method includes determining multiple first object tracks by a sensor fusion system based on first sensor data obtained from a first set of sensors. The method further includes determining a set of second object tracks by the sensor fusion system based on second sensor data obtained from a second set of sensors. The method also includes selecting a pseudo-measurement type from a plurality of pseudo-measurement types, the selected pseudo-measurement type being more likely to be accurate than each of the other pseudo-measurement types from the plurality of pseudo-measurement types, the selection being at least in part based on a predefined error covariance associated with the first set of sensors. The method further includes using the pseudo-measurement type to determine bounding boxes represented by the multiple first object tracks and the set of second object tracks. The method then outputs an indication of the bounding boxes as a match between one or more object tracks from the plurality of first object tracks and at least one object track from the set of second object tracks.

[0003] In one example, the system includes a processor configured to perform the methods described herein and other methods. In another example, a system including means for performing the methods described herein and other methods is described. This document also describes a non-transitory computer-readable storage medium having instructions that, when executed, configure the processor to perform the methods summarized above and other methods described herein.

[0004] This invention presents a simplified concept of multi-hypothesis-based fusion of sensor data, which will be further described below in the detailed embodiments and accompanying drawings. This invention is not intended to identify essential features of the claimed subject matter, nor is it intended to define the scope of the claimed subject matter. Therefore, although primarily described in the context of improving fusion tracker matching algorithms, multi-hypothesis-based fusion of sensor data can also be applied to other applications where it is desirable to match multiple low-level tracks at a high rate. Attached Figure Description

[0005] This document describes in detail one or more aspects of the multi-hypothesis-based fusion of sensor data with reference to the following figures. The same numbers are generally used throughout the figures to refer to similar features and components.

[0006] Figure 1 An example environment of the technology according to this disclosure is shown, in which the system is configured to perform multi-hypothesis-based fusion of sensor data.

[0007] Figure 2 An example of an automotive system configured to perform multi-hypothesis fusion of sensor data according to the technology of this disclosure is shown.

[0008] Figure 3 Showing from Figure 1-1 The environment is displayed in more detail.

[0009] Figure 4-1 and Figure 4-2 An example pseudo-boundary box defined under the multi-hypothesis fusion of sensor data according to the techniques of this disclosure is shown.

[0010] Figure 5 An example method for multi-hypothesis-based fusion of sensor data according to the techniques of this disclosure is shown. Detailed Implementation

[0011] Overview

[0012] Determining high-level matches between different sensor tracking methods (e.g., radar, vision cameras, lidar) can be challenging. A bounding box around a single object may correspond to one or more low-level object tracks, fused from multiple sensors with overlapping or nearly adjacent fields of view (FOVs). In crowded scenes (where multiple objects can be identified within a combined field of view), fusion trackers may struggle to resize and position bounding boxes around multiple nearby objects. The accuracy of these bounding boxes can be questionable. Before relying on this data, some fusion trackers weight object tracking based on the error covariance obtained from the vision camera. However, as a practical problem, this error covariance is not always reliable for fusion tracking. EKF can improve performance, but for speed improvements, large amounts of low-level tracking data are not used, as low-level tracking data does not contribute to improving bounding box accuracy.

[0013] Sensors can output pseudo-measurements to help fusion trackers calculate the position and size of bounding boxes assigned to objects. Pseudo-measurements can have varying degrees of accuracy; pseudo-measurements from vision cameras may be less accurate than, for example, radar. Range or longitudinal position output from a vision camera may be unusable in forward-driving scenarios. Even so, visual angles or vision-based lateral positions are likely sufficient for most cases and can help position the bounding box to the correct reference point. On the other hand, radar may be more reliable and accurate than these vision-based pseudo-states for distance and bounding box calculations. Therefore, several assumptions must be considered when evaluating whether pseudo-measurements from a particular type of sensor should be used for relative positioning estimation.

[0014] According to the technology and system disclosed herein, a multi-hypothesis-based data fusion tracker is described. Each hypothesis is consistent with a different pseudo-measurement type. The fusion tracker automatically determines which pseudo-measurement type is more likely to be accurate in the current situation using a predefined error covariance associated with radar. The fusion tracker may rely on one of two combined radar and vision calculations, or it may ignore vision-based pseudo-measurements and rely solely on radar pseudo-measurements. By selecting between three different bounding boxes (boxes based on visual angles, boxes based on visual lateral position, or boxes based solely on radar), the fusion tracker can balance accuracy and speed when drawing, repositioning, or resizing bounding boxes, even in congested traffic or other high-volume conditions.

[0015] Example Environment

[0016] Figure 1An example environment 100 according to the technology of this disclosure is shown, in which system 102 is configured to perform multi-hypothesis-based fusion of sensor data. In the depicted environment 100, system 102 is a vehicle. Sometimes also referred to as vehicle 102, system 102 can represent any type of device or machine, including manned and unmanned systems that can be used for various purposes. Some non-exhaustive and non-limiting examples of vehicle 102 include motorcycles, buses, tractors, semi-trailers, watercraft, aircraft, or other equipment or machines.

[0017] The object may be located near vehicle 102, for example, Figure 1 Another vehicle 110 is depicted in front of vehicle 102 and traveling in the same direction as vehicle 102. Utilizing sensor fusion system 104, vehicle 102 has an instrumented field of view (FOV) 114 that encompasses the other vehicle 110. Sensor fusion system 104 can capture the FOV 114 from any external surface of vehicle 102. Manufacturers can position radar and vision camera components connected to sensor fusion system 104 to give sensor fusion system 104 a specific FOV. For example, positioning the radar and / or vision camera in a certain way ensures that the FOV of sensor fusion system 104 includes the area above, near, or on the road where vehicle 102 may be traveling. Manufacturers can integrate at least a portion of sensor fusion system 104 into side mirrors, bumpers, roofs, or any other part of vehicle 102.

[0018] Sensor fusion system 104 includes fusion module 108 and one or more sensor interfaces 106, including radar interface 106-1 and vision camera interface 106-2. Although not described in... Figure 1 The exact details are shown, but the fusion module 108 executes on a processor or other hardware. During execution, the fusion module 108 may track objects based on sensor data obtained at radar interface 106-1 and vision camera interface 106-2. Radar interface 106-1 receives radar data from at least one radar of vehicle 102, and the vision camera receives camera data from one or more vision cameras of vehicle 102. The fusion module 108 of sensor fusion system 104 accesses radar interface 106-1 and vision camera interface 106-2 to acquire radar data and vision camera data, respectively. As will be understood from other parts of this specification, the sensor fusion system may include, in addition to Figure 1 Additional sensor interfaces 106 (e.g., LiDAR) in addition to those shown.

[0019] The fusion module 108 configures the sensor fusion system 104 to combine different types of sensor data obtained from the sensor interface 106 into object tracking, bounding boxes 112-1, 112-2, or other available forms for tracking objects in the FOV 114. The sensor fusion module 108 generates bounding boxes 112-1 and 112-2, each (conceptually) indicating the estimated size, shape, and relative position of another vehicle 110 compared to vehicle 102. The fusion module 108 determines multiple object tracks based on first sensor data (e.g., obtained from the radar interface 106-1), and identifies a set of object tracks based on second sensor data (e.g., obtained from the camera interface 106-2). Bounding box 112-1 corresponds to the estimated size and position of another vehicle 110 inferred from given information inferred from one or more radar object tracks included in the radar data obtained at the radar interface 106-1. From the visual camera interface 106-2, bounding boxes 112-2 are generated based on visual camera object tracking to estimate the size and position of another vehicle 110.

[0020] Bounding boxes 112-1 and 112-2 are offset; or in other words, the two bounding boxes 112-1 and 112-2 track another vehicle as two distinct objects, each with a unique size, shape, and relative position to vehicle 102. Even though both bounding boxes 112-1 and 112-2 are tracking the same vehicle 110, they appear separate. The fusion module 108 ultimately involves associating bounding boxes 112-1 and 112-2 so that they appear similar in size, shape, and position to correspond to the same portion of the same vehicle 110, rather than tracking and following different portions of one or two different vehicles.

[0021] To correlate radar data with vision camera data, fusion module 108 performs a sensor fusion process that matches low-level object tracking appearing in the radar data with corresponding low-level object tracking appearing in the vision camera data. Each object tracking inferred from the first and second sensor data is associated with an object in the FOV 114 (e.g., another vehicle 110). Fusion module 108 identifies object tracking corresponding to the same object from different sensor interfaces 106. By correlating sensor data obtained at two or more locations in interface 106, fusion module 108 generates an accurate representation of another vehicle 110 or a portion of another vehicle 110 as a single bounding box relative to the position of vehicle 102. By performing sensor fusion, sensor fusion system 104 reliably and accurately tracks objects that need to be avoided in various driving scenarios. By fusing or combining sensor data, fusion module 108 enables vehicle 102 to accurately track and avoid obstacles within the FOV 114.

[0022] When fusing multiple large sets of candidate object tracks together, fusion module 108 can generate a feasibility matrix as described by Schiffmann et al. in U.S. Patent No. 10,565,468 (hereinafter referred to herein as "Schiffmann"). In Schiffmann, the sensor fusion system assigns identifiers to each candidate object track associated with an object candidate detected using a visual camera, and assigns identifiers to each candidate object track acquired using radar (e.g., detection). A two-dimensional feasibility matrix is ​​created; the first dimension represents the total number of columns, with each assigned radar identifier as one column. The second dimension represents the total number of rows, with each camera identifier assigned to camera data as one row. For each combination of camera candidates and radar candidates represented by the feasibility matrix, a probability is determined.

[0023] These probabilities can be represented as an evidence matrix. The entries of the evidence matrix can be determined based on the error between visual camera and radar object tracking, and each entry indicates the degree of confidence or feasibility corresponding to the same object tracked by the visual camera and maintained by the radar. For each intersection of the columns and rows of the feasibility matrix, the evidence matrix includes a corresponding entry, which can be a value between zero and one hundred percent, or some other equivalent value within that range. This value indicates the probability that a particular radar and visual camera object tracking pair (mapped to that intersection of the feasibility matrix) has tracked the same object.

[0024] EKF or other filters can be applied to probabilities calculated over time. For example, it can be assumed that at least one pair of radar and vision camera detections has a reasonable chance of matching, and if so, EKF can identify the combination of objects to be tracked by the fusion that has the highest probability of occurrence, rather than other combinations with too low a probability (e.g., values ​​that do not meet a threshold). Correctly selecting the pseudo-measurement type (described below) can be crucial for accurate measurement updates using EKF, as an incorrect definition of the pseudo-boundary box can lead to poor estimations of the size, shape, and position of the object being tracked.

[0025] Example Architecture

[0026] Figure 2 An example of an automotive system 200 configured to perform multi-hypothesis fusion of sensor data according to the technology of this disclosure is shown. The automotive system 200 can be integrated into... Figure 1 The vehicle 102 shown and described in this context. For example, vehicle system 200 includes controller 202 and sensor fusion system 104-1, which is an example of sensor fusion system 104. Sensor fusion system 104-1 and controller 202 communicate via link 212. Link 212 can be a wired or wireless link and in some cases includes a communication bus. Controller 202 performs operations based on information received via link 212, such as indications of bounding boxes output from sensor fusion system 104-1 when objects in the FOV are identified according to processed and merged object tracking.

[0027] The controller 202 includes a processor 204-1 and a computer-readable storage medium (CRM) 206-1 (e.g., memory, long-term storage, short-term storage), which stores instructions for the automotive module 208. In addition to the vision camera interface 106-2, the sensor fusion system 104-1 also includes a radar interface 106-1. Any number of other sensor interfaces 106 can be used, including or excluding lidar interfaces or other sensor interfaces 106-n. The sensor fusion system 104-1 may include processing hardware including a processor 204-2 (e.g., a hardware processor, processing unit) and a computer-readable storage medium (CRM) 206-2, which stores instructions associated with the fusion module 108-1. As an example of the fusion module 108, the fusion module 108-1 includes a data type selector submodule 210-1 and a matching submodule 210-2.

[0028] Processors 204-1 and 204-2 may be two separate processing units, a single processing unit (e.g., a microprocessor), or a pair of system-on-a-chip (SoC) or a single SoC of a computing device, controller, or control unit. Processors 204-1 and 204-2 execute computer-executable instructions stored within CRM 206-1 and CRM 206-2. As an example, processor 204-1 may execute vehicle module 208 to perform driving functions of vehicle system 200 (e.g., autonomous lane change maneuvers, semi-autonomous lane keeping features) or other operations. Similarly, processor 204-2 may execute fusion module 108-1 to infer objects in the field of view (FOV) based on sensor data obtained from multiple different sensor interfaces 106 of system 102. Vehicle module 208, when executed at processor 204, may receive indications for one or more objects detected by fusion module 108-1 in response to fusion module 108-1 combining and analyzing sensor data generated at each of the sensor interfaces 106.

[0029] Typically, vehicle system 200 executes vehicle module 208 to perform vehicle functions using output from sensor fusion system 104-1. For example, vehicle module 208 may provide adaptive cruise control and detection of the presence of objects in or near FOV 114 to reduce speed and prevent collisions with the rear of vehicle 110. In such an example, fusion module 108-1 provides sensor data or derivatives thereof (e.g., a feasibility matrix) as output to vehicle module 208. When data obtained from fusion module 108-1 indicates that one or more objects are crossing in front of vehicle 102, vehicle module 208 may provide an alert or trigger specific maneuvers.

[0030] For simplicity, the data type selector submodule 210-1 and the matching submodule 210-2 are described primarily with reference to radar interface 106-1 and vision camera interface 106-2 (without reference to lidar or another sensor interface 106-n). However, it should be understood that the fusion module 108-1 can combine sensor data from more than two different types of sensors and can rely on sensor data output from other types of sensors besides vision camera and radar alone. To further reduce the complexity of the description, the matching submodule 210-2 can operate according to the rule that radar interface 106-1 can detect any vehicle that vision camera interface 106-2 can detect. Additionally, vision camera interface 106-2 can be configured to generate at most one object track for each object of interest in FOV 114; while radar interface 106-1 can be configured to generate several object tracks for each object of interest in FOV 114 (e.g., forty or fifty object tracks in the case of semi-trucks and trailers).

[0031] Radar interface 106-1 can operate independently of vision camera interface 106-2 and independently of lidar / other sensor interface 106-n. Radar interface 106-1 maintains a list of "detections" and corresponding detection times, assumed to primarily be the dispersion center tracking the vehicles it detects. Each detection typically consists of a distance value, a distance rate of change value, and an azimuth value. For each vehicle that is not obstructed within FOV 114 and is relatively close to vehicle 102, there is typically more than one detection.

[0032] Similar to the radar interface 106-1, the vision camera interface 106-2 provides a vision camera-based object tracking list. The vision camera interface 106-2 outputs sensor data (which can be provided in various forms, such as a list of candidate objects being tracked) and estimates for each of the following: object position, velocity, object class, and reference angles (e.g., azimuth to a "centroid" reference point on the object (such as the center of the rear surface of the moving vehicle 110), and other "extended angles" to the near corner of the rear surface of the moving vehicle 110). The vision camera interface 106-2 can estimate azimuth and object classification more accurately than other sensor types; however, it may have limitations in estimating some parameters, such as longitudinal position or range, velocity, etc. The radar interface 106-1 accurately measures object distance and rate of change of distance, but may be less accurate in measuring azimuth (this is where the vision camera excels). The complementary nature of the vision camera and radar contributes to the accuracy when matching data between the sensor interfaces 106 and 106-2.

[0033] According to the technology of this disclosure, the fusion module 108-1 performs multiple hypothesis-based fusions of sensor data, seeking to improve the performance of fusion tracking without sacrificing accuracy by ignoring data. Each hypothesis considered by the fusion module 108-1 corresponds to a different pseudo-measurement type. The fusion tracker 108-1 can automatically determine which pseudo-measurement type is more likely to be accurate for the current situation using a predefined error covariance associated with the radar interface 106-1.

[0034] Without a specific order, the first pseudo-measurement type is a calculation of the visual camera angle and radar distance. Radar distance measurements and visual camera angle measurements at a reference point (e.g., the location of vehicle 102) are used to calculate the corresponding reference point for the pseudo-boundary box. The second pseudo-measurement type is a calculation of the visual camera lateral position and radar distance. This pseudo-measurement results in a reference point for another pseudo-boundary box, this time calculated based on the radar distance and visual lateral position. The third pseudo-measurement type is a calculation based solely on radar information. The pseudo-measurement associated with this calculation corresponds to a reference point for the bounding box calculated solely based on radar information (including radar range).

[0035] The fusion module 108-1 may rely on one of the two combined radar and vision range calculations, or it may ignore vision-based pseudo-measurements and rely only on radar pseudo-measurements. The data type selector submodule 210-1 improves the speed and efficiency of the matching submodule 210-2 by enabling it to avoid evaluating some matches that are unlikely to be associated with the same object.

[0036] The matching submodule 210-2 efficiently calculates the probability that each camera-based object tracker should match a finite number of candidate radar-based object tracks for each camera-based object tracker. The matching submodule 210-2 associates the list of candidate detections generated at the radar interface 106-1 with the list of candidate objects reported at the vision camera interface 106-2. When evaluating radar-based object tracks relative to vision camera-based object tracks, the data type selector submodule 210-1 eliminates some of the guessing work otherwise performed by the matching submodule 210-2. Some combinations of radar-based and vision camera-based object tracks are eliminated rather than evaluated, allowing matching to occur in fewer steps than without the data type selector submodule 210-1. The fusion module 108-1 can generate one or more pseudo-boundary boxes using different pseudo-measurement types. By selecting between three different bounding boxes (view angle-based boxes, view lateral position-based boxes, or radar-based boxes only), the fusion tracker can balance accuracy and speed when drawing, repositioning, or resizing bounding boxes, even in congested traffic or other high-volume conditions.

[0037] Example Scenario

[0038] Figure 3 Showing from Figure 1-1The environment, of which part 300 is shown in more detail. The fusion module 108 reports information relative to a reference point corresponding to the estimated collision point between vehicle 102 and another vehicle 110, given their current tracking. Note that for some fusion trackers (e.g., fusion module 108-1), the reference point associated with each low-level track can be synchronized during fusion. That is, the information reported by one low-level track is converted and transmitted by the fusion module 108-1 relative to a reference point inferred for the corresponding low-level track. A visual angle 302 relative to vehicle 102 and radar range 304 is depicted. From... Figure 1 Zooming in, the offsets in bounding boxes 112-1 and 112-2 are clearly visible; each originates from radar data at radar interface 106-1 or visual camera data at visual camera interface 106-2. Bounding box 112-1 is located in a different position than bounding box 112-2; this offset is caused by technical differences in how the radar and visual camera perceive different parts of the same vehicle 110.

[0039] The fusion module 108-1 can apply additional fusion techniques to fuse bounding boxes 112-1 and 112-2 with respect to a common reference point. Therefore, the fusion module 108-1 seeks reference points for synchronized low-level object tracking to fuse the two data sources. Bounding box 112-1 is generated based on radar-based low-level object tracking reported relative to a reference point that may coincide with the predicted collision point between vehicle 102 and the object represented by bounding box 112-1. Bounding box 112-2 can be drawn relative to another reference point based on low-level vision camera-based object tracking.

[0040] The fusion module 108-1 converts the information reported by the radar and vision camera and uses it to create bounding boxes 112-1 and 112-2 to create fusion information for generating fused bounding boxes, which replace bounding boxes 112-1 and 112-2 at the fusion reference point. Combining the radar and vision camera into the fusion tracker allows for very good azimuth estimation from the vision camera, while also providing accurate distance predictions about the reference point.

[0041] Figure 4-1 and Figure 4-2 Example pseudo-boundary boxes defined under multi-hypothesis fusion of sensor data according to the techniques of this disclosure are shown. These pseudo-boundary boxes enable the fusion module 108-1 to resolve two otherwise offset bounding boxes as being around a common reference point, which improves the accuracy of maneuvering vehicle 102 to avoid collisions with another vehicle 110.

[0042] Figure 4-1 Including with Figure 1The environment 100 is similar to environment 400-1. Vehicle 102 has another vehicle 110 (not shown) in its FOV. The visual angle 402-1 is reported at the visual camera interface 106-2; the radar range 404-1 is reported at the radar interface 106-1. The radar-based bounding box 406-1 is offset to the lower right of the camera-based bounding box 408-1.

[0043] Figure 4-2 Including with Figure 1 Environment 100 is similar to environment 400-2. Vehicle 102 has another vehicle 110 (not shown) in its FOV. Visual angle 402-2 is reported at visual camera interface 106-2; radar range 406-2 is reported at radar interface 106-1. However, this time the radar-based bounding box 406-2 is offset to the upper right of the camera-based bounding box 408-2.

[0044] As indicated above, radar interface 106-1 can provide pseudo-measurements of radar range. Additionally, vision camera interface 106-2 can report pseudo-measurements including visual range, visual angle, lateral position, and longitudinal position. Pseudo-measurements from radar interface 106-1 are reliable most of the time; however, some pseudo-measurements obtained from the vision camera are more reliable than others. For example, in a forward-driving scenario, visual range and longitudinal position are unlikely to be accurate enough for repositioning the bounding box indicating the fusion between radar and vision camera object tracking. However, visual angle and lateral position obtained from vision camera interface 106-2 are reliable for the purpose of evaluating the accuracy of the bounding box. Therefore, when considering the repositioning of the fusion-based bounding box, fusion module 108-1 can consider multiple combinations of pseudo-measurement types, as outlined in Table 1 below:

[0045]

[0046] Table 1 notes that these are just some example combinations; other combinations of pseudo-measurement types may be considered when other sensors or other types of pseudo-measurements are available.

[0047] First go to Figure 4-1When considering combination A from the table above, the data type selector submodule 210-1 uses the radar range 404-1 at the reference point of the object tracking and the visual camera angle 402-1 to calculate the reference point of the pseudo-boundary box 410-1. The data type selector submodule 210-1 selects combination A as the pseudo-measurement type, which has a greater probability of being accurate than each of the other combinations B and C of pseudo-measurement types. Combination A represents a first pseudo-measurement derived from the radar range indicated by one or more object tracks from a plurality of first object tracks and further derived from the visual angle inferred from at least one object track from a set of second object tracks. In Cartesian coordinates, relative to the reference point A(x) of the bounding box 410-1... 伪 ,y 伪 The product of the radar range 404-1 and the cosine of the visual angle 402-1 in the x-direction and the sine of the radar range 404-1 and the visual angle 402-1 in the y-direction are expressed as Equation 1 below:

[0048]

[0049] Where, x 伪 y 伪 This represents the position of a reference point in Cartesian coordinates on the pseudo-boundary box 410-1, derived from radar and vision. R 雷达 This indicates the radar distance to the reference point reported by radar interface 106-1, measured by 404-1, and θ 视觉 It is visual angle 402-1, which is the azimuth angle of the same reference point reported by visual camera interface 106-2.

[0050] Switch to Figure 4-2 When considering combination A from the table above, the data type selector submodule 210-1 uses the radar range and visual camera angle at the reference point of object tracking to calculate the reference point of the pseudo-boundary box 410-1. The data type selector submodule 210-1 selects the reference point A(x) of combination A relative to the bounding box 410-1 in the Cartesian coordinate system. 伪 ,y 伪 According to Equation 1, the reference point A(x) 伪 The product of the radar range 404-2 and the visual angle 402-2 in the x-direction is expressed as the product of the cosines of the radar range 404-2 and the visual angle 402-2 in the y-direction, and therefore x 伪 y 伪 On pseudo-boundary box 410-2. R 雷达 This indicates the radar range to the reference point as reported by radar interface 106-1, measured by 404-2, and θ 视觉 It is a visual angle of 402-2.

[0051] Go to Figure 4-1 When considering combination B from the table above, the data type selector submodule 210-1 uses the radar range and visual camera angle at the reference point of the object tracking to calculate the reference point of the pseudo-boundary box 412-1. The data type selector submodule 210-1 selects combination B as the pseudo-measurement type, which has a greater probability of being accurate than each of the other combinations A and C of the other pseudo-measurement types. Combination B represents a second pseudo-measurement derived from the radar range 404-1 indicated by one or more object tracks from a plurality of first object tracks and further derived from the visual lateral position inferred from at least one object track from a set of second object tracks. The reference point B(x) relative to the bounding box 412-1 in the Cartesian coordinate system is... 伪 ,y 伪 In the x-direction, it is represented as the square root of the difference between the square of the radar range 404-1 and the square of the visual lateral position, and in the y-direction, it is represented as the product of the visual lateral positions, as shown in Equation 2 below:

[0052]

[0053] Among them, R 雷达 This indicates that the radar range 404-2 reported by radar interface 106-2 at the reference point is a pseudo-measurement, and y 视觉 It is the lateral position of the same reference point in the Cartesian coordinate system reported by the vision camera interface 106-2.

[0054] Return to Figure 4-2 When considering combination B from the table above, the data type selector submodule 210-1 uses the radar range and visual lateral position at the reference point of object tracking to calculate the reference point of the pseudo-boundary box 412-2. The data type selector submodule 210-1 selects the reference point B(x) of combination B relative to the bounding box 412-2 in the Cartesian coordinate system. 伪 ,y 伪 According to Equation 2, bounding box 412-2 in the x-direction is represented as the square root of the difference between the square of the radar range 404-2 and the square of the visual lateral position, and bounding box 412-2 in the y-direction is represented as the product of the visual lateral positions. 雷达 This indicates that the radar range reported by radar interface 106-2 is measured by 404-2, and y 视觉 It is the lateral position of the same reference point in the Cartesian coordinate system reported by the vision camera interface 106-2.

[0055] Finally, in this example, return Figure 4-1When considering combination C in the table above, data type selector submodule 210-1 uses radar information including radar range 404-1 at radar interface 106-1, regardless of any pseudo-measurements from vision camera interface 106-2. Data type selector submodule 210-1 selects combination C as the pseudo-measurement type, which is more likely to be accurate than each of the other pseudo-measurement types represented by combinations A and B. Data type selector submodule 210-1 selects a third pseudo-measurement derived from radar information (including radar range 404-1) indicated by one or more object tracks from a plurality of first object tracks and not from a visual angle or visual lateral position inferred from at least one object track from a set of second object tracks. The reference point C(x,y) relative to the bounding box in Cartesian coordinates is represented as having the same position as the bounding box 406-1 reported by radar interface 106-1, regardless of vision camera interface 106-2.

[0056] Figure 4-1 and Figure 4-2 This illustrates how bounding boxes can be generated using different pseudo-measurement types. The appropriate selection of pseudo-measurement types by the data type selector submodule 210-1 can aid in EKF filtering, as otherwise incorrectly defined pseudo-bounding boxes can lead to poor estimates about the object being tracked. Since different pseudo-measurement types are recommended for use in different tracking scenarios, the data type selector submodule 210-1 employs a multi-hypothesis-based approach to make decisions regarding pseudo-measurement types.

[0057] Assume the position error covariance σ of radar interface 106-1 雷达 If known, or if not known, can be derived empirically through experiments and observations of the radar interface 106-1 over time. Assume the radar tracking position is X. 雷达 =[x r ,y r ]^T, and its error covariance σ 雷达 In equation 3:

[0058]

[0059] Note that if only radar range accuracy σ is provided... r and radar azimuth accuracy σ θ As an additional pseudo-measurement value obtained at radar interface 106-1, it is assumed that the radar range accuracy σ r and radar azimuth accuracy σ θ If they are independent, then the position error covariance in the Cartesian coordinate system can be derived, as shown in Equation 4:

[0060]

[0061] To determine the probability of each location hypothesis (e.g., each combination A, B, C), the squared Mahalanobis distance between the reference points of combinations A and B is calculated when compared with the reference points of combination C (i.e., the locations of radar bounding boxes 406-1 and 406-2), as shown in Equation 5.

[0062] d 2 (X H ,X 雷达 )=(X H -X 雷达 ) T σ 雷达 -1 (X H -X 雷达 )

[0063] Equation 5

[0064] X H The reference point location is defined by each combination A, B, or C. Intuitively, a larger Mahalanobis distance indicates a smaller probability of the corresponding spurious measurement type, which can provide a better performance estimate. It is well known that the square of the Mahalanobis distance follows a chi-square distribution with two degrees of freedom, and its cumulative distribution is defined by Equation 6:

[0065] 1-e -x / 2

[0066] Equation 6

[0067] When x is the squared Mahalanobis distance, the probabilities of combinations A and B can be defined as the cumulative distribution of a 2-degree chi-square distribution from positive infinity to its squared Mahalanobis distance, i.e.:

[0068]

[0069] Note that combinations A and B are not independent, therefore the probability of combination C (which is only a radar pseudo-measurement type) can be defined as:

[0070] P C =1-(P) A +P B -P(A,B))

[0071] Equation 8

[0072] The joint probability of combination A and combination B can be defined by Bayes' theorem, as shown in Equation 9:

[0073] P(A,B)=P A *P(B|A)

[0074] Equation 9

[0075] Given combination A, the conditional distribution of the measurement type of combination B is accurate and can be defined as:

[0076]

[0077] The squared Mahalanobis distance, using Equation 10, is defined as follows:

[0078] d 2 (X B ,X A )=(X B -X A ) T σ 雷达 -1 (X B -X A )

[0079] Equation 11

[0080] Therefore, the possibility of a "radar-only" pseudo-measurement type can be further deduced as follows:

[0081]

[0082]

[0083] With the probability value of each hypothesis well defined using Equation 1-12, for a given situation, the fusion module 108-1 can use the most accurate combination of the three combinations A, B, and C of pseudo-measurement types to fuse sensor data from multiple interfaces 106.

[0084] Example Method

[0085] Figure 5 An example method for multi-hypothesis-based fusion of sensors according to the technology of this disclosure is illustrated. Method 500 is shown as a set of operations (or actions) performed in the order or combination of the operations shown or described. Furthermore, any operations may be repeated, combined, or rearranged to provide other methods. In the following discussion sections, reference may be made to the foregoing figures when describing some non-limiting examples of method 500.

[0086] At point 502, multiple first candidate object tracks are determined based on first sensor data obtained from the first set of sensors. Each object track from the multiple first candidate object tracks may be associated with at least a portion of a stationary object or a portion of a moving object (such as, vehicle 110).

[0087] At point 504, a second set of object tracking is determined based on second sensor data obtained from the second set of sensors. In some examples, the first set of sensors includes one or more radar sensors, lidar sensors, or ultrasonic sensors, and the second set of sensors includes one or more vision sensors, including optical cameras or infrared cameras.

[0088] At point 506, a pseudo-measurement type with a greater probability of being accurate than each of the other pseudo-measurement types is selected from a plurality of pseudo-measurement types. Fusion module 108-1 selects a pseudo-measurement type based on a first probability, a second probability, and a third probability, which has a greater probability of being accurate compared to the other two pseudo-measurement types. For example, data type selector submodule 210-2 can determine a first probability of a first pseudo-measurement derived from a radar range indicated by one or more object tracks from a plurality of first object tracks and further derived from a visual angle inferred from at least one object track from a set of second object tracks. Data type selector submodule 210-2 can determine a second probability of a second pseudo-measurement derived from the radar range and further derived from a visual lateral position inferred from at least one object track from a set of second object tracks. Data type selector submodule 210-2 can determine a third probability of a third pseudo-measurement derived from radar information including the radar range and not further derived from a visual angle or visual lateral position. The fusion module 108-1 selects the pseudo-measurement type associated with the larger of the first probability, the second probability, and the third probability.

[0089] At 508, the selected pseudo-measurement type is used to determine the bounding box represented by multiple first object traces and a set of second object traces. In some examples, selecting a pseudo-measurement type from multiple pseudo-measurement types that has a greater probability of being accurate than each of the other pseudo-measurement types includes selecting a pseudo-measurement type based on the history of pseudo-measurement types selected during previous update cycles. For example, a pseudo-measurement type can be kept fixed until a pattern appears. Using the history of selecting a particular pseudo-measurement type, the fusion module 108-1 obtains the confidence level that the selected pseudo-measurement is correct and should be used.

[0090] At 506, the bounding box is output as an indication of the match between one or more object traces from a plurality of first object traces and at least one object trace from a set of second object traces.

[0091] Additional examples

[0092] Additional examples of multi-hypothesis fusion of sensor data are provided in the following sections.

[0093] Example 1. A method comprising: determining a plurality of first object tracks by a sensor fusion system based on first sensor data obtained from a first set of sensors; determining a set of second object tracks by the sensor fusion system based on second sensor data obtained from a second set of sensors; selecting from a plurality of pseudo-measurement types a pseudo-measurement type that is more likely to be accurate than each of the other pseudo-measurement types in the plurality of pseudo-measurement types, the selection being based at least in part on a predefined error covariance associated with the first set of sensors; using the pseudo-measurement type to determine bounding boxes represented by the plurality of first object tracks and the set of second object tracks; and outputting an indication of the bounding boxes as a match between one or more object tracks from the plurality of first object tracks and at least one object track from the set of second object tracks.

[0094] Example 2. Any of the methods in the preceding examples, wherein the first set of sensors is different from the second set of sensors.

[0095] Example 3. The method of the foregoing example, wherein the first set of sensors includes one or more radar sensors, lidar sensors or ultrasonic sensors, and the second set of sensors includes one or more vision sensors, wherein the one or more vision sensors include an optical camera or an infrared camera.

[0096] Example 4. A method of any of the preceding examples, wherein selecting a pseudo-measurement type from the plurality of pseudo-measurement types that has a greater probability of being accurate than each of the other pseudo-measurement types includes: selecting a first pseudo-measurement, the first pseudo-measurement being derived from a radar range indicated by one or more object tracks from the plurality of first object tracks, and further derived from a visual angle inferred from the at least one object track from the set of second object tracks.

[0097] Example 5. The method of Example 4, wherein the reference point (x, y) to the bounding box in the Cartesian coordinate system is represented in the x-direction as the product of the radar range and the cosine of the visual angle, and in the y-direction as the product of the radar range and the sine of the visual angle.

[0098] Example 6. A method of any of the preceding examples, wherein selecting a pseudo-measurement type from the plurality of pseudo-measurement types that has a greater probability of being accurate than each other pseudo-measurement type comprises: selecting a second pseudo-measurement, the second pseudo-measurement being derived from a radar range indicated by one or more object tracks from the plurality of first object tracks, and further derived from a visual lateral position inferred from the at least one object track from the set of second object tracks.

[0099] Example 7. The method of Example 6, wherein the reference point (x, y) to the bounding box in the Cartesian coordinate system is represented in the x-direction as the square root of the difference between the square of the radar range and the square of the visual lateral position, and in the y-direction as the product of the visual lateral positions.

[0100] Example 8. A method of any of the preceding examples, wherein selecting a pseudo-measurement type from the plurality of pseudo-measurement types that has a greater probability of being accurate than each other pseudo-measurement type includes: selecting a third pseudo-measurement, the third pseudo-measurement being derived from radar information including radar range indicated by one or more object tracks from the plurality of first object tracks, and not derived from visual angle or visual lateral position inferred from the at least one object track from the set of second object tracks.

[0101] Example 9. The method of Example 8, wherein the reference point (x,y) to the bounding box in the Cartesian coordinate system is represented as the radar distance in the y-direction.

[0102] Example 10. A method of any of the preceding examples, wherein selecting from the plurality of pseudo-measurement types the pseudo-measurement type having a greater probability of being accurate than each other pseudo-measurement type comprises: determining a first probability of a first pseudo-measurement, the first pseudo-measurement being derived from a radar range indicated by one or more object tracks from the plurality of first object tracks, and further derived from a visual angle inferred from at least one object track from the set of second object tracks; determining a second probability of a second pseudo-measurement, the second pseudo-measurement being derived from the radar range, and further derived from a visual lateral position inferred from at least one object track from the set of second object tracks; determining a third probability of a third pseudo-measurement, the third pseudo-measurement being derived from radar information indicated by the one or more object tracks from the plurality of first object tracks, the radar information including the radar range, the third pseudo-measurement not being further derived from the visual angle or the visual lateral position.

[0103] Example 11. The method of any of the foregoing examples further includes: determining the error covariance associated with the first set of sensors.

[0104] Example 12. A method of any of the preceding examples, wherein selecting a pseudo-measurement type from the plurality of pseudo-measurement types that has a greater probability of being accurate than each other pseudo-measurement type comprises: selecting the pseudo-measurement type based on a history of previous frames of the first sensor data and the second sensor data.

[0105] Example 13. A system comprising: a processor configured to perform a method of any of the preceding examples.

[0106] Example 14. The system of Example 13 further includes a vehicle, the vehicle including the processor.

[0107] Example 15. A system comprising means for performing a method of any of the preceding examples.

[0108] Example 16. A non-transient computer-readable storage medium comprising instructions that, when executed, cause at least one processor of the system to perform a method of any of the preceding examples.

[0109] Conclusion

[0110] While various embodiments of the present disclosure have been described in the foregoing description and illustrated in the accompanying drawings, it should be understood that the present disclosure is not limited thereto, but can be practiced in various ways within the scope of the following claims. It will be apparent from the foregoing description that various modifications can be made without departing from the scope of the present disclosure as defined by the appended claims. Problems associated with combination explosion may also arise in other systems where the number of agents or tasks is large. Therefore, although described as one way to improve vehicle-based matching technology, the techniques described above can be applied to other allocation problems to reduce the total number of assignments between tasks and agents.

[0111] Unless the context explicitly states otherwise, the use of "or" and grammatically related terms indicates an unrestricted, non-exclusive alternative. As used herein, the phrase referring to "at least one" of a list of items means any combination of those items, including a single member. As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

Claims

1. A method for a vehicle, the method comprising: determining, by a sensor fusion system, a plurality of first object tracks from first sensor data obtained from a first set of sensors; determining, by the sensor fusion system, a set of second object tracks from second sensor data obtained from a second set of sensors; selecting, from a plurality of pseudo-measurement types, a pseudo-measurement type that has a greater likelihood of being accurate than each other pseudo-measurement type from the plurality of pseudo-measurement types, the selecting based at least in part on a predefined error covariance associated with the first set of sensors; determining, using the pseudo-measurement type, a bounding box represented by the plurality of first object tracks and the set of second object tracks; and outputting an indication of the bounding box as a match between one or more object tracks from the plurality of first object tracks and at least one object track from the set of second object tracks; wherein selecting, from the plurality of pseudo-measurement types, the pseudo-measurement type that has a greater likelihood of being accurate than each other pseudo-measurement type comprises: determining a first probability of a first pseudo-measurement, the first pseudo-measurement derived from a radar range indicated by the one or more object tracks from the plurality of first object tracks and further derived from a visual angle inferred from the at least one object track from the set of second object tracks; determining a second probability of a second pseudo-measurement, the second pseudo-measurement derived from the radar range and further derived from a visual lateral position inferred from the at least one object track from the set of second object tracks; determining a third probability of a third pseudo-measurement, the third pseudo-measurement derived from radar information indicated by the one or more object tracks from the plurality of first object tracks, the radar information comprising a radar distance, the third pseudo-measurement not further derived from the visual angle or the visual lateral position; and selecting, based on the first probability, the second probability, and the third probability, the pseudo-measurement type that has a greater likelihood of being accurate.

2. The method of claim 1, wherein, the first set of sensors is different from the second set of sensors.

3. The method of claim 1, wherein, the first set of sensors comprises one or more radar sensors, lidar sensors, or ultrasonic sensors and the second set of sensors comprises one or more vision sensors, the one or more vision sensors comprising optical cameras or infrared cameras.

4. The method of claim 1, wherein, selecting, from the plurality of pseudo-measurement types, the pseudo-measurement type that has a greater likelihood of being accurate than each other pseudo-measurement type comprises selecting a first pseudo-measurement, the first pseudo-measurement derived from a radar range indicated by the one or more object tracks from the plurality of first object tracks and further derived from a visual angle inferred from the at least one object track from the set of second object tracks.

5. The method of claim 4, wherein, a reference point (x, y) to the bounding box is expressed in an x-direction as a product between a cosine of the radar range and the visual angle and in a y-direction as a product between a sine of the radar range and the visual angle.

6. The method of claim 1, wherein, selecting, from the plurality of pseudo-measure types, the pseudo-measure type that has a greater likelihood of being accurate than each other pseudo-measure type includes selecting a second pseudo-measure that is derived from radar range indicated by the one or more object tracks from the plurality of first object tracks and further derived from visual lateral position inferred from the at least one object track from the set of second object tracks.

7. The method of claim 6, wherein, a reference point (x, y) to the bounding box is expressed in the x-direction as a square root of a difference between a square of the radar range and a square of the visual lateral position, and in the y-direction as a product of the visual lateral position.

8. The method of claim 1, wherein, selecting, from the plurality of pseudo-measure types, the pseudo-measure type that has a greater likelihood of being accurate than each other pseudo-measure type includes selecting a third pseudo-measure that is derived from radar information including radar range indicated by the one or more object tracks from the plurality of first object tracks and is not derived from visual angle or visual lateral position inferred from the at least one object track from the set of second object tracks.

9. The method of claim 8, wherein, a reference point (x, y) to the bounding box is expressed in the y-direction as the radar distance.

10. The method of claim 1, wherein, further comprising: determining the error covariance associated with the first set of sensors.

11. The method of claim 1, wherein, selecting, from the plurality of pseudo-measure types, the pseudo-measure type that has a greater likelihood of being accurate than each other pseudo-measure type includes selecting the pseudo-measure type based on a history of pseudo-measure types selected based on previous frames of the first sensor data and the second sensor data.

12. A system for a vehicle, the system comprising: a processor configured to: determine a plurality of first object tracks from first sensor data obtained from a first set of sensors; determine a set of second object tracks from second sensor data obtained from a second set of sensors; select, from a plurality of pseudo-measure types, a pseudo-measure type that has a greater likelihood of being accurate than each other pseudo-measure type from the plurality of pseudo-measure types, the selection based at least in part on a predefined error covariance associated with the first set of sensors; determine, using the pseudo-measure type, a bounding box represented by the plurality of first object tracks and the set of second object tracks; and output the bounding box as a match between one or more object tracks from the plurality of first object tracks and at least one object track from the set of second object tracks; wherein the processor is further configured to: select, from the plurality of pseudo-measure types, the pseudo-measure type that has a greater likelihood of being accurate than each other pseudo-measure type by selecting at least a first pseudo-measure that is derived from radar range indicated by the one or more object tracks from the plurality of first object tracks and further derived from visual angle inferred from the at least one object track from the set of second object tracks; or by selecting, from the plurality of pseudo-measure types, the pseudo-measure type that has a greater likelihood of being accurate than each other pseudo-measure type by at least selecting a second pseudo-measure, the second pseudo-measure being derived from radar range indicated by the one or more object tracks from the plurality of first object tracks and further derived from visual lateral position inferred from the at least one object track from the set of second object tracks; or by selecting, from the plurality of pseudo-measure types, the pseudo-measure type that has a greater likelihood of being accurate than each other pseudo-measure type by at least selecting a third pseudo-measure, the third pseudo-measure being derived from radar information indicated by the one or more object tracks from the plurality of first object tracks, the radar information comprising radar range, the third pseudo-measure not being further derived from the visual angle or the visual lateral position.

13. The system of claim 12, wherein, further comprising: a vehicle, the vehicle comprising the processor.

14. The system of claim 12, wherein, the first set of sensors is different from the second set of sensors.

15. The system of claim 12, wherein, the first set of sensors comprises one or more radar sensors, lidar sensors, or ultrasonic sensors, and the second set of sensors comprises one or more visual sensors, the one or more visual sensors comprising optical cameras or infrared cameras.

16. The system of claim 12, wherein, a reference point A (x, y) to the bounding box is expressed in the x-direction as a product between the radar range and the cosine of the visual angle, and in the y-direction as a product between the radar range and the sine of the visual angle.

17. A non-transitory computer-readable storage medium comprising instructions that, when executed, configure at least one processor of a system to: determine a plurality of first object tracks from first sensor data obtained from a first set of sensors; determine a set of second object tracks from second sensor data obtained from a second set of sensors; select, from a plurality of pseudo-measure types, a pseudo-measure type, the selected pseudo-measure type having a greater likelihood of being accurate than each other pseudo-measure type from the plurality of pseudo-measure types, the selection being based at least in part on a predefined error covariance associated with the first set of sensors; determine, using the pseudo-measure type, a bounding box represented by the plurality of first object tracks and the set of second object tracks; and output an indication of the bounding box as a match between one or more object tracks from the plurality of first object tracks and at least one object track from the set of second object tracks; wherein the processor is further configured to: by selecting, from the plurality of pseudo-measure types, the pseudo-measure type that has a greater likelihood of being accurate than each other pseudo-measure type by at least selecting a first pseudo-measure, the first pseudo-measure being derived from radar range indicated by the one or more object tracks from the plurality of first object tracks and further derived from visual angle inferred from the at least one object track from the set of second object tracks; or by selecting, from the plurality of pseudo-measure types, the pseudo-measure type that has a greater likelihood of being accurate than each other pseudo-measure type by at least selecting a third pseudo-measure, the third pseudo-measure being derived from radar information indicated by the one or more object tracks from the plurality of first object tracks, the radar information comprising radar range, the third pseudo-measure not being further derived from the visual angle or the visual lateral position. by selecting at least a second pseudo-measurement from the plurality of pseudo-measurement types, the second pseudo-measurement derived from radar range indicated by the one or more object tracks from the plurality of first object tracks and further derived from visual lateral position inferred from the at least one object track from the set of second object tracks, the second pseudo-measurement selected from the plurality of pseudo-measurement types having a greater likelihood of being accurate than each other pseudo-measurement type; or by selecting at least a third pseudo-measurement from the plurality of pseudo-measurement types, the third pseudo-measurement derived from radar information indicated by the one or more object tracks from the plurality of first object tracks, the radar information comprising radar range, the third pseudo-measurement not further derived from the visual angle or the visual lateral position, the third pseudo-measurement selected from the plurality of pseudo-measurement types having a greater likelihood of being accurate than each other pseudo-measurement type.

Images