Method and system for sensor fusion

Abstract

This document describes kurtosis-based pruning for a sensor fusion system. The kurtosis-based pruning minimizes the total amount of comparisons performed when fusing large data sets together. Multiple candidate radar tracks can be aligned with one of multiple candidate visual tracks. For each candidate visual track, a weight or other match evidence is assigned to each candidate radar track. The inverse of the match error between each candidate visual and each candidate radar track contributes to this evidence, which can be normalized to generate, for each candidate visual track, a distribution associated with all candidate radar tracks. The kurtosis or shape of the distribution is computed. Based on the kurtosis value, some candidate radar tracks are selected for matching and others remaining candidate radar tracks are pruned. The kurtosis helps determine how many candidates to keep and how many to prune. In this way, the kurtosis-based pruning can prevent combinatorial explosion due to large-scale matching.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 115,142, filed November 18, 2020, pursuant to 35U.SC119(e), the disclosure of which is incorporated herein by reference in its entirety. Background Technology

[0003] In some vehicles, sensor fusion systems, or so-called “fusion trackers,” combine information from multiple sensors (e.g., vehicle speed, yaw rate, position) to support autonomous or semi-autonomous control. The combined sensor data can be used to estimate objects of interest present in the field of view (FOV), or so-called instrument FOV. Position, velocity, trail, size, class, and other parameters can be inferred for each tracked object; the accuracy of these inferences improves when different categories of sensors are used. Correlating inputs from multiple sensors for object tracking purposes can be computationally complex, often requiring sophisticated and expensive processing hardware to handle the potential computational explosion. To improve throughput and processing speed, some fusion trackers specialize in or track only specific types of objects (e.g., vehicle objects); these specialized fusion trackers may intentionally discard candidate trails for certain categories of objects (e.g., pedestrians). However, for safety and precise control, it is desirable for sensor fusion systems to track multiple categories of objects simultaneously (e.g., vehicles and pedestrians) without latency and the risk of combination explosion. Summary of the Invention

[0004] This document describes kurtosis-based pruning for sensor fusion systems. In one example, a method includes: determining a plurality of first candidate object traces by the sensor fusion system based on first sensor data obtained from a first set of sensors; determining a set of second object traces by the sensor fusion system based on second sensor data obtained from a second set of sensors; and applying corresponding weights to a specific object trace from that set of second object traces for each of the plurality of first candidate object traces. The method further includes: determining a distribution of corresponding weights applied to the specific object trace by the sensor fusion system based on the corresponding weights applied to the specific object trace for each of the plurality of first candidate object traces; determining the kurtosis of the distribution of the corresponding weights applied to the specific object trace by the sensor fusion system; and the method may further include: pruning at least one candidate object trace from the plurality of first candidate object traces based on the kurtosis by the sensor fusion system. The method further includes: even after at least one object trace has been pruned, matching the specific object trace with one or more remaining candidate object traces from the plurality of first candidate object traces by the sensor fusion system.

[0005] In one example, the system includes a processor configured to perform this method and other methods. In another example, a system including means for performing this method and other methods is described. In addition to describing systems configured to perform the methods outlined above and other methods set forth herein, this document also describes a computer-readable storage medium including instructions that, when executed, configure a processor to perform the methods outlined above and other methods set forth herein.

[0006] This invention presents a simplified concept of kurtosis-based pruning for sensor fusion systems, which will be further described below in the detailed embodiments and accompanying drawings. This invention is not intended to identify essential features of the claimed subject matter, nor is it intended to define the scope of the claimed subject matter. That is, one problem addressed by the described technique is the problem of combinatorial explosion, which can occur in other assignment scenarios where the number of agents and / or tasks increases exponentially. Therefore, although primarily described in the context of improving fusion tracker matching algorithms, kurtosis-based pruning can also be applied to other applications where it is desirable to prevent combinatorial explosion to reduce possible matching assignments. Attached Figure Description

[0007] This document describes in detail one or more aspects of kurtosis-based pruning for sensor fusion systems with reference to the following figures. The same numbers are generally used throughout the figures to refer to similar features and components.

[0008] Figure 1 An example environment of the technology according to this disclosure is shown, in which the system is configured to perform kurtosis-based pruning for a sensor fusion system.

[0009] Figure 2 An example of an automotive system configured to perform kurtosis-based pruning for a sensor fusion system, according to the technology of this disclosure, is shown.

[0010] Figures 3-1 to 3-3 The details of an example of kurtosis-based trimming based on the technology described in this disclosure are shown.

[0011] Figure 4 An example method for kurtosis-based pruning of a sensor fusion system based on the techniques of this disclosure is shown. Detailed Implementation

[0012] Overview

[0013] As already mentioned, fusion trackers face the challenge of rapidly and accurately determining high-level matches between low-level candidate object tracks from multiple sensors (e.g., radar, ultrasound, infrared cameras, optical cameras or "vision" cameras, LiDAR). Each candidate high-level fusion-based object track corresponds to a single object, multiple objects, or a portion of a single object identifiable within the fusion tracker's field of view (FOV). In crowded FOVs (e.g., when many objects are present, including vehicles and pedestrian traffic, or when the FOV covers a large space with a 360-degree field of view), fusion trackers may perform poorly due to combinatorial explosions that can occur as they attempt to simultaneously match and track numerous low-level candidate object tracks moving in a large volume space (e.g., outdoors). To improve the combinatorial explosions during matching, kurtosis-based pruning is described.

[0014] Kurtosis-based pruning aims to reduce the number of comparisons performed when fusing multiple sets of sensor data (e.g., two sets of object tracks). Consider an example where multiple radar-based candidate object tracks may be aligned with one of multiple camera-based object tracks. For each camera-based object track, a weight or other matching evidence is assigned to each possible radar-based candidate object track. This evidence is derived from the matching errors that exist between each camera-based object track and each possible radar-based candidate object track. The inverse of these errors represents evidence that can be normalized to generate an evidence distribution associated with all radar-based candidate object tracks for each camera-based object track.

[0015] The kurtosis, or shape, of this distribution was calculated. Based on the kurtosis, radar-based candidate object traces were selected for matching, and the remaining radar-based candidate object traces were pruned. Kurtosis was used to determine the number of N object traces to retain, where N represents the highest-weighted radar-based candidate object trace, which is the candidate for matching a specific camera-based object trace. The remaining object traces that were not retained could be discarded or pruned as candidates. By eliminating cross-comparisons of candidate traces that might occur during matching between two or more different types of sensor data, kurtosis-based pruning prevents combinatorial explosion that other sensor fusion systems cannot prevent.

[0016] Example Environment

[0017] Figure 1 An example environment according to the technology of this disclosure is shown, in which system 102 is configured to perform kurtosis-based pruning for a sensor fusion system. In the depicted environment 100, sensor fusion system 104 is mounted to or integrated within vehicle 102. Although shown as a car, vehicle 102 may represent other types of vehicles and machinery (e.g., motorcycles, buses, tractors, semi-trailer trucks, watercraft, aircraft, or other heavy equipment) that can be used for various purposes, whether manned or unmanned. Vehicle 102 may travel on road 118, which may be arranged with street signs 112, vegetation 116, or other stationary objects (not shown) including buildings and parked vehicles. Moving objects (e.g., pedestrians 120 and moving vehicles 110) may also be in or near road 118. Figure 1 In the process, vehicle 102 travels along road 118, and the sensor fusion system 104 of vehicle 102 has a field of view (FOV) covering road 118 except for street signs 112, vegetation 116, pedestrians 114 and moving vehicles 110.

[0018] Sensor fusion system 104 can track objects in a field of view (FOV) based on sensor data obtained from multiple sensors of vehicle 102. Matching objects across multiple different sensors enables sensor fusion system 104 to reliably and accurately track objects that vehicle 102 may need to avoid while navigating road 118.

[0019] Typically, the manufacturer can mount the sensor fusion system 104 to any moving platform that can travel on road 118. The sensor fusion system 104 can project its field of view (FOV) from any external surface of the vehicle 102. For example, the vehicle manufacturer can integrate at least a portion of the sensor fusion system 104 into a side mirror, bumper, roof, or any other internal or external location (where the FOV includes road 118 and moving or stationary objects near road 118). The manufacturer can design the location of the sensor fusion system 104 to provide a specific FOV that adequately covers road 118 on which the vehicle 102 may travel. In the depicted environment 100, a portion of the sensor fusion system 104 is mounted near the rear quarter of the vehicle 102.

[0020] The sensor fusion system 104 includes a fusion module 108 and one or more sensor interfaces 106-1 to 106-n (collectively referred to as "sensor interfaces 106"). The sensor interfaces 106-1 to 106-n include a lidar interface 106-1, a camera interface 106-2, a radar interface 106-3, and one or more other sensor interfaces 106-n. Each of the sensor interfaces 106 provides a specific type of sensor data to the sensor fusion system 104. For example, the lidar interface 106-1 generates lidar data generated by one or more lidar sensors, and the radar interface 106-3 generates radar data generated by a set of radar sensors communicating with the radar interface 106-3.

[0021] By fusing or combining sensor data, the fusion module 108 enables the vehicle 102 to accurately track and avoid obstacles within the field of view (FOV). For example, although not in Figure 1 While precisely illustrated, the fusion module 108 executes on a processor or other hardware, configuring the sensor fusion system 104 to combine different types of sensor data obtained from sensor interface 106 into object tracks or other available forms for tracking objects in the FOV. The fusion module 108 determines multiple candidate object tracks based on first sensor data (e.g., obtained from radar interface 106-3), and identifies a set of candidate object tracks based on second sensor data (e.g., obtained from camera interface 106-2). Each candidate object track inferred from the first and second sensor data is associated with an object in the FOV. For example, any of objects 110, 112, 114, and 116 may be associated with a candidate object track indicated at sensor interface 106. The fusion module 108 identifies candidate object tracks associated with the same object and fuses the information derived from the first and second sets of sensor data to produce an accurate representation of the object or a portion of an object identified in the FOV.

[0022] When fusing multiple large sets of candidate object tracks together, fusion module 108 can generate a feasibility matrix as described by Schiffmann et al. in U.S. Patent No. 10,565,468 (hereinafter referred to as "Schiffmann"), the entire contents of which are incorporated herein by reference. In Schiffmann, the sensor fusion system assigns identifiers to each candidate object track associated with an object candidate detected using a camera, and assigns identifiers to each candidate object track acquired using radar (e.g., detection). For example, a two-dimensional feasibility matrix is ​​created. The first dimension represents the total number of columns, with each assigned radar identifier as one column. The second dimension represents the total number of rows, with each camera identifier assigned to camera data as one row. Probabilities are determined for combinations of camera and radar candidates represented by the feasibility matrix.

[0023] These probabilities can represent an evidence matrix. The entries of the evidence matrix are determined based on the error between the camera and radar candidate object tracks, and each entry indicates the degree of confidence or feasibility of associating a candidate object track detected by the camera with a candidate object candidate track that is the same as the one detected by the radar. For each intersection of the columns and rows of the feasibility matrix, the evidence matrix includes a corresponding entry, which can be a value between zero and one hundred percent or some other equivalent value. This value indicates the probability that the radar identifier and camera identifier represent (mapped to that intersection of the feasibility matrix) tracking the same object. Additional filtering can be applied to the probabilities calculated over time. For example, it can be assumed that at least one combination of candidate radar and camera detections has a reasonable chance of matching, and if so, the combination of candidates with the highest probability of occurrence can be relied upon, rather than other combinations with excessively low probabilities (e.g., values ​​that do not meet a threshold).

[0024] Return to Figure 1In environment 100, fusion module 108 joins datasets with the help of kurtosis-based pruning, which prevents fusion module 108 from experiencing combinatorial explosion during the process due to large-scale matching. For example, fusion module 108 is at risk of combinatorial explosion when camera data and lidar data, including detailed observations of FOV, are dense (e.g., high-resolution). Kurtosis-based pruning minimizes the total number of comparisons ultimately performed by “normalizing” (e.g., zeroing) some elements of the feasibility matrix to improve other operations of the data-dependent vehicle 102. For example, from the feasibility matrix that has been modified by kurtosis pruning, fusion module 108 can quickly identify specific combinations of camera-based candidate object tracks and one or more lidar-based candidate object tracks that have the highest probability of being associated with the same object. After identifying the camera-based candidate object tracks and one or more lidar-based candidate object tracks, location, size, object category, and orientation can be determined. This information, along with other information, can help vehicle 102 perform autonomous or semi-autonomous driving functions.

[0025] Figure 2 An example of an automotive system 200 configured to perform kurtosis-based pruning for a sensor fusion system, according to the technology of this disclosure, is shown. The automotive system 200 may be integrated into a vehicle 102. For example, the automotive system 200 includes a controller 202 and a sensor fusion system 104-1. The sensor fusion system 104-1 is an example of the sensor fusion system 104 and may be integrated into an automotive environment or other vehicle environment. The sensor fusion system 104 and the controller 202 communicate via a link 212. The link 212 may be a wired or wireless link and, in some cases, includes a communication bus. The controller 202 performs operations based on information received via the link 212, such as data output from the sensor fusion system 104 when objects in the field of view are identified by processing and merging object traces.

[0026] The controller 202 includes a processor 204-1 and a computer-readable storage medium (CRM) 206-1 (e.g., memory, long-term storage, short-term storage) that stores instructions for the vehicle module 208. In addition to the camera interface 106-2, the sensor fusion system 104-1 also includes a radar interface 106-3. Any number of other sensor interfaces, including sensor interface 106, can also be used similarly. The sensor fusion system 104-1 may include processing hardware including the processor 204-2 and a computer-readable storage medium (CRM) 206-2 that stores instructions associated with the fusion module 108-1. As an example of fusion module 108, the fusion module 108-1 includes a trimming submodule 210-1 and a matching submodule 210-2.

[0027] Processors 204-1 and 204-2 may be two separate microprocessors, a single microprocessor, a pair of systems-on-a-chip (SoCs) of computing devices, controllers, or control units, or a single SoC. Processors 204-1 and 204-2 execute computer-executable instructions stored within CRM 206-1 and CRM 206-2. As an example, processor 204-1 may execute vehicle module 208 to perform driving functions or other operations of vehicle system 200. Similarly, processor 204-2 may execute fusion module 108-1 to infer objects in the field of view (FOV) based on sensor data obtained from multiple different sensor interfaces 106 of system 102. When executed at processor 204-1, vehicle module 208 may receive indications of one or more objects detected by fusion module 108-1 in response to fusion module 108-1 combining and analyzing sensor data generated at each of the sensor interfaces 106 (e.g., camera interface 106-2 and radar interface 106-3).

[0028] Typically, vehicle system 200 executes vehicle module 208 to perform functions. For example, vehicle module 208 may provide adaptive cruise control and monitoring for the presence of objects in or near road 118. In such an example, fusion module 108-1 provides sensor data or derivatives thereof (e.g., a feasibility matrix) to vehicle module 208. When data obtained from fusion module 108-1 indicates that one or more objects are crossing in front of vehicle 102, vehicle module 208 may provide an alert.

[0029] For simplicity, the following description of the matching submodule 210-2 and trimming submodule 210-1 of the fusion module 108-1 primarily refers to camera interface 106-2 and radar interface 106-3 (without referring to other sensor interfaces 106). However, it should be understood that the fusion module 108-1 can combine sensor data from more than two different types of sensors and can rely on sensor data output from other types of sensors besides just cameras and radar. To further reduce the complexity of the description, the matching submodule 210-2 can operate according to the rules that radar interface 106-3 can detect any vehicle that camera interface 106-2 can detect. Furthermore, camera interface 106-2 is configured to generate at most one object track for each object of interest in the FOV; radar interface 106-3 is configured to generate several object tracks for each object of interest in the FOV (e.g., forty or fifty object tracks in the case of semi-trucks and trailers).

[0030] In the following example, a "matching pattern" refers to a specific way of assigning one of M different radar-based candidate object tracks to one of N different camera-based object tracks. It is anticipated that some of the M radar-based candidate object tracks may not match any of the N camera-based object tracks because they correspond to objects detected by radar but not by vision. The term "feasible match" refers to a specific matching pattern considered a legitimate candidate pair of candidate object tracks for matching purposes, based on considerations of possible errors in radar and camera systems. A "feasible matching pattern" represents a matching pattern consisting of N different feasible matching candidates. A feasible matching pattern can then be considered "infeasible" (e.g., due to object occlusion, due to sensor occlusion, and other problems with the matching pattern). If only a single match is infeasible, then the matching pattern is infeasible even if all other N matches in the matching pattern are feasible. A single infeasible match renders the entire matching pattern infeasible.

[0031] Camera interface 106-2 can operate independently of radar interface 106-3 and can operate independently of other sensor interfaces 106. Camera interface 106-2 outputs sensor data (which can be provided in various forms, such as a list of candidate objects being tracked) and estimates for each of the following: object position, velocity, object class, and reference angles (e.g., azimuth angle to a “centroid” reference point on the object (such as the center of the rear surface of the moving vehicle 110), and other “extended angles” to the near corner of the rear surface of the moving vehicle 110).

[0032] Radar interface 106-3 maintains a list of "detections" and their corresponding detection times. Each detection typically consists of a distance value, a distance rate of change value, and an azimuth value. For each vehicle that is not obstructed within the FOV and is relatively close to vehicle 102, there is usually more than one detection. Similar to camera interface 106-2, radar interface 106-3 provides a list of radar-based candidate object tracks, assuming that the radar-based candidate object tracks are primarily tracking the scattering center on the detected vehicle.

[0033] Camera interface 106-2 can estimate azimuth and object classification more accurately than other sensor types; however, it may have limitations in estimating some parameters, such as object position and velocity. Radar interface 106-3 can accurately measure object distance and rate of change of distance, but may be less accurate in measuring azimuth. The complementary nature of the camera and radar contributes to the accuracy of data matching between sensor interfaces 106-2 and 106-3 in fusion module 108-1.

[0034] Matching submodule 210-2 associates the list of candidate detections generated at radar interface 106-3 with the list of candidate objects reported at camera interface 106-2. Matching submodule 210-2 efficiently calculates for each camera-based object track the probability that the camera-based object track should match each of a finite number of candidate radar-based object tracks.

[0035] The pruning submodule 210-1 improves the speed and efficiency of the matching submodule 210-2 by allowing it to avoid evaluating some matches that are unlikely to be associated with the same object. In other words, the pruning submodule 210-1 eliminates some of the "guessing work" by eliminating some combinations of radar-based candidate and camera-based object traces that the matching submodule 210-2 ultimately attempts to identify. This allows the correct association between the radar interface 106-3 and the camera interface 106-2 to be identified in fewer steps than would be possible without the pruning submodule 210-1.

[0036] During kurtosis-based pruning in environment 100, pruning submodule 210-1 applies appropriate weights or evidence to a specific camera-based object track from a set of camera-based object tracks for each of multiple lidar-based or radar-based object tracks. In an example where multiple lidar-based or radar-based candidate object tracks may align with a single (i.e., specific) camera-based object track, normalized weights or other matching evidence are assigned to each possible lidar-based or radar-based candidate object track for each camera-based object track. The error between each camera-based object track and its corresponding lidar-based or radar-based candidate track is calculated. The inverse of these matching errors between each camera-based object track and each possible lidar-based or radar-based candidate object track is calculated as evidence. The inverse of the error used as evidence can be normalized to generate an evidence distribution associated with all lidar-based or radar-based candidate object tracks for each camera-based object track.

[0037] Based on the applied weights, the pruning submodule 210-1 determines the distribution of the corresponding weights applied to a specific camera-based object track. The "kurtosis" is calculated based on the distribution of evidence values ​​for all candidates considered for each camera-based object track. The kurtosis of the evidence distribution represents a measure of the shape of the distribution. Based on the shape, some candidates with more evidence are selected for matching, and the remaining candidates with less evidence are discarded (e.g., zeroed out). The pruning submodule 210-1 can prune at least one LiDAR-based or radar-based candidate object track based on the corresponding weights applied to a matching pattern that includes at least one object track. In some examples, for instance, if the number of candidates to be retained is equal to the total number of candidates based on kurtosis, the pruning submodule 210-1 avoids pruning any candidate object tracks. Note that it is feasible for LiDAR-based or radar-based candidate object tracks to be applied to more than one camera-based object track. Therefore, pruning a LiDAR-based or radar-based candidate object track for one camera-based object track does not necessarily mean that the LiDAR-based or radar-based candidate object track is also pruned for other camera-based object tracks. Pruning involves discarding LiDAR-based or radar-based object traces as candidate object traces only for this specific camera-based object trace, without affecting other camera-based object traces. This pruning process ensures a reduction in the total number of candidate matching patterns involving LiDAR-based or radar-based candidate object traces for each camera-based object trace, thus reducing combinatorial complexity. After pruning the candidate object traces, the matching submodule 210-2 matches the camera-based object traces with one or more remaining LiDAR-based or radar-based candidate object traces after pruning at least one object trace.

[0038] Example Architecture

[0039] Figures 3-1 to 3-3 Details of an example of kurtosis-based pruning according to the technology of this disclosure are shown. More specifically, the feasibility matrix 302-1 can be generated by the matching submodule 210-2, as... Figure 3-1 As shown. However, before evaluating the feasibility matrix 302-1 for potential targets or obstacles in the FOV, the pruning submodule 210-1 refines the feasibility matrix 302-1 and generates a feasibility matrix 302-3 at its location, as shown. Figure 3-3As shown. Of particular note is that feasibility matrix 302-1 includes fifteen different feasibility matches (e.g., each number represents a feasibility match between radar-based object tracks and camera-based object tracks). After kurtosis pruning of feasibility matrix 302-1, pruning submodule 210-1 generates feasibility matrix 302-3, which has only eight feasibility matches. Matching submodule 210-2 processes feasibility matrix 302-3 less than it would have processed feasibility matrix 302-1, because fewer feasibility matches are tested after kurtosis pruning of some (e.g., seven percent or more than forty percent) of the feasible matches.

[0040] Feasibility matrix 302-1 includes columns V1 to V6 and rows R1 to R10. In this example, matching submodule 210-2 obtains radar data indicating ten different radar-based candidate object tracks via radar interface 106-3. Additional camera data indicating six different camera-based object tracks is obtained via camera interface 106-2. Feasibility matrix 302-1 includes an estimated mapping between radar-based candidate object tracks R1 to R10 and camera-based object tracks V1 to V6. Matching submodule 210-2 applies various gatings to the radar and camera data to obtain the example feasibility matrix 302-1.

[0041] A weighting is applied to the feasibility matrix 302-1. The weighting is based on a unique matching error, quantized for each feasible match. For example, the pruning submodule 210-1 calculates the matching error for each feasible match. The matching error between camera-based object tracks and radar-based candidate object tracks can be calculated according to Equation 1:

[0042] Matching error = k_matching quality_azimuth_weight x az_error_factor x delta_az+

[0043] k_matching quality_distance_weight x rng_error factor x delta_rng+

[0044] k_matching quality_VEL_weight x delta_speed

[0045] Equation 1

[0046] The sum of "matching errors" in Equation 1 represents the weighted sum of the squared differences of the corresponding azimuth (delta_az), corresponding distance (delta_rng), and corresponding velocity (delta_speed) between the camera-based object tracks and radar-based candidate object tracks for feasible matches. Applying the equation to each feasible match produces an error value, thus forming the basis for evidence matrix 302-2, as follows: Figure 1 As shown. To obtain the evidence matrix 302-2, the inverse of each non-zero element of the error matrix is ​​determined. The inverse of each non-zero element of the error matrix can be used to determine the elements in the evidence matrix.

[0047] These evidence values ​​in evidence matrix 302-2 can be normalized for all feasible matches of a specific camera-based object trace. The smaller the matching error, the higher the evidence value. For example, feasible match R1V2 has a lower matching error relative to other feasible matches in evidence matrix 302-2, while feasible match R5V3 has a higher matching error relative to other feasible matches.

[0048] like Figure 3-2 As shown by curve 306, the pruning submodule 210-1 determines the kurtosis value 304 based on the evidence distribution indicated by the evidence matrix 302-2. By definition, kurtosis is a measure of the "tail" of a distribution. It is the fourth standardized moment, defined in Equation 2 as:

[0049]

[0050] As shown in curve 306, the kurtosis of the normal distribution is 3. This value decreases as the distribution becomes flatter than a normal distribution. As the distribution reaches higher peak values ​​than a normal distribution, the value increases beyond 3 and becomes even higher.

[0051] Kurtosis is calculated for each evidence distribution. That is, for each column in evidence matrix 302-2, kurtosis value 304 is calculated. Based on evidence matrix 302-2 with normalized evidence values, each camera-based object track V1 to V6 has a corresponding evidence distribution attributable to radar-based object tracks R1 to R10. The kurtosis values ​​304 of the distributions of camera-based object tracks V1, V2, and V3 are equal. This indicates that the evidence distributions of these three candidate object tracks are similar. According to evidence matrix 302-2, for all corresponding radar-based candidate object tracks, the values ​​of these camera-based object tracks V1, V2, and V3 are mostly zero, with only one radar-based candidate object track having a very high evidence value of approximately 1.0. This means that for each of the camera-based object tracks V1, V2, and V3, only one radar-based candidate object track contains almost all the evidence, while the remaining radar-based candidate object tracks (with almost no evidence) remain zero. Their distributions have high peak values ​​or are around 0.0. For camera-based object traces V4 and V6, only two radar-based candidate object traces contain almost all the evidence; therefore, the peak value of each of their distributions is slightly smaller than the peak values ​​of camera-based object traces V1, V2, and V3. Camera-based object trace V5 also has two candidates with high evidence values ​​and therefore contain "all the evidence," while the remaining radar-based candidate object traces are around zero. However, as shown by the calculated kurtosis value of 304, the peak value of camera-based object trace V5 is slightly smaller than the peak values ​​of candidate object traces V4 and V6.

[0052] The pruning submodule 210-1 selects feasible matches to be pruned based on the kurtosis value 304. Based on the kurtosis value 304 of each camera-based object track V1 to V6, the pruning submodule 210-1 selects the N radar-based candidate object tracks with the highest evidence and discards the remaining radar-based candidate object tracks from the consideration made with that camera-based object track. The value of N depends on the kurtosis value 304 and can be tuned accordingly. Typically, N will decrease linearly as the kurtosis value 304 increases, and vice versa. The algorithm executed by the pruning module 210-1 when such settings are achieved is provided below.

[0053] In all loops of camera-based object traces V1 to V6: The pruning submodule 210-1 checks if the kurtosis value is greater than or equal to 8.0, and if true, sets N to one. Otherwise, the pruning submodule 210-1 checks if the kurtosis value is greater than or equal to 6.0 and less than 8.0, and if true, sets N to two. Otherwise, the pruning submodule 210-1 checks if the kurtosis value is greater than or equal to 4.0 and less than 6.0, and if true, sets N to three. Finally, the pruning submodule 210-1 infers that the kurtosis value is less than 4.0 and sets N to 4. The pruning submodule 210-1 compares the kurtosis value of each camera-based object trace, setting N to one, two, three, or four each time.

[0054] The pruning submodule 210-1 discards selected feasible matches from the feasibility storage matrix by inserting empty data, zero, or other very low values ​​at the selected feasible match positions, and assigns the highest evidence value (e.g., equal to 1) to the unselected feasible matches. As shown in Figure 3, when comparing feasibility matrix 302-1 and feasibility matrix 302-3, the top N candidates with the highest evidence values ​​are retained, and the rest are discarded as candidates.

[0055] In this example, after this pruning, the number of candidate feasible patterns was reduced from 144 to only eight. Feasible matches with the highest evidence value were retained, and only those with low evidence values ​​were discarded, the number of which was based on kurtosis. Reducing the number of feasible candidate matches has a significant impact on the computational performance of the matching submodule 210-2, especially when the feasibility matrix 302-1 is not sparse and when the number of camera-based object traces is large. This reduction helps to keep the number of feasible matches within a more manageable threshold limit.

[0056] Example Method

[0057] Figure 4 An example method for kurtosis-based pruning of a sensor fusion system according to the technology of this disclosure is illustrated. Method 400 is shown as a set of operations (or actions) performed in the order or combination of the operations shown or described. Furthermore, any operations may be repeated, combined, or rearranged to provide other methods. In the following discussion sections, reference may be made to the foregoing figures when describing some non-limiting examples of method 400.

[0058] At 402, a plurality of first candidate object traces are determined based on first sensor data obtained from a first set of sensors. Each object trace from the plurality of first candidate object traces may be associated with at least a portion of a stationary object or a portion of a moving object. At 404, a set of second object traces is determined based on second sensor data obtained from a second set of sensors. In some examples, the first set of sensors includes one or more radar sensors, lidar sensors, or ultrasonic sensors, and the second set of sensors includes one or more vision sensors, including an optical camera or an infrared camera. At 406, for each first object trace from the plurality of first candidate object traces, a corresponding weight is applied to a specific object trace from that set of second object traces.

[0059] At 408, the distribution of the weights applied to a particular object trace is determined based on the weights applied to each of the first object traces from a plurality of first object traces. This may include determining the matching error between one or more candidate object traces from the plurality of first object traces and one or more candidate object traces from the set of second object traces. The weights applied to a particular object trace can be determined based on the matching error. For example, the matching error can be determined as an error matrix (or other suitable structure) containing one or more probabilities indicating the likelihood of a false detection (e.g., a false positive or a false negative), and the weights to be applied to the particular object trace are subsequently determined from the inverse of the error matrix.

[0060] At 410, the kurtosis of the distribution of the corresponding weights applied to a specific object track is determined. At 412, at least one object track can be pruned from multiple first object tracks based on the kurtosis. In this scheme, it is not always necessary to discard at least one object track. For example, for a particular camera-based object track, there may only be one candidate radar-based object track, and this single candidate radar-based object track will not be discarded because such an operation would prevent the camera-based object track from having the option to pair with the candidate radar-based object track. Kurtosis is used to determine the number of N object tracks to retain, where N represents the highest-weighted radar-based candidate object track, which is the candidate used to match a particular camera-based object track. The value of N is greater than or equal to one. Figure 3-3In the example shown above, feasibility matrices 302-1 and 302-3 each have only one candidate R2 for visual trace V1. After kurtosis-based pruning of feasibility matrix 302-1, R2 is retained. For visual trace V5, there are two candidate radar-based object traces, and after kurtosis-based pruning, both candidate radar-based object traces are retained because the kurtosis value indicates that retaining these two candidate radar-based object traces (if no more candidate radar-based object traces exist) is acceptable.

[0061] The pruning at 412 may include discarding the at least one candidate object trace from a plurality of first candidate object traces in response to determining that the corresponding weight applied to the specific object trace for at least one candidate object trace is less than the corresponding weight applied to the specific object trace for one or more remaining candidate object traces. For example, based on kurtosis, the number of candidate object traces to be retained from the plurality of first candidate object traces is determined. The plurality of first candidate object traces may then be divided into two subsets, the two subsets comprising a first subset of candidate object traces to be retained and a second subset of candidate object traces to be discarded. The first subset of candidate object traces to be retained may include the N highest-weighted candidate object traces from the plurality of first candidate object traces applied to the specific object trace, where N is equal to the number of candidate object traces to be retained. Pruning at 412 is accomplished by discarding at least one candidate object trace from the plurality of first candidate object traces.

[0062] Although at least one candidate object trace may be discarded from a plurality of first candidate object traces, it may be reused in matching with different candidate object traces from the set of second object traces. Even if a candidate object trace might be pruned, it may be retained for subsequent matching. Discarding at least one candidate object trace from a plurality of first candidate object traces may include retaining the at least one candidate object trace for matching with different candidate object traces from the set of second object traces. Returning to the example above of creating two subsets, pruning at 412 may include avoiding discarding a first subset of candidate object traces from a plurality of first candidate object traces, so that after at least one candidate object trace has been pruned, a particular object trace is matched with one or more remaining candidate object traces from a plurality of first candidate object traces.

[0063] At 414, after at least one object trace has been pruned, the specific object trace is matched with one or more remaining candidate object traces from a plurality of first candidate object traces. Matching the specific object trace with one or more remaining candidate object traces at 414 can occur by matching at least one candidate object trace from one or more remaining candidate object traces to the specific object trace. For example, the one or more remaining candidate object traces include candidate object traces whose total number is less than the total number of candidate object traces included in the plurality of first candidate object traces determined at 402.

[0064] Candidate object tracks with higher corresponding weights / evidence values ​​are selected for matching, and the remaining object tracks are pruned as candidates. Kurtosis helps determine how many object tracks with high evidence values ​​will be retained as candidates for each matching comparison performed with different candidate object tracks. Kurtosis-based pruning of candidate object tracks minimizes the total number of comparisons performed when a large number of object tracks are fused together. Other problems encountered by fusion trackers can be avoided through kurtosis pruning, which prevents the potential explosion of possible matching combinations from overloading the sensor fusion system.

[0065] Additional examples

[0066] Additional examples of kurtosis-based pruning for sensor fusion systems are provided in the following sections.

[0067] Example 1. A method comprising: determining a plurality of first object tracks by a sensor fusion system based on first sensor data obtained from a first set of sensors; determining a set of second object tracks by the sensor fusion system based on second sensor data obtained from a second set of sensors; applying corresponding weights to a specific object track from the set of second object tracks for each of the plurality of first object tracks by the sensor fusion system; determining a distribution of the corresponding weights applied to the specific object track by the sensor fusion system and based on the corresponding weights applied to the specific object track for each of the plurality of first object tracks; determining the kurtosis of the distribution of the corresponding weights applied to the specific object track by the sensor fusion system; pruning the at least one object track from the plurality of first object tracks by the sensor fusion system based on the corresponding weights applied to the specific object track for at least one object track from the plurality of first object tracks; and after the at least one object track has been pruned, matching the specific object track with one or more remaining object tracks from the plurality of first object tracks by the sensor fusion system.

[0068] Example 2. The method of Example 1, wherein pruning the at least one object trace from the plurality of first object traces includes: discarding the at least one object trace from the plurality of first object traces in response to determining that the corresponding weight applied to the particular object trace for the at least one object trace is less than the corresponding weight applied to the particular object trace for each of the one or more remaining object traces.

[0069] Example 3. The method of any of the preceding examples further includes: determining, based on the kurtosis, the number of candidate object traces to be retained from the plurality of first candidate object traces; dividing the plurality of first candidate object traces into two subsets, the two subsets including a first subset of candidate object traces to be retained and a second subset of candidate object traces to be discarded, the first subset of candidate object traces to be retained including N highest-weighted candidate object traces from the plurality of first candidate object traces applied to the particular object trace, N equal to the number of candidate object traces to be retained; and discarding the at least one candidate object trace from the plurality of first candidate object traces by discarding the second subset of candidate object traces from the plurality of first candidate object traces.

[0070] Example 4. A method in any of the preceding examples, wherein discarding the at least one candidate object trace from the plurality of first candidate object traces includes: retaining the at least one candidate object trace for matching with different candidate object traces from the set of second object traces.

[0071] Example 5. A method in any of the preceding examples, wherein determining the distribution of the corresponding weights applied to the particular object trace comprises: determining a matching error between one or more candidate object traces from the plurality of first candidate object traces and one or more candidate object traces from the set of second object traces; and determining the corresponding weights applied to the particular object trace based on the matching error.

[0072] Example 6. A method in any of the preceding examples, wherein the first set of sensors includes one or more radar sensors, lidar sensors, or ultrasonic sensors, and the second set of sensors includes one or more vision sensors, wherein the one or more vision sensors include an optical camera or an infrared camera.

[0073] Example 7. A method from any of the preceding examples, wherein each object trace from the plurality of first object traces is associated with at least a portion of a stationary object or a moving object.

[0074] Example 8. A method in any of the preceding examples, wherein matching the specific object trace with the one or more remaining object traces includes: matching at least one object trace from the one or more remaining object traces with the specific object trace.

[0075] Example 9. The method of any of the preceding examples, wherein the one or more remaining candidate object traces include object traces whose total amount is less than the total amount of object traces included in the plurality of first object traces.

[0076] Example 10. The method of any of the preceding examples further includes: outputting to a controller for controlling a vehicle an indication of one or more remaining candidate object tracks that match the particular object track.

[0077] Example 11. The method of any of the preceding examples further includes: avoiding outputting an indication to the controller for controlling the vehicle of the at least one candidate object track pruned from the plurality of first candidate object tracks based on the kurtosis, in order to improve vehicle safety.

[0078] Example 12. A system comprising: a processor configured to: determine a plurality of first object tracks based on first sensor data obtained from a first set of sensors; determine a set of second object tracks based on second sensor data obtained from a second set of sensors; apply corresponding weights to a specific object track from the set of second object tracks for each of the plurality of first object tracks; determine a distribution of the corresponding weights applied to the specific object track based on the corresponding weights applied to the specific object track for each of the plurality of first object tracks; determine the kurtosis of the distribution of the corresponding weights applied to the specific object track; prune the at least one object track from the plurality of first object tracks based on the corresponding weights applied to the specific object track for the at least one object track from the plurality of first object tracks; and after the at least one object track has been pruned, match the specific object track with one or more remaining object tracks from the plurality of first object tracks.

[0079] Example 13. A system in any of the preceding examples, wherein the processor is configured to prune the at least one object trace from the plurality of first object traces by: discarding the at least one object trace from the plurality of first object traces in response to determining that the corresponding weight applied to the particular object trace for the at least one object trace is less than the corresponding weight applied to the particular object trace for each of the one or more remaining object traces.

[0080] Example 14. A system in any of the preceding examples, wherein the processor is further configured to: determine, based on the kurtosis, the number of candidate object traces to be retained from the plurality of first candidate object traces; divide the plurality of first candidate object traces into two subsets comprising a first subset of candidate object traces to be retained and a second subset of candidate object traces to be discarded, the first subset of candidate object traces to be retained comprising N highest-weighted candidate object traces from the plurality of first candidate object traces applied to the particular object trace, N equal to the number of candidate object traces to be retained; and discard the at least one candidate object trace from the plurality of first candidate object traces by discarding the second subset of candidate object traces from the plurality of first candidate object traces.

[0081] Example 15. A system in any of the preceding examples, wherein the processor is further configured to: avoid discarding the first subset of candidate object traces from the plurality of first candidate object traces, in order to match a particular object trace with one or more remaining candidate object traces from the plurality of first candidate object traces after the at least one candidate object trace has been pruned.

[0082] Example 16. A system in any of the preceding examples, wherein the processor is configured to determine the distribution of the corresponding weights applied to the particular object trace by means of the following steps: determining a matching error between one or more candidate object traces from the plurality of first candidate object traces and one or more candidate object traces from the set of second object traces; and determining the corresponding weights applied to the particular object trace based on the matching error.

[0083] Example 17. A system in any of the preceding examples, wherein the first set of sensors includes one or more radar sensors, lidar sensors, or ultrasonic sensors, and the second set of sensors includes one or more vision sensors, the one or more vision sensors including an optical camera or an infrared camera.

[0084] Example 18. A method in any of the preceding examples, wherein the processor is configured to match the particular object trace with the one or more remaining object traces by: matching at least one object trace from the one or more remaining object traces with the particular object trace.

[0085] Example 19. A system in any of the preceding examples, wherein the system includes a sensor fusion system communicatively coupled to a controller of a vehicle, the processor being further configured to: output to the controller an indication of one or more remaining candidate object tracks that match the particular object track, for the purpose of controlling the vehicle.

[0086] Example 20. A system in any of the preceding examples, wherein each candidate object trace from the plurality of first candidate object traces is associated with at least a portion of a stationary object or a moving object.

[0087] Example 21. In any of the preceding examples, the system wherein the one or more remaining candidate object traces include candidate object traces whose total number is less than the total number of candidate object traces included in the plurality of first candidate object traces.

[0088] Example 22. A system comprising: means for determining a plurality of first object tracks based on first sensor data obtained from a first set of sensors; means for determining a set of second object tracks based on second sensor data obtained from a second set of sensors; means for applying a corresponding weight to a specific object track from the set of second object tracks for each of the plurality of first object tracks; means for determining a distribution of the corresponding weights applied to the specific object track based on the corresponding weights applied to the specific object track for each of the plurality of first object tracks; means for determining the kurtosis of the distribution of the corresponding weights applied to the specific object track; means for pruning the at least one object track from the plurality of first object tracks based on the corresponding weights applied to the specific object track for at least one object track from the plurality of first object tracks; and means for matching the specific object track with one or more remaining object tracks from the plurality of first object tracks after the at least one object track has been pruned.

[0089] Example 23. A system comprising a sensor fusion system having at least one processor configured to perform a method of any of the preceding examples.

[0090] Example 24. A system of any of the preceding examples, further comprising a controller configured to control a vehicle using information output from a sensor fusion system in response to the at least one processor performing a method of any of the preceding examples.

[0091] Example 25. A system of any of the preceding examples, further including the means of transport.

[0092] Example 26. A system comprising means for performing any of the methods in the preceding examples.

[0093] in conclusion

[0094] While various embodiments of the present disclosure have been described in the foregoing description and illustrated in the accompanying drawings, it should be understood that the present disclosure is not limited thereto, but can be practiced in various ways within the scope of the following claims. It will be apparent from the foregoing description that various modifications can be made without departing from the spirit and scope of the present disclosure as defined by the following claims. Problems associated with combinatorial explosion may also arise in other systems where the number of agents or tasks is large. Therefore, although described as one way to improve vehicle-based matching technology, the techniques described above can be applied to other allocation problems to reduce the total number of assignments between tasks and agents.

[0095] Unless the context explicitly states otherwise, the use of "or" and grammatically related terms indicates an unrestricted, non-exclusive alternative. As used herein, the phrase referring to "at least one" of a list of items means any combination of those items, including a single member. As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

Claims

1. A method for sensor fusion, comprising: determining, by a sensor fusion system, a plurality of first candidate object tracks from first sensor data obtained from a first set of sensors; determining, by the sensor fusion system, a set of second object tracks from second sensor data obtained from a second set of sensors; for each of the first candidate object tracks: determining a respective matching error between the first candidate object track and each of the second object tracks; computing a respective weight as an inverse of the respective matching error between the first candidate object track and each of the second object tracks, the respective weight for considering matching the first candidate object track to each of the second object tracks; for the first candidate object track, applying the respective weight to each of the second object tracks; for the first candidate object track, determining a respective distribution of all respective weights applied to the second object tracks; and for the first candidate object track, determining a kurtosis value of the respective distribution; pruning, by the sensor fusion system, a first candidate object track to discard from the plurality of first candidate object tracks based on the kurtosis value; and after the first candidate object track to discard is pruned, matching, by the sensor fusion system, each second object track to one or more remaining first candidate object tracks from the plurality of first candidate object tracks.

2. The method of claim 1, wherein, pruning the first candidate object track to discard from the plurality of first candidate object tracks comprises: in response to determining that the respective weight applied to each second object track for the first candidate object track to discard is less than the respective weight applied to each second object track for each of the one or more remaining first candidate object tracks, discarding the first candidate object track to discard from the plurality of first candidate object tracks.

3. The method of claim 2, further comprising: based on the kurtosis value, determining a number of first candidate object tracks to retain from the plurality of first candidate object tracks; dividing the plurality of first candidate object tracks into two subsets, the two subsets comprising a first subset of first candidate object tracks to retain and a second subset of first candidate object tracks to discard, the first subset of first candidate object tracks to retain comprising N highest weighted first candidate object tracks from the plurality of first candidate object tracks applied to each second object track, N equaling the number of first candidate object tracks to retain; and discarding the first candidate object track to discard from the plurality of first candidate object tracks by discarding the second subset of first candidate object tracks from the plurality of first candidate object tracks.

4. The method of claim 3, wherein, discarding the first candidate object track to discard from the plurality of first candidate object tracks comprises retaining at least one first candidate object track for matching to a different candidate object track from the set of second object tracks.

5. The method of claim 1, wherein, The first set of sensors includes one or more radar sensors, lidar sensors, or ultrasonic sensors, and the second set of sensors includes one or more vision sensors including optical or infrared cameras.

6. The method of claim 1, wherein, Each first candidate object track from the plurality of first candidate object tracks is associated with at least a portion of a stationary object or a moving object.

7. The method of claim 1, wherein, Matching each second object track to the one or more remaining first candidate object tracks includes: matching at least one first candidate object track from the one or more remaining first candidate object tracks to each second object track.

8. The method of claim 1, wherein, The one or more remaining first candidate object tracks include a total amount of first candidate object tracks that is less than a total amount of first candidate object tracks included in the plurality of first candidate object tracks.

9. A system for sensor fusion, the system comprising: a processor configured for: determining a plurality of first candidate object tracks from first sensor data obtained from a first set of sensors; determining a set of second object tracks from second sensor data obtained from a second set of sensors; for each of the first candidate object tracks: determining respective matching errors between the first candidate object track and each of the second object tracks; computing respective weights as inverses of the respective matching errors between the first candidate object track and each of the second object tracks, the respective weights for considering matching the first candidate object track to each of the second object tracks; for the first candidate object track, applying the respective weights to each of the second object tracks; for the first candidate object track, determining a respective distribution of all respective weights applied to the second object tracks; and for the first candidate object track, determining a kurtosis value of the respective distribution; based on the kurtosis value, pruning a first candidate object track to discard from the plurality of first candidate object tracks; after the first candidate object track to discard is pruned, matching each second object track to one or more remaining first candidate object tracks from the plurality of first candidate object tracks.

10. The system of claim 9, wherein, The processor is configured for pruning the first candidate object track to discard from the plurality of first candidate object tracks by: in response to determining that the respective weight applied to each second object track for the first candidate object track to discard is less than the respective weight applied to each second object track for each of the one or more remaining first candidate object tracks, discarding the first candidate object track to discard from the plurality of first candidate object tracks.

11. The system of claim 9, wherein, The processor is further configured for: based on the kurtosis value, determining a number of first candidate object tracks to retain from the plurality of first candidate object tracks; dividing the plurality of first candidate object tracks into two subsets, the two subsets including a first subset of first candidate object tracks to be retained and a second subset of first candidate object tracks to be discarded, the first subset of first candidate object tracks to be retained including N highest weighted first candidate object tracks from the plurality of first candidate object tracks applied to each second object track, N equaling the number of first candidate object tracks to be retained; and discarding the second subset of first candidate object tracks to be discarded from the plurality of first candidate object tracks.

12. The system of claim 11, wherein, the processor is further configured for: avoiding discarding the first subset of first candidate object tracks from the plurality of first candidate object tracks to match each second object track to one or more remaining first candidate object tracks from the plurality of first candidate object tracks after the first candidate object tracks to be discarded are pruned.

13. The system of claim 9, wherein, the first set of sensors includes one or more radar sensors, lidar sensors, or ultrasonic sensors and the second set of sensors includes one or more vision sensors including optical cameras or infrared cameras.

14. The system of claim 9, wherein, the processor is configured for matching each second object track to the one or more remaining first candidate object tracks by: matching at least one object track from the one or more remaining first candidate object tracks to each second object track.

15. The system of claim 9, wherein, the system includes a sensor fusion system communicatively coupled to a controller of a vehicle, the processor is further configured for: outputting, to the controller, an indication of the one or more remaining first candidate object tracks matched to each second object track for controlling the vehicle.

16. The system of claim 9, wherein, each first candidate object track from the plurality of first candidate object tracks is associated with at least a portion of a stationary object or a moving object.

17. The system of claim 9, wherein, the one or more remaining first candidate object tracks include a total quantity of first candidate object tracks less than a total quantity of first candidate object tracks included in the plurality of first candidate object tracks.

18. A system for sensor fusion, the system comprising: means for determining a plurality of first candidate object tracks from first sensor data obtained from a first set of sensors; means for determining a set of second object tracks from second sensor data obtained from a second set of sensors; for each of the first candidate object tracks, means for: determining a respective matching error between the first candidate object track and each of the second object tracks; computing a respective weight as an inverse of the respective matching error between the first candidate object track and each of the second object tracks, the respective weight for considering matching the first candidate object track to each of the second object tracks; for the first candidate object track, applying the respective weight to each of the second object tracks; determining, for the first candidate object track, a respective distribution of all respective weights applied to the second object track; and determining, for the first candidate object track, a kurtosis value of the respective distribution; means for pruning, based on the kurtosis value, a first candidate object track to discard from the plurality of first candidate object tracks; and means for matching, after the first candidate object track to discard is pruned, each second object track to one or more remaining first candidate object tracks from the plurality of first candidate object tracks.

Images