Multi-scan sensor fusion for object tracking

Abstract

This document describes techniques, systems, and methods for multi-scan sensor fusion for object tracking. Sensor fusion systems can acquire radar and visual tracking data generated for a vehicle's environment. The system maintains a set of hypotheses about the association between visual and radar tracking based on multiple scans of radar and visual data. This set of hypotheses includes quality values ​​for the associations. The system determines a probability value for each hypothesis. Based on these probability values, a match is determined between radar and visual tracking. The system then outputs the match to a semi-autonomous or autonomous driving system to control the vehicle's operation. In this way, the described techniques, systems, and methods can provide high-precision object tracking with quantifiable uncertainty.

Description

Background Technology

[0001] In some vehicles, sensor fusion systems or trackers combine information from multiple sensors to track objects and support autonomous or semi-autonomous driving functions. Some sensor fusion systems combine data from various sensor types (e.g., vision, radar, lidar). When more than one type of sensor is available (e.g., radar and vision), track-to-track fusion (T2TF) of sensor datasets can produce more accurate or robust estimates of objects in the instrument's field of view than using only one type of sensor dataset (e.g., radar or vision). Existing T2TF techniques can apply observations to tracking associations, but are often computationally too complex for many vehicle systems. Implementing these T2TF techniques may also require expensive processing hardware. Furthermore, uncertainties in estimates from existing T2TF systems may not be quantifiable, or even if they are, they may not meet safety regulations governing their use in autonomous or semi-autonomous driving functions. Summary of the Invention

[0002] This document describes techniques, systems, and methods for multi-scan sensor fusion for object tracking. For example, this document describes a method that includes obtaining radar tracking and visual tracking generated for the environment of a vehicle. Radar tracking is generated from radar data from radar sensors on the vehicle. Similarly, visual tracking is generated from visual data from visual sensors on the vehicle. The radar and visual sensors have at least partially overlapping fields of view. The method also includes at least a set of hypotheses to maintain the association between visual tracking and radar tracking based on multiple radar and visual data scans. The hypotheses include quality values ​​of the association between visual tracking and radar tracking. The method further includes determining a probability value for each hypothesis, indicating the likelihood that each hypothesis is accurate. Based on the probability values, one or more matches between radar tracking and visual tracking are determined based on the probability value exceeding a decision threshold. Finally, the method includes outputting the matches to a semi-autonomous or autonomous driving system to control the operation of the vehicle. In this way, the described method can provide high-precision object tracking with quantifiable uncertainty, even meeting the stringent requirements of some state-of-the-art driver assistance systems and autonomous driving systems.

[0003] To perform the example method, the example system includes one or more processors or a computer-readable storage medium having instructions that, when executed, configure the one or more processors to perform the method. The example system may additionally or separately include means for performing the method.

[0004] This document also describes other operations of the methods outlined above and other systems described herein, as well as the means for performing these methods.

[0005] This invention provides a simplified concept related to multi-scan sensor fusion for object tracking, which will be further described below in the detailed description and accompanying drawings. This invention is not intended to identify essential features of the claimed subject matter, nor is it intended to define the scope of the claimed subject matter. Attached Figure Description

[0006] This document describes in detail one or more aspects of multi-scan sensor fusion for object tracking with reference to the following figures. The same numbers are used throughout the figures to refer to similar features and components.

[0007] Figure 1 An example environment according to this disclosure is shown, in which a vehicle is equipped with a sensor fusion system that performs multi-scan sensor fusion for object tracking;

[0008] Figure 2 An example automotive system configured to perform multi-scan sensor fusion for object tracking, according to the technology of this disclosure, is shown.

[0009] Figure 3 An example concept diagram illustrating a technique for performing multi-scan sensor fusion for object tracking, according to this disclosure, is shown.

[0010] Figure 4 An example diagram is shown illustrating the calculation of distance when generating quality values ​​for multi-scan sensor fusion according to this disclosure;

[0011] Figure 5 An example probability matrix is ​​shown for the assumptions used to correlate visual tracking and radar tracking;

[0012] Figure 6 An example flowchart of multi-scan sensor fusion for object tracking according to this disclosure is shown;

[0013] Figure 7-1 and Figure 7-2 This disclosure describes the use of a multi-scan window to perform multi-scan sensor fusion for object tracking; and

[0014] Figure 8 An example flowchart of a method for performing multi-scan sensor fusion for object tracking according to this disclosure is shown. Detailed Implementation

[0015] Overview

[0016] Sensor fusion systems are a crucial technology for driver assistance and autonomous driving systems. Some driving technologies utilize data from multiple types of sensors (e.g., radar, lidar, and vision systems) to monitor objects in the environment of the primary vehicle. Advanced driver assistance systems (e.g., adaptive cruise control (ACC), automatic emergency braking (AEB)) and autonomous driving systems match tracking data from different types of sensors (e.g., radar tracking and vision tracking) to track objects with high precision. This T2TF (Time-to-Time) technology is also essential for the driving scenario understanding, trajectory planning, and control operations of these advanced driver assistance and autonomous driving systems.

[0017] Some sensor fusion systems employ T2TF (Time-to-Time) technology, which uses a single data scan to match one set of tracks (e.g., radar tracks) with another set of tracks (e.g., visual tracks). Typically, tracks from one sensor type (e.g., radar tracks) are forced to be matched or associated with another set of tracks (e.g., visual tracks). Such systems and methods often cannot associate multiple tracks from one sensor type (e.g., multiple radar tracks) with a single track from another sensor type (e.g., a single visual track). Furthermore, these systems are often complex, involving many ad-hoc parameters in the fusion approach.

[0018] In contrast, this document describes a less complex and more accurate sensor fusion technique for performing T2TF. For example, these techniques and systems can use information from multiple scans to provide a more nuanced view of the environment, supporting multiple hypothetical associations between radar tracking and visual tracking (or additional tracking from other sensor types). Furthermore, these techniques and systems can use Dempster-Shafer theory to determine the confidence and plausibility parameters of the assumptions. In this way, the described methods and systems can quantify the uncertainty of the association between radar tracking and visual tracking, better meeting the requirements of advanced driver assistance systems and autonomous driving systems.

[0019] For example, this document describes a technique for performing multi-scan sensor fusion for object tracking. A sensor fusion system for a vehicle can acquire radar and visual tracking generated for the environment. The sensor fusion system maintains a set of hypotheses relating the visual and radar tracking based on multiple scans of radar and visual data. The set of hypotheses includes a mass value for the association. The sensor fusion system determines a probability value for each hypothesis. Based on the probability values, a match is determined between the radar and visual tracking. The sensor fusion system then outputs the match to a semi-autonomous or autonomous driving system to control the operation of the vehicle. In this way, the described techniques, systems, and methods can provide high-precision obstacle tracking with quantifiable uncertainty.

[0020] This section describes only one example of how the described techniques, systems, and methods can perform multi-scan sensor fusion for object tracking. Other examples and implementations are described in this document.

[0021] Operating environment

[0022] Figure 1 An example environment 100 according to this disclosure is shown, in which a vehicle 102 is equipped with a sensor fusion system 104 that performs multi-scan sensor fusion for object tracking. The vehicle 102 can represent any device or machine, including operating systems and unmanned systems for various purposes. Some non-exhaustive and non-limiting examples of the vehicle 102 include other motorized vehicles (e.g., motorcycles, buses, tractors, semi-trailers, watercraft, aircraft, or construction equipment).

[0023] In the depicted environment 100, the sensor fusion system 104 is mounted on or integrated into the vehicle 102. The vehicle 102 can travel on the road 114 and navigate within the environment 100 using the sensor fusion system 104. Objects can be located near the vehicle 102. For example, Figure 1 Another vehicle 112 is depicted traveling along road 114 in front of vehicle 102 and in the same direction as vehicle 102.

[0024] Using sensor fusion system 104, vehicle 102 has an instrument or sensor field of view 116 that covers another vehicle 112. The sensors (e.g., radar systems, vision systems) may have the same, similar, or different instrument fields of view 116 over road 114. For example, a vision system may project an instrument field of view 116-1 (…). Figure 1 (Not shown in the image). The radar system can project the instrument's field of view 116-2 from any external surface of the vehicle 102. Figure 1(Not shown in the image). Vehicle manufacturers may integrate at least a portion of a vision or radar system into side mirrors, bumpers, roofs, or any other internal or external location where the field of view 116 includes road 114. Generally, vehicle manufacturers may design sensor locations to provide a field of view 116 that adequately covers the road 114 over which vehicle 102 may be traveling. For example, sensor fusion system 104 may monitor another vehicle 112 (as detected by the sensor system) on road 114 over which vehicle 102 is traveling.

[0025] Sensor fusion system 104 includes a fusion module 108 and one or more sensor interfaces 106, including a vision interface 106-1 and a radar interface 106-2. Sensor interfaces 106 may include additional sensor interfaces, including another sensor interface 106-n, where n represents the number of sensor interfaces. For example, sensor fusion system 104 may include interfaces to other sensors (e.g., lidar sensors) or other vision or radar systems. Although not explicitly stated... Figure 1 As shown, the fusion module 108 executes on a processor or other hardware. During execution, the fusion module 108 can track objects based on sensor data obtained at the vision interface 106-1 and the radar interface 106-2. The vision interface 106-1 receives visual or camera data from one or more vision systems of the vehicle 102. The radar interface 106-2 receives radar data from one or more radar systems of the vehicle 102. Specifically, the fusion module 108 can access the vision interface 106-1 and the radar interface 106-2 to obtain visual data and radar data, respectively.

[0026] The fusion module 108 configures the sensor fusion system 104 to combine different types of sensor data obtained from the sensor interface 106 into visual tracking and radar tracking. Figure 1 (Not shown in the image) for tracking objects in the field of view 116. The fusion module 108 can determine multiple object tracks based on first sensor data (e.g., obtained from the vision interface 106-1) and identify additional object tracks based on second sensor data (e.g., obtained from the radar interface 106-2). Alternatively, the fusion module 108 receives visual tracking and radar tracking from the respective sensor systems via the sensor interface 106. The fusion module 108 can determine the correlation between visual tracking and radar tracking and maintain or store these correlations in the tracking correlation data storage 110.

[0027] For example, fusion module 108 can use a sliding window with multiple scans of sensor data to create associations between tracks generated by different types of sensor systems. These tracking associations can be stored in tracking association data storage 110. For example, sensor fusion system 104 can create associations between one or more visual tracks collected or acquired through vision interface 106-1 and one or more radar tracks collected or acquired through radar interface 106-2. A single scan of environment 100 typically lacks sufficient contextual information to provide a nuanced view of the environment. Because evidence from a single scan may be insufficient to support multiple hypothetical associations, fusion module 108 uses a sliding window with multiple scans of sensor data to create associations between tracks.

[0028] The fusion module 108 can determine multiple hypotheses for one or more associations. The fusion module 108 can also use Dempster-Shaffer theory applied to the multiple hypotheses to combine quality values ​​and generate a fused quality value. Dempster-Shaffer theory provides a framework for determining and reasoning about the uncertainty between multiple hypotheses. Dempster-Shaffer theory allows evidence from different sources (e.g., radar tracking and visual tracking) to be combined to arrive at a certain degree of confidence in the hypotheses when all available evidence is considered. Each given hypothesis is included in the identification framework. Applying Dempster-Shaffer theory to sensor fusion is sometimes referred to as Dempster-Shaffer fusion. The sensor fusion system 104 can also use Dempster-Shaffer fusion to output confidence parameters (e.g., probability) and plausibility parameters (e.g., confidence level) associated with each of the one or more hypotheses based on the fused quality value output. (This document references...) Figures 2 to 8 The operation and components of the sensor fusion system 104 and fusion module 108 are described in more detail.

[0029] Vehicle 102 may also include one or more vehicle-based systems. Figure 1(Not shown in the image), one or more vehicle-based systems can use tracking assignments, confidence parameters, and plausibility parameters from sensor fusion system 104 to operate vehicle 102 on road 114. Vehicle-based systems may include driver assistance systems and autonomous driving systems (e.g., adaptive cruise control (ACC), traffic jam assist (TJA), lane centering assist (LCA), and L0-L4 autonomous driving systems). Typically, vehicle-based systems can use tracking assignments provided by sensor fusion system 104 to operate the vehicle and perform specific driving functions. For example, a driver assistance system can provide adaptive cruise control and monitor the presence of another vehicle 112 in the lane in which vehicle 102 is traveling or in an adjacent lane. As another example, a driver assistance system can provide an alert when another vehicle 102 crosses lane markings and enters the same lane as vehicle 102.

[0030] The autonomous driving system can move vehicle 102 to a specific location on road 114 while avoiding collisions with objects detected by different systems on vehicle 102 (e.g., radar and vision systems) and with another vehicle 112. Tracking assignment provided by sensor fusion system 104 can provide information about the position and trajectory of another vehicle 112, enabling the autonomous driving system to perform lane changes or steer vehicle 102.

[0031] Example Architecture

[0032] Figure 2 An example automotive system 200 configured to perform multi-scan sensor fusion for object tracking according to the technology of this disclosure is shown. The automotive system 200 can be integrated into... Figure 1 The vehicle 102 shown is described in this context. For example, the vehicle system 200 includes a controller 202 and a sensor fusion system 104-1, which is... Figure 1 An example of a sensor fusion system 104 is provided. Sensor fusion system 104-1 and controller 202 communicate via link 212. Link 212 can be wired or wireless, and in some cases includes a communication bus. Controller 202 performs operations based on information received via link 212, such as confidence and plausibility parameters of multiple hypotheses output from sensor fusion system 104-1 when processing and associating radar tracking and visual tracking to identify objects in the field of view 116.

[0033] The controller 202 includes a processor 204-1 (e.g., a hardware processor, a processing unit) and a computer-readable storage medium (CRM) 206-1 (e.g., a memory, long-term storage, short-term storage) storing instructions for the vehicle module 208.

[0034] The sensor fusion system 104-1 includes a vision interface 106-1 and a radar interface 106-2. As discussed above, any number of other sensor interfaces 106 can be used, including lidar interfaces or other sensor interfaces 106-n. The sensor fusion system 104-1 may include processing hardware, including a processor 204-2 (e.g., a hardware processor, processing unit) and a CRM 206-2, which stores and fuses the data with the fusion module 108-1 (fusion module 108-1 is...). Figure 1 Examples of fusion module 108) and instructions associated with credibility and rationality module 210. Fusion module 108-1 includes tracking associated data storage 110-1.

[0035] Processors 204-1 and 204-2 may be two separate processing units or a single processing unit (e.g., a microprocessor). Processors 204-1 and 204-2 may also be a pair of system-on-a-chip (SoC) or a single SoC of a computing device, controller, or control unit. Processors 204-1 and 204-2 execute computer-executable instructions stored within CRM 206-1 and CRM 206-2. As an example, processor 204-1 may execute vehicle module 208 to perform driving functions of vehicle system 200 (e.g., autonomous lane change maneuvering, semi-autonomous lane keeping feature, ACC function, TJA function, LCA function, or other autonomous or semi-autonomous driving functions) or other operations. Similarly, processor 204-2 may execute fusion module 108-1 to infer and track objects in field of view 116 based on sensor data obtained from multiple different sensor interfaces 106 of vehicle system 200. When executed at processor 204-1, vehicle module 208 can receive one or more objects (e.g., detected by fusion module 108-1) in response to fusion module 108-1 combining and analyzing sensor data obtained from each of sensor interfaces 106. Figure 1 The instructions for another vehicle (112) shown in the figure.

[0036] Typically, vehicle system 200 executes vehicle module 208 to perform vehicle functions using the output from sensor fusion system 104-1. For example, vehicle module 208 can provide adaptive cruise control and monitor for objects in or near field of view 116 to slow vehicle 102 and prevent collisions with the rear of other vehicles 112. In such an example, fusion module 108-1 can provide sensor data or its derivatives (e.g., confidence parameters and plausibility parameters) as output to vehicle module 208. Vehicle module 208 can also provide an alert or trigger specific maneuvers when data obtained from fusion module 108-1 indicates that one or more objects are crossing in front of vehicle 102.

[0037] For simplicity, the tracking-related data storage 110-1 and the credibility and plausibility module 210 are described primarily with reference to the vision interface 106-1 and the radar interface 106-2 (without reference to the other sensor interface 106-n). However, it should be understood that the fusion module 108-1 can combine sensor data or tracking from more than two different types of sensors and can rely on sensor data output from other types of sensors besides vision and radar systems.

[0038] The vision interface 106-1 provides a list of vision-based object tracking (also referred to herein as "visual tracking"). The vision interface 106-1 outputs sensor data (which may be provided in various forms, such as a list of candidate objects being tracked) and estimates of each of the following: the object's position, velocity, object class, and reference angles (e.g., azimuth angles to a centroid reference point on the object (such as the center of the rear surface of another vehicle 112), and other extended angles to the near corner of the rear of the other vehicle 112).

[0039] The radar interface 106-2 can operate independently of the vision interface 106-1 and other sensor interfaces 106-n, similar to the vision interface 106-1. The radar interface 106-2 can maintain a list of detections and corresponding detection times, assumed to primarily be the dispersion center tracking the vehicles and other objects it detects. Each detection typically consists of distance, rate of change of distance, and azimuth. For each vehicle and object that is not obstructed within the field of view 116 and is relatively close to the vehicle 102, there is typically more than one detection.

[0040] The vision interface 106-1 can estimate azimuth and object classification more accurately than other sensor types. However, the vision interface 106-1 may have limitations in estimating some parameters (such as longitudinal position or distance, velocity, etc.). The radar interface 106-2 can accurately measure object distance and rate of change of distance, but may be less accurate in measuring azimuth; vision systems are generally more accurate in measuring azimuth. The complementary nature of vision and radar systems leads to an accuracy advantage when matching vision tracking with radar tracking.

[0041] The fusion module 108-1 maintains a set of associations between visual tracking and radar tracking in the tracking association data storage 110-1. The fusion module 108-1 can determine multiple hypotheses for the set of associations between radar tracking and visual tracking (e.g., a first hypothesis that one or more radar tracks match a specific visual track, a second hypothesis that a specific radar track does not correspond to any visual track, or a third hypothesis that a specific visual track does not match any radar track). In other respects, the set of associations can be based on local tracking confidence and time uncertainty.

[0042] The fusion module 108-1 can use Dempster-Schafer fusion to combine quality values ​​from multiple hypotheses to generate a fused quality value. The credibility and plausibility module 210 can use the fused quality value to generate credibility and plausibility parameters for tracking associations within each group. The credibility and plausibility module 210 can also output the credibility and plausibility parameters to the vehicle module 208.

[0043] Figure 3 A sample concept diagram 300 is shown according to this disclosure, illustrating a technique for performing multi-scan sensor fusion for object tracking. Concept diagram 300 includes a fusion module 108-2, which is... Figure 1 The fusion module 108 and Figure 2 Example of fusion module 108-1.

[0044] Visual tracking 302 and radar tracking 304 are provided as inputs to fusion module 108-2. Visual tracking 302 is generated based on visual data. Similarly, radar tracking 304 is generated based on radar data. Fusion module 108-2 includes input processing module 306, evidence fusion module 314, and output processing module 322. Fusion module 108-2 outputs matching 328, which can be provided to vehicle module 208. Generally, input processing module 306 can preprocess visual tracking 302 and radar tracking 304 by performing tracking compensation 308, evaluate the feasibility pairs of visual tracking 302 and radar tracking 304 by performing feasibility matrix formation 310, and generate a quality value for each pair by performing quality extraction.

[0045] The fusion module 108-2 uses a Framework of Identification (FOD), which is defined as the set of all possible hypotheses describing the matching of visual tracking 302 with radar tracking 304. The elements of the FOD are mutually exclusive and exhaustive. For convenience, visual tracking 302 and radar tracking 304 are represented by equations (1) and (2), respectively:

[0046]

[0047]

[0048] Where M represents the number of visual tracks 302, and N represents the number of radar tracks 304. The number of visual tracks 302, M, can be less than, equal to, or greater than the number of radar tracks 304, N.

[0049] Input processing module 306 can define a first FOD by considering matching each radar track 304 with a visual track 302. The first FOD is defined by equation (3):

[0050]

[0051] Where g i r represents the index of feasible visual tracking 302 after the gating step. i It is the i-th radar tracking 304, and Y indicates r. i Not associated with any visual tracking 302.

[0052] Input processing module 306 can define a second FOD by considering matching each visual track 302 with a radar track 304. The second FOD is defined by equation (4):

[0053]

[0054] Where g i This indicates the index of feasible radar tracking 304 after the gating step, v i It is the i-th visual tracking 302, and Ξ indicates r i It is not associated with any radar tracking 304.

[0055] The output processing module 306 assigns a quality value to each element in the FOD based on different information sources. Single visual tracking 302(T) v ) and a single radar tracker 304 (T) r The following can be represented respectively: It has a covariance matrix and It has a covariance matrix For convenience, we assume that each track has the same length (e.g., n1 = n2 = n).

[0056] Quality extraction 312 can be performed based on distance, bounding box size, object category information, or object heading information. Input processing module 306 can use the distance between two tracks to calculate the quality value. The distance can be calculated using equation (5) without considering covariance:

[0057]

[0058] Alternatively, (6) can be used to calculate the distance between two tracks, taking covariance into account:

[0059]

[0060] The distance between two tracks can also be calculated using KL divergence, as shown in equation (7):

[0061]

[0062] Given a set of visual tracking 302 and radar tracking 304, the input processing module 306 can determine the mass value of the first FOD based on the distance using equations (8a) and (8b):

[0063]

[0064]

[0065] The gating threshold can be applied to d≤α and d Υ =α is used to set the threshold. For each source of evidence, the input processing module 306 can assume that the threshold is fixed. For the second FOD, the input processing module 306 can similarly calculate the quality value based on the distance.

[0066] The input processing module 306 can also calculate the quality value of the association between visual tracking 302 and radar tracking 304 based on the association probability between visual tracking 302 and radar tracking 304. From a probabilistic perspective, T v The probability density function can be expressed by equation (9):

[0067]

[0068] Then, the similarity is described by the negative log-likelihood (NLL) in equation (10):

[0069]

[0070] Given a set of visual tracking 302 and radar tracking 304, the input processing module 306 can determine the quality value of the first FOD based on the similarity using equations (11a) and (11b):

[0071]

[0072]

[0073] For the second FOD, the input processing module 306 can similarly calculate the quality value based on similarity, where

[0074] The input processing module 306 can also calculate the associated quality value of visual tracking 302 and radar tracking 304 based on the bounding box size information. The bounding box information can be used based on the assumptions that the object size is the same in each tracking and that the matching of these tracking methods is perfect. Based on these assumptions, the input processing module 306 can use the area of ​​the bounding box to calculate the quality value. For convenience, the area of ​​visual tracking 302 is... Given. Similarly, the area of ​​radar tracking 304 is given by The distance between the two tracks, representing the area information, is given by equation (12):

[0075]

[0076] Instead of using area, input processing module 306 can similarly use length and width directly, as shown in equation (13):

[0077]

[0078] lh k It indicates length and height information.

[0079] Given a set of visual tracking 302 and radar tracking 304, the input processing module 306 can determine the mass value of the first FOD using equations (14a) and (14b) based on the distance of the area from equation (12) or (13):

[0080]

[0081]

[0082] For the second FOD, the input processing module 306 can similarly calculate the quality value based on the bounding box size information, whereby...

[0083] The input processing module 306 can also calculate the associated quality value of visual tracking 302 and radar tracking 304 based on object category information. The object category can be defined as C = {c1, c2, ..., c...} m Accordingly, the category of visual tracking 302 is determined by... Given. Similarly, the category of radar tracking 304 is determined by... The distance of category information between two tracks is given by equation (15):

[0084]

[0085] in The cross-entropy is described by equation (16):

[0086]

[0087] Given a set of visual tracking 302 and radar tracking 304, the input processing module 306 can determine the quality value of the first FOD based on the distance of the object category information using equations (17a) and (17b):

[0088]

[0089]

[0090] For the second FOD, the input processing module 306 can similarly calculate the quality value based on the object category information, wherein...

[0091] The input processing module 306 can also calculate the associated mass value of visual tracking 302 and radar tracking 304 based on the object's heading information.

[0092] The input processing module 306 can also calculate the associated quality value of visual tracking 302 and radar tracking 304 based on object category information. The input processing module 306 considers the latest object heading information, as this information is captured in the position information section. The heading of visual tracking 302 is determined by h. v Given. Similarly, the radar tracking heading of 304 is given by h. r The distance between the heading information of the two tracks is given by equation (18):

[0093] d h =|1-cos(|h v -h r |)| (18)

[0094] Given a set of visual tracking 302 and radar tracking 304, the input processing module 306 can determine the mass value of the first FOD based on the distance of the heading information using equations (19a) and (19b):

[0095]

[0096]

[0097] For the second FOD, the input processing module 306 can similarly calculate the quality value based on the object category information, wherein... The input processing module 306 can similarly use speed information to extract quality values.

[0098] The evidence fusion module 314 can determine the credibility and plausibility parameters of each pair by performing stability score calculation 316, generate a fused quality value by performing quality fusion 318 via Dempster-Schafer fusion, and determine the probability parameters by performing probability calculation 320.

[0099] The evidence fusion module 314 can use Dempster-Shaffer fusion to fuse the quality values ​​of multiple hypotheses. First, based on the quality values ​​from different sources, the evidence fusion module 314 uses Dempster-Shaffer theory to obtain a fused quality value for a specific hypothesis. Two information sources can be fused to determine the joint quality value m using equations (20) to (22). 1,2 (·):

[0100]

[0101]

[0102] in

[0103]

[0104] The expression (1-K) represents the normalization coefficients, where K indicates the conflict between pieces of evidence, m1(·) and m2(·) represent the quality values ​​corresponding to specific hypotheses of the first and second pieces of evidence, respectively, and B and C are hypotheses determined by FOD. The evidence fusion module 314 may include the calculation of the quality value for each element in a given identification framework based on different information sources (e.g., distance, similarity, tracking heading information, bounding box information, object category information, or object heading information).

[0105] Secondly, based on the quality value of the fusion, the evidence fusion module 314 calculates the credibility parameter and the reasonableness parameter. The credibility bel(H) and reasonableness pl(H) of a specific hypothesis H are given by equations (23) and (24):

[0106] bel(H)=∑ B|B∈H m(B) (23)

[0107]

[0108] During the initialization phase, the evidence fusion module 314 has no available quality values ​​and uses only information from measurements.

[0109] The output processing module 322 can identify conflict-free hypotheses by performing matching hypothesis extraction 324, and can determine conflicting hypotheses by performing conflicting hypothesis maintenance 326. As a result of the described processing technique, the output processing module 322 outputs a match 328 with improved accuracy.

[0110] Output processing module 322 can use the first FOD (reference) Figure 5 The output processing module 322 extracts associations from the probability matrix of the first FOD (described in more detail) or the second FOD. Specifically, the output processing module 322 can extract associations whose probabilities exceed a predefined decision threshold (e.g., 0.5). If no association exceeds the decision threshold, the output processing module 322 can extract fuzzy associations with probabilities exceeding a predefined significance threshold (e.g., 0.3). The same extraction is then performed on the second FOD. Associations or hypotheses extracted that do not conflict between the first and second FODs and exceed the decision threshold are maintained in the data storage. If there is a conflict between associations in the probability matrix or among associations exceeding the significance threshold, these hypotheses are maintained in the conflict data storage. Hypotheses in the conflict data storage are not used as tracking association outputs, but the output processing module 322 maintains these hypotheses for the next scan. In the next scan, if the probability is below the significance threshold, the output processing module 322 prunes the conflicting hypothesis, or if the probability is above the decision threshold, the conflicting hypothesis is accepted. If a conflicting hypothesis passes the significance threshold but not the decision threshold, the output processing module 322 maintains the conflicting hypothesis for tracking.

[0111] Figure 4 An example figure 400 is shown illustrating the calculation of distance when generating quality values ​​for multi-scan sensor fusion according to this disclosure. Figure 400 includes visual tracking 302 defined by points x1, x2, x3, x4, x5, x6 (points 402, 404, 406, 408, 410, and 412, respectively) and... Radar tracking 304 is defined as points 414, 416, 418, 420, 422, and 424, respectively. Input processing module 306 can calculate the distance between points on visual tracking 302 and points on radar tracking 304 (e.g., the distance between points 402 and 414). For example, this can be done at x6 and... The distance is calculated as 426. In example figure 400, input processing module 306 assumes that the radar tracking length is the same as the visual tracking length. Without considering covariance, the distance calculation can be defined by equation (5):

[0112]

[0113] Alternatively, in consideration of by Given a defined covariance matrix, distance calculation can be defined by equation (6):

[0114]

[0115] Figure 5 An example probability matrix 500 is shown for the assumptions used to correlate visual tracking 302 and radar tracking 304. Probability matrix 500 is for the first FOD (e.g., The first FOD is defined as the set of all possible hypotheses that associate radar tracking 304 with visual tracking 302 or visual tracking 302 with radar tracking 304. The probability matrix 500 indicates the probability of each possible tracking association in the first FOD. Specifically, the probability matrix 500 considers a set of radar tracking 502 and a set of visual tracking 504. In the probability matrix 500, each element p... ij Based on the quality function corresponding to the first FOD and The associated probability. For example, tracking the first radar. With first-person visual tracking The probability of association is 510(p). 11 ).

[0116] A similar probability matrix can be used for the second FOD (e.g., This is formed. In this probability matrix, each element... Based on the quality function corresponding to the second FOD and The associated probabilities. In the second FOD, the probability matrix can also consider the possibility of a single visual track matching multiple radar tracks. Probabilities can be determined using a pignistic transformation. After gating or clustering, many elements in the probability matrix are zero, and fewer possible associations need to be considered.

[0117] Figure 6 An example flowchart 600 for multi-scan sensor fusion for object tracking according to this disclosure is shown. The operation of flowchart 600 can be performed by... Figures 1 to 3 The fusion module 108, 108-1, or 108-2, or a similar fusion module, is executed. Flowchart 600 may also include reference modules. Figure 6 The described operation is fewer or more operations.

[0118] At 602, the fusion module 108 receives or acquires visual tracking and radar tracking. For example, the fusion module 108 may receive visual tracking and radar tracking via visual interface 106-1 and radar interface 106-2, respectively. In other implementations, the fusion module 108 may receive visual data and radar data via visual interface 106-1 and radar interface 106-2, respectively.

[0119] At 604, the fusion module 108 uses multiple scans to initialize the tracking match, and then uses a sliding window to continue processing the match. The use of multiple scans is described in more detail with reference to Figure 7.

[0120] At 606, the fusion module 108 can perform clustering to reduce the number of tracks to be processed and considered. Clustering can be based on geometric distance or probability density. It is assumed that the tracks are well aligned in time, and Euclidean distance is used as the metric for clustering the tracks. If the different tracks are not well aligned in time, a dynamic temporal warp distance metric (DTW) can be used.

[0121] At 608, fusion module 108 determines whether each radar track and each visual track has previously been matched or associated with another track. If the tracks have previously been matched, fusion module 108 calculates a stability score for the match or association at 610. At 612, fusion module 108 determines whether the stability score is greater than a decision threshold. If the stability score is greater than the decision threshold, the association between the radar track and the visual track is output as a match at 614. If the matched radar track and visual track have been verified to be maintained with good confidence in previous iterations of flowchart 600, fusion module 108 can skip the matching process. Fusion module 108 uses the stability score to account for the time uncertainty of the current window and the uncertainty of how well each tracker maintains the track. The stability score is defined using equation (25):

[0122] SS = e -αΔt ·S v,Δt ·S r,Δt (25)

[0123] Where Δt is the time distance between the current scan and the last scan where the match was verified. v,Δt Define using equations (26) and (27):

[0124]

[0125] in

[0126]

[0127] p d It is the detection probability, and It is a possibility of a tracker. Similarly, S r,Δt Define using equations (28) and (29):

[0128]

[0129] in

[0130]

[0131] The fusion module 108 uses a threshold R to determine whether a matching operation can be skipped in a specific iteration of flowchart 600. The fusion module 108 can also set an interruption threshold N. break Defined as the number of unavailable scans from a specific track. A match is broken when the number of unavailable scans is greater than or equal to the break threshold. A sliding window is then used again to associate the track pair.

[0132] At 616, if a tracking was not previously matched, or if the stability score is less than a decision threshold, the fusion module 108 calculates a quality value for each potential hypothesis to correlate the unmatched tracking with other tracking using all available measurements. Hypotheses may include one or more radar trackings matching a specific visual tracking, radar tracking not matching visual tracking, or visual tracking not matching radar tracking. At 618, after calculating the quality value for each hypothesis, the fusion module 108 identifies consistent and conflicting pairs, as shown regarding... Figure 5 As described. At 620, the status update of the matched pair is provided to the vehicle module 208.

[0133] Figure 7-1 and Figure 7-2 This disclosure illustrates the use of a multi-scan window to perform multi-scan sensor fusion for object tracking. Figure 7-1 Example Figure 700 illustrates the initialization of a multi-scan window. Windows (e.g., windows 730, 732, and 734) can be used to maintain multi-scan information instead of a single scan by using a "sliding" window based on certain standards. Windows can also be used for... Figures 3 to 6 The input selection for the matching algorithm is described in detail below. For example, consider single visual tracking information x1, c2, c…, c wz ,x wz+1 ,x wz+2 (Visual tracking data 702, 704, 706, 708, 710, 712, and 714, respectively) and radar tracking information (Represented by radar tracking data 716, 718, 720, 722, 724, 726, and 728). Individual tracks can be assigned to windows and processed accordingly. For example, the first window 730, the second window 732, and the third window 734 are processed sequentially. Because the length of each visual track and radar track is less than the size of a single window or scan, tracking information accumulates in a sliding window. The currently available information for each track (e.g., within a specific window) is used to calculate the similarity between different visual and radar tracks.

[0134] Figure 7-2 Example Figure 740 illustrates the use of a sliding window to perform multi-scan sensor fusion for object tracking. A first window 742 at time k includes tracking information from multiple scans (e.g., visual tracking data 702, 704, 706, and 708, and radar tracking data 716, 718, 720, and 722). A second window 744 at time k+1 includes tracking information from multiple scans (e.g., visual tracking data 704, 706, 708, and 710, and radar tracking data 718, 720, 722, and 724). The first window 742 and the second window 744 represent different windows. The tracking information used at time k+1 can be obtained by "sliding" the window forward one step or by scanning from the beginning of time k. The sliding window allows the fusion module 108 to maintain a history of tracking data and hypotheses, which can then be used to minimize the number of potential matches or reduce the complexity of the matching.

[0135] Example Method

[0136] Figure 8 An example flowchart 800 of a method for performing multi-scan sensor fusion for object tracking according to this disclosure is shown. Example flowchart 800 is shown as a set of operations 802 to 810 performed in the order or combination of the operations shown or described (but not necessarily limited to). Furthermore, any of operations 802 to 810 may be repeated, combined, or rearranged to provide other methods. In the various sections discussed below, reference may be made to environment 100 and the entities detailed above, which are referred to by way of example only. This technique is not limited to being performed by one or more entities.

[0137] At point 802, the vehicle's sensor fusion system obtains radar tracking and visual tracking generated for the vehicle's environment. For example, the vehicle's sensor fusion system 104 can obtain visual tracking 302 and radar tracking 304 generated for the environment 100. Visual tracking 302 is generated based on visual data from one or more visual sensors on the vehicle 102. Radar tracking 304 is generated based on radar data from one or more radar sensors on the vehicle 102. The radar sensors have fields of view that at least partially overlap with the fields of view of the visual sensors.

[0138] At point 804, the sensor fusion system maintains at least one set of hypotheses regarding the association between visual tracking and radar tracking based on multiple scans of radar and visual data. For example, sensor fusion system 104 may maintain multiple sets of hypotheses regarding the association between visual tracking 302 and radar tracking 304 based on multiple scans of radar and visual data. The hypotheses include at least one of the following: one or more radar tracks match visual tracking; visual tracking matches one or more radar tracks; radar tracking does not match visual tracking; or visual tracking does not match radar tracking. Specifically, the hypotheses may include a first set of hypotheses (FOD) and a second set of hypotheses (FOD). The first set of hypotheses may include at least one of the following: one or more radar tracks match visual tracking; and radar tracking does not match visual tracking. The second set of hypotheses may include at least one of the following: visual tracking matches one or more radar tracks; and visual tracking does not match radar tracking.

[0139] The sensor fusion system 104 uses a window of multiple scans from radar and visual tracking, which includes radar and visual data from the current sensor scan and at least one previous sensor scan. The window moves forward one step in each iteration of operation 804. The window is initialized using a single scan of radar and visual tracking and accumulates additional scans of radar and visual tracking in each iteration of operation 804 until the window reaches a predetermined window size.

[0140] The sensor fusion system 104 can also determine the quality value of the association between visual tracking 302 and radar tracking 304. The quality value can be assigned based on the distance between the tracks, the similarity between the tracks, heading information, bounding box information, object category information, velocity information, or object heading information. In some implementations, the quality value can be assigned based on at least three of the following: distance between the tracks, the similarity between the tracks, heading information, bounding box information, object category information, velocity information, or object heading information. The sensor fusion system 104 can determine a confidence parameter and a plausibility parameter for each hypothesis based on the quality value. The quality values ​​can be combined using a Dempster-Schafer fusion process to generate a fused quality value. The confidence parameter and plausibility parameter can be determined based on the fused quality value.

[0141] At point 806, the sensor fusion system determines a probability value for each hypothesis in the at least one set of hypotheses based on the quality value. For example, sensor fusion system 104 may determine a probability matrix for each hypothesis in the hypothesis set based on the quality value. The probability value indicates the likelihood that a particular hypothesis is accurate.

[0142] At point 808, the sensor fusion system determines one or more matches between one or more radar tracking and visual tracking based on the probability value of each hypothesis. For example, sensor fusion system 104 may determine one or more matches between radar tracking and visual tracking based on the probability values ​​of hypotheses. Matches include one or more hypotheses with probability values ​​exceeding a decision threshold. Sensor fusion system 104 may also extract one or more first hypotheses with probability values ​​exceeding the decision threshold from a first set of hypotheses and one or more second hypotheses with probability values ​​exceeding the decision threshold from a second set of hypotheses. Sensor fusion system 104 may then identify matches as those of the first and second hypotheses that exceed the decision threshold.

[0143] Sensor fusion system 104 may extract one or more third hypotheses with probability values ​​exceeding a significance threshold from the first set of hypotheses and one or more fourth hypotheses with probability values ​​exceeding a significance threshold from the second set of hypotheses, in response to the absence of any hypothesis exceeding a decision threshold in either the first or second set of hypotheses. The significance threshold has a value lower than the decision threshold (e.g., 0.5) (e.g., 0.3). Sensor fusion system 104 may maintain conflicting hypotheses in the first and second sets that do not simultaneously exceed the decision threshold, as well as hypotheses exceeding the significance threshold. If the stability score of a match is higher than a threshold, sensor fusion system 104 may skip the verification of the match in subsequent iterations of operation 808. The stability score indicates the local tracking confidence and time uncertainty associated with each match. Sensor fusion system 104 may also maintain one or more third hypotheses and one or more fourth hypotheses.

[0144] At point 810, the sensor fusion system outputs the one or more matches to the vehicle's semi-autonomous or autonomous driving system and a tracker. The tracker's output is then used to control the vehicle's operation. For example, sensor fusion system 104 can output the matches to a tracker. The tracker's output can be provided to vehicle module 208 to control the operation of vehicle 102. Sensor fusion system 104 can also output confidence and rationality parameters of the matches to further update the matches and control the operation of vehicle 102 based on the confidence and rationality parameters.

[0145] Additional examples

[0146] The following sections provide additional examples of multi-scan sensor fusion for object tracking.

[0147] Example 1. A method comprising: obtaining environment-generated radar tracking and visual tracking for a vehicle, the radar tracking being generated based on radar data from one or more radar sensors on the vehicle, and the visual tracking being generated based on visual data from one or more visual sensors on the vehicle, the one or more radar sensors having a first field of view that at least partially overlaps with a second field of view of the one or more visual sensors; maintaining at least one set of hypotheses about the association between the visual tracking and the radar tracking based on multiple scans of the radar data and the visual data, the at least one set of hypotheses including a quality value of the association between the visual tracking and the radar tracking; determining a probability value for each of the at least one set of hypotheses based on the quality value, the probability value indicating the likelihood that each hypothesis is accurate; determining one or more matches between the one or more radar tracking and the visual tracking based on the probability value of each hypothesis, the one or more matches including one or more hypotheses having a probability value exceeding a decision threshold; and outputting one or more matches to a semi-autonomous or autonomous driving system of the vehicle to control the operation of the vehicle.

[0148] Example 2. The method of Example 1, further comprising: assigning a quality value to the association between visual tracking and radar tracking based on at least one of the following: distance between tracking, similarity between tracking, heading information, bounding box information, object category information, velocity information, or object heading information.

[0149] Example 3. The method of Example 2, wherein a quality value is assigned to the association between visual tracking and radar tracking based on at least three of the following: distance between tracks, similarity between tracks, heading information, bounding box information, object category information, velocity information, or object heading information.

[0150] Example 4. The method of any of the preceding examples, wherein at least one set of assumptions includes at least one of the following: one or more radar tracks match visual tracks, visual tracks match one or more radar tracks, radar tracks do not match visual tracks, or visual tracks do not match radar tracks.

[0151] Example 5. The method of any of the preceding examples, wherein at least one set of assumptions includes a first set of assumptions and a second set of assumptions, the first set of assumptions including at least one of the following: one or more radar tracks match visual tracks, and radar tracks do not match visual tracks, and the second set of assumptions including at least one of the following: visual tracks match one or more radar tracks, and visual tracks do not match radar tracks.

[0152] Example 6. The method of Example 5, wherein determining one or more matches between one or more radar tracking and visual tracking comprises: extracting one or more first hypotheses from a first set of hypotheses having probability values ​​exceeding a decision threshold; extracting one or more second hypotheses from a second set of hypotheses having probability values ​​exceeding a decision threshold; and identifying one or more matches as hypotheses in which both one or more first hypotheses and one or more second hypotheses exceed the decision threshold.

[0153] Example 7. The method of Example 6, the method further comprising: in response to the assumption that no hypothesis in the first set of hypotheses exceeds a decision threshold, extracting one or more third hypotheses having a probability value exceeding a significance threshold, the significance threshold having a value lower than the decision threshold; in response to the assumption that no hypothesis in the second set of hypotheses exceeds a decision threshold, extracting one or more fourth hypotheses having a probability value exceeding a significance threshold; maintaining conflicting hypotheses among one or more first hypotheses and one or more second hypotheses that do not simultaneously exceed the decision threshold; and maintaining one or more third hypotheses and one or more fourth hypotheses.

[0154] Example 8. A method of any of the preceding examples, wherein if the stability score of one or more matches is higher than a threshold, then one or more matches are not validated in subsequent iterations, the stability score indicating the local tracking confidence and time uncertainty associated with each of the one or more matches.

[0155] Example 9. The method of any of the preceding examples, wherein the window of multiple scans for radar tracking and visual tracking includes radar data and visual data from the current sensor scan and at least one previous sensor scan.

[0156] Example 10. The method of Example 9, wherein the multi-scanning window moves forward one step in each iteration of at least one set of assumptions maintaining the association between visual tracking and radar tracking.

[0157] Example 11. The method of Example 9, wherein the multi-scan window is initialized with a single scan of radar tracking and visual tracking, and additional scans of radar tracking and visual tracking are accumulated in each iteration of at least one set of assumptions maintaining the association between visual tracking and radar tracking, until the multi-scan window reaches a predetermined window size.

[0158] Example 12. A method of any of the foregoing examples, the method further comprising: determining a confidence parameter and a plausibility parameter for each of at least one set of hypotheses based on a quality value of the association between visual tracking and radar tracking; and outputting the confidence parameter and plausibility parameter for one or more matches and one or more matches to a semi-autonomous or autonomous driving system of the vehicle to further update one or more matches and control the operation of the vehicle based on the confidence parameter and plausibility parameter of one or more matches.

[0159] Example 13. The method of Example 12, the method further includes combining quality values ​​of the association between visual tracking and radar tracking to generate at least one set of fused quality values ​​for each of the hypotheses, wherein a confidence parameter and a plausibility parameter are determined based on the fused quality values.

[0160] Example 14. The method of Example 13, wherein the Dempster-Schaffer fusion process is used to combine the quality values ​​of the correlation between visual tracking and radar tracking.

[0161] Example 15. A system comprising a processor configured to: acquire, for example, environmentally generated radar tracking and visual tracking of a vehicle, the radar tracking being generated based on radar data from one or more radar sensors on the vehicle, and the visual tracking being generated based on visual data from one or more visual sensors on the vehicle, the one or more radar sensors having a first field of view that at least partially overlaps with a second field of view of the one or more visual sensors; maintain, based on multiple scans of the radar tracking and visual tracking, at least one set of hypotheses of an association between the visual tracking and radar tracking, the at least one set of hypotheses including a quality value of the association between the visual tracking and radar tracking; determine, based on the quality value, a probability value for each of the at least one set of hypotheses, the probability value indicating the likelihood that each hypothesis is accurate; determine, based on the probability value of each hypothesis, one or more matches between the one or more radar tracking and visual tracking, the one or more matches including one or more hypotheses having a probability value exceeding a decision threshold; and output the one or more matches to a semi-autonomous or autonomous driving system of the vehicle to control the operation of the vehicle.

[0162] Example 16. The system of Example 15, wherein the processor is further configured to assign a quality value to the association between visual tracking and radar tracking based on at least one of the following: distance between tracks, similarity between tracks, heading information, bounding box information, object category information, velocity information, or object heading information.

[0163] Example 17. The system of Example 15 or 16, wherein at least one set of assumptions includes a first set of assumptions and a second set of assumptions, the first set of assumptions including at least one of the following: one or more radar tracks match visual tracks, and radar tracks do not match visual tracks, and the second set of assumptions including at least one of the following: visual tracks match one or more radar tracks, and visual tracks do not match radar tracks.

[0164] Example 18. The system of Example 17, wherein the processor is configured to determine one or more matches between one or more radar tracking and visual tracking by: extracting one or more first hypotheses from a first set of hypotheses having probability values ​​exceeding a decision threshold; extracting one or more second hypotheses from a second set of hypotheses having probability values ​​exceeding a decision threshold; and identifying one or more matches as hypotheses in which both one or more first hypotheses and one or more second hypotheses exceed the decision threshold.

[0165] Example 19. The system of Example 18, wherein the processor is further configured to: extract one or more third hypotheses having probability values ​​exceeding a significance threshold in response to a first set of hypotheses having no hypothesis exceeding a decision threshold, the significance threshold having a value lower than the decision threshold; extract one or more fourth hypotheses having probability values ​​exceeding a significance threshold in response to a second set of hypotheses having no hypothesis exceeding a decision threshold; maintain conflicting hypotheses among one or more first hypotheses and one or more second hypotheses that do not simultaneously exceed the decision threshold; and maintain significant hypotheses among one or more third hypotheses and one or more fourth hypotheses.

[0166] Example 20. A system comprising a processor configured to perform a method of any one of Examples 1 through 14.

[0167] Example 21. A computer-readable storage medium comprising computer-executable instructions that, when executed, cause a processor in a vehicle to perform a method of any one of Examples 1 to 14.

[0168] Example 22. A computer-readable storage medium comprising computer-executable instructions, which, when executed, cause a processor in a vehicle to: obtain radar tracking and visual tracking generated for the environment of the vehicle, the radar tracking being generated based on radar data from one or more radar sensors on the vehicle, and the visual tracking being generated based on visual data from one or more visual sensors on the vehicle, the one or more radar sensors having a first field of view that at least partially overlaps with a second field of view of the one or more visual sensors; maintain at least one set of hypotheses on the association between the visual tracking and the radar tracking based on multiple scans of the radar tracking and the visual tracking, the at least one set of hypotheses including a quality value of the association between the visual tracking and the radar tracking; determine a probability value for each of the at least one set of hypotheses based on the quality value, the probability value indicating the likelihood that each hypothesis is accurate; determine one or more matches between the one or more radar tracking and the visual tracking based on the probability value of each hypothesis, the one or more matches including one or more hypotheses having a probability value exceeding a decision threshold; and output the one or more matches to a semi-autonomous or autonomous driving system of the vehicle to control the operation of the vehicle.

[0169] Conclusion

[0170] While various examples of multi-scan sensor fusion for object tracking have been described in the foregoing description and illustrated in the accompanying drawings, it should be understood that this disclosure is not limited thereto, but can be implemented in different ways to be practiced within the scope of the following claims. It will be apparent from the foregoing description that various modifications can be made without departing from the spirit and scope of this disclosure as defined by the following claims.

[0171] Unless the context explicitly states otherwise, the use of "or" and grammatically related terms indicates an unrestricted, non-exclusive alternative. As used herein, the phrase referring to "at least one" of a list of items means any combination of those items, including a single member. As an example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

Claims

1. A method, the method comprising: The radar tracking and visual tracking of a vehicle are generated based on radar data from one or more radar sensors on the vehicle, and the visual tracking is generated based on visual data from one or more visual sensors on the vehicle, the one or more radar sensors having a first field of view that at least partially overlaps with the second field of view of the one or more visual sensors. Based on multiple scans of the radar data and the visual data, at least one set of assumptions is maintained regarding the correlation between the visual tracking and the radar tracking, the at least one set of assumptions including a quality value of the correlation between the visual tracking and the radar tracking; Based on the quality value, a probability value is determined for each of the at least one set of hypotheses, the probability value indicating the likelihood that each hypothesis is an accurate correlation between the visual tracking and the radar tracking; Based on the probability value of each hypothesis, one or more matches between one or more radar tracking and visual tracking are determined, the one or more matches including one or more hypotheses with probability values ​​exceeding a decision threshold; as well as Output one or more matches to the semi-autonomous or autonomous driving system of the vehicle to control the operation of the vehicle.

2. The method of claim 1, further comprising: The quality value is assigned to the association between the visual tracking and the radar tracking based on at least one of the following: distance between tracking, similarity between tracking, heading information, bounding box information, object category information, velocity information, or object heading information.

3. The method as described in claim 2, characterized in that, The quality value is assigned to the association between the visual tracking and the radar tracking based on at least three of the following: distance between tracking, similarity between tracking, heading information, bounding box information, object category information, velocity information, or object heading information.

4. The method as described in claim 1, characterized in that, The at least one set of assumptions includes at least one of the following: one or more radar tracking matches visual tracking, visual tracking matches one or more radar tracking, radar tracking does not match visual tracking, or visual tracking does not match radar tracking.

5. The method as described in claim 1, characterized in that, The at least one set of assumptions includes a first set of assumptions and a second set of assumptions. The first set of assumptions includes at least one of the following: one or more radar tracking matches visual tracking, and radar tracking does not match visual tracking. The second set of assumptions includes at least one of the following: visual tracking matches one or more radar tracking, and visual tracking does not match radar tracking.

6. The method as described in claim 5, characterized in that, Determining one or more matches between one or more radar tracking and visual tracking includes: Extract one or more first hypotheses from the first set of hypotheses that have probability values ​​exceeding the decision threshold; Extract one or more second hypotheses from the second set of hypotheses that have probability values ​​exceeding the decision threshold; and The one or more matches are identified as hypotheses in which both the one or more first hypotheses and the one or more second hypotheses exceed the decision threshold.

7. The method of claim 6, further comprising: In response to the first set of hypotheses not having a hypothesis exceeding the decision threshold, one or more third hypotheses are extracted having a probability value exceeding a significance threshold, the significance threshold having a value lower than the decision threshold; In response to the fact that none of the second set of hypotheses exceeds the decision threshold, one or more fourth hypotheses with probability values ​​exceeding the significance threshold are extracted; Maintain conflicting hypotheses among the one or more first hypotheses and the one or more second hypotheses that do not simultaneously exceed the decision threshold; as well as Maintain the one or more third hypotheses and the one or more fourth hypotheses.

8. The method as described in claim 1, characterized in that, If the stability score of one or more matches is higher than a threshold, the one or more matches will not be validated in subsequent iterations. The stability score indicates the local tracking confidence and time uncertainty associated with each of the one or more matches.

9. The method as described in claim 1, characterized in that, The window for the multiple scans of the radar tracking and the visual tracking includes the radar data and the visual data from the current sensor scan and at least one previous sensor scan.

10. The method as described in claim 9, characterized in that, The window, which undergoes multiple scans, moves forward one step in each iteration of at least one set of assumptions that maintain the association between the visual tracking and the radar tracking.

11. The method as described in claim 9, characterized in that, The window of the multiple scans is initialized using a single scan of the radar tracking and the visual tracking, and additional scans of the radar tracking and the visual tracking are accumulated in each iteration of at least a set of assumptions maintaining the association between the visual tracking and the radar tracking, until the window of the multiple scans reaches a predetermined window size.

12. The method of claim 1, further comprising: Based on the quality value of the correlation between the visual tracking and the radar tracking, determine the confidence parameter and the reasonableness parameter of each hypothesis in the at least one set of hypotheses; as well as The vehicle's semi-autonomous or autonomous driving system outputs the credibility parameter and the rationality parameter for the one or more matches, as well as the one or more matches, to further update the one or more matches and control the operation of the vehicle based on the credibility parameter and the rationality parameter of the one or more matches.

13. The method of claim 12, the method further comprising combining the quality values ​​of the association between the visual tracking and the radar tracking to generate a fused quality value for each of the at least one set of hypothetical associations, the confidence parameter and the plausibility parameter being determined based on the fused quality value.

14. The method as described in claim 13, characterized in that, The Dempster-Schaffer fusion process is used to combine the quality values ​​associated with the visual tracking and the radar tracking.

15. A system comprising a processor configured to: The radar tracking and visual tracking of a vehicle are generated based on radar data from one or more radar sensors on the vehicle, and the visual tracking is generated based on visual data from one or more visual sensors on the vehicle, the one or more radar sensors having a first field of view that at least partially overlaps with the second field of view of the one or more visual sensors. Based on multiple scans of the radar tracking and the visual tracking, maintain at least one set of assumptions regarding the association between the visual tracking and the radar tracking, the at least one set of assumptions including a quality value of the association between the visual tracking and the radar tracking; Based on the quality value, a probability value is determined for each of the at least one set of hypotheses, the probability value indicating the likelihood that each hypothesis is an accurate correlation between the visual tracking and the radar tracking; Based on the probability value of each hypothesis, one or more matches between one or more radar tracking and visual tracking are determined, the one or more matches including one or more hypotheses with probability values ​​exceeding a decision threshold; as well as Output one or more matches to the semi-autonomous or autonomous driving system of the vehicle to control the operation of the vehicle.

16. The system as described in claim 15, characterized in that, The processor is also configured to assign the quality value to the association between the visual tracking and the radar tracking based on at least one of the following: distance between tracking, similarity between tracking, heading information, bounding box information, object category information, velocity information, or object heading information.

17. The system as claimed in claim 15, characterized in that, The at least one set of assumptions includes a first set of assumptions and a second set of assumptions. The first set of assumptions includes at least one of the following: one or more radar tracking matches visual tracking, and radar tracking does not match visual tracking. The second set of assumptions includes at least one of the following: visual tracking matches one or more radar tracking, and visual tracking does not match radar tracking.

18. The system as claimed in claim 17, characterized in that, The processor is configured to determine one or more matches between one or more radar tracking and visual tracking in the following manner: Extract one or more first hypotheses from the first set of hypotheses that have probability values ​​exceeding the decision threshold; Extract one or more second hypotheses from the second set of hypotheses that have probability values ​​exceeding the decision threshold; as well as The one or more matches are identified as hypotheses in which both the one or more first hypotheses and the one or more second hypotheses exceed the decision threshold.

19. The system as claimed in claim 18, characterized in that, The processor is also configured to: In response to the first set of hypotheses not having a hypothesis exceeding the decision threshold, one or more third hypotheses are extracted having a probability value exceeding a significance threshold, the significance threshold having a value lower than the decision threshold; In response to the fact that none of the second set of hypotheses exceeds the decision threshold, one or more fourth hypotheses with probability values ​​exceeding the significance threshold are extracted; Maintain conflicting hypotheses among the one or more first hypotheses and the one or more second hypotheses that do not simultaneously exceed the decision threshold; as well as Maintain the significant assumptions in the one or more third assumptions and the one or more fourth assumptions.

20. A computer-readable storage medium comprising computer-executable instructions, which, when executed, cause a processor in a vehicle to: The radar tracking and visual tracking for the vehicle's environment are obtained, the radar tracking being generated based on radar data from one or more radar sensors on the vehicle, and the visual tracking being generated based on visual data from one or more visual sensors on the vehicle, the one or more radar sensors having a first field of view that at least partially overlaps with the second field of view of the one or more visual sensors. Based on multiple scans of the radar tracking and the visual tracking, maintain at least one set of assumptions regarding the association between the visual tracking and the radar tracking, the at least one set of assumptions including a quality value of the association between the visual tracking and the radar tracking; Based on the quality value, a probability value is determined for each of the at least one set of hypotheses, the probability value indicating the likelihood that each hypothesis is an accurate correlation between the visual tracking and the radar tracking; Based on the probability value of each hypothesis, one or more matches between one or more radar tracking and visual tracking are determined, the one or more matches including one or more hypotheses with probability values ​​exceeding a decision threshold; as well as Output one or more matches to the semi-autonomous or autonomous driving system of the vehicle to control the operation of the vehicle.

Images