Obstruction constraints for resolving tracking from multiple types of sensors

Abstract

This document describes techniques for resolving tracking from multiple types of sensors using occlusion constraints. In aspects, occlusion constraints are applied to associations between radar tracking and visual tracking to indicate an occlusion probability. In other aspects, techniques are described for a vehicle to avoid evaluating occluded radar tracking and visual tracking collected by a perception system. The occlusion probability is used to fade radar tracking and visual tracking pairs that have a high likelihood of being occluded and are therefore useless for tracking. The disclosed techniques can provide improved perception data that more tightly represents multiple complex data sets of a vehicle for preventing the vehicle from colliding with occluded objects while operating in an environment.

Description

Background Technology

[0001] In some vehicles, perception systems can combine information from multiple sensors (e.g., azimuth, vehicle speed, yaw rate, position) to support autonomous or semi-autonomous control. The combined sensor data can be used to estimate objects of interest present within the field of view (FOV). Position, velocity, trail, size, class, and other parameters can be inferred for each tracked object; the accuracy of these inferences improves when different types of sensors are used. Correlating inputs from multiple sensors for object tracking purposes can be computationally complex, often requiring sophisticated and expensive processing hardware to handle collision results. For safe and precise vehicle control to avoid object collisions, perception systems are expected to be able to analyze large amounts of sensor data accurately and efficiently. Summary of the Invention

[0002] This document describes techniques for resolving tracking from multiple types of sensors using occlusion constraints. Systems, methods, and apparatuses based on these techniques enable perception systems to modify radar and visual tracking data used for object tracking by using occlusion constraints that are diluted from consideration as occluded radar and visual tracking pairs, to improve the speed and accuracy of identifying objects within the field of view (e.g., based on undiminished radar and visual tracking pairs).

[0003] Among various aspects, a method for resolving occlusion constraints from tracking from multiple types of sensors is disclosed. This method includes determining, by a vehicle's perception system, whether occlusion conditions exist between visual tracking and radar tracking generated for an environment. In response to determining the existence of occlusion conditions between the visual tracking and radar tracking, occlusion constraints are applied to the association maintained between the visual tracking and the radar tracking. The occlusion constraints effectively modify the association maintained between the visual tracking and the radar tracking for considerations during tracking. After applying the occlusion constraints to the association maintained between the visual tracking and the radar tracking, the perception system determines objects in the environment at locations derived from another visual tracking and another radar tracking, based on another association maintained between another visual tracking and another radar tracking. The method includes outputting an indication of the object in the environment, which is used to track the object to control the vehicle or enable active safety functions of the vehicle to prevent collisions with the object while the vehicle is operating in the environment.

[0004] In other aspects, the system includes a processor configured to perform the method and other methods. A system having means for performing the method and other methods for resolving tracking from multiple types of sensors using occlusion constraints is disclosed.

[0005] The present invention is provided to introduce a simplified concept of techniques and apparatus for resolving occlusion constraints from tracking from various types of sensors, the concept of which is further described in the following detailed description and figures. Attached Figure Description

[0006] The following figures illustrate a technique for resolving tracking from multiple types of sensors using occlusion constraints. These figures are used throughout the text and some of the same reference numerals are used to refer to examples of similar features and components:

[0007] Figure 1 An example operating environment is shown for implementing occlusion constraints for resolving tracking from multiple types of sensors;

[0008] Figure 2 An example of the sensors and data used in parsing occlusion constraints from tracking from multiple types of sensors is shown;

[0009] Figure 3-1 An example scenario is shown for implementing sensors and data used in occlusion constraints for parsing tracking from multiple types of sensors;

[0010] Figure 3-2 Additional example scenarios are shown for implementing sensors and data used in occlusion constraints for parsing tracking from multiple types of sensors;

[0011] Figure 4 An example method is shown for resolving occlusion constraints in tracking from multiple types of sensors;

[0012] Figure 5-1 An example implementation is shown for performing a feasibility matrix for resolving occlusion constraints from tracking from multiple types of sensors;

[0013] Figure 5-2 Another example implementation is shown for performing the cost matrix for resolving occlusion constraints from tracking from multiple types of sensors;

[0014] Figure 6 Additional example methods are shown for resolving occlusion constraints in tracking from multiple types of sensors; and

[0015] Figure 7 An example method for resolving occlusion constraints from tracking from multiple types of sensors is shown. Detailed Implementation

[0016] Overview

[0017] This document describes techniques for resolving tracking data from multiple types of sensors using occlusion constraints. Vehicle perception systems face the challenge of quickly and accurately determining the correlations between multiple sets of sensor data or sensor readings (e.g., radar sensor readings, ultrasonic sensor readings, infrared camera sensor readings, optical or "visual" camera sensor readings, lidar sensor readings). Each sensor can collect information for identifying a single object, multiple objects, or a portion of a single object within the perception system's field of view (FOV). In congested FOVs (e.g., when many objects, including vehicles and pedestrian traffic, are present, or when the FOV covers a large space with a 360-degree field of view), perception systems may perform poorly due to the attempt to process large amounts of data. As described, the performance of perception systems can be improved by effectively using occlusion constraints to enhance or preprocess sensor data, enabling more efficient or faster object identification.

[0018] A vehicle's perception system processes vast amounts of data accurately and efficiently to navigate without colliding with moving objects in its environment. Multiple sensors (e.g., radar sensors, vision sensors) can send rapidly evolving data streams about the constantly changing environment. Problems can arise when the perception system is overloaded by too much sensor data queuing up for processing. The perception system may then provide delayed or inaccurate data, leading to object collisions.

[0019] When the occlusion constraints described in this paper are used to resolve tracking from multiple types of sensors, vehicles can process sensor data faster and with a higher degree of accuracy. By reducing some data in the tracking association from consideration based on occlusion constraints, the functional performance and scalability of the perception system are increased, as the perception system is able to quickly process even larger combinations of sensors or larger datasets.

[0020] Although the techniques and apparatuses described for resolving occlusion constraints from various types of sensors can be implemented in any number of different environments, the aspects described in the following examples are primarily used for controlling or assisting in the control of vehicles or enabling safety systems for vehicles.

[0021] Example Environment

[0022] Figure 1An example operating environment 100 of a vehicle 102 with a perception system 104 is shown, which is configured to perform occlusion constraints for resolving tracking from multiple types of sensors according to the techniques described herein. The illustrated perception system 104 may include a sensor interface 106 connected to multiple types of sensors for object tracking. As some examples, the sensor interface 106 includes a vision interface 106-1, a radar interface 106-2, and another sensor interface 106-n (e.g., a lidar interface, an ultrasonic interface, an infrared interface). The perception system 104 may also include a fusion module 108. The fusion module 108 may include a tracking-related data storage 110 for storing information obtained from the vision interface 106-1, the radar interface 106-2, and the other sensor interface 106-N. As shown, the perception system 104 is used to perceive the presence of another vehicle 112; however, the perception system 104 can be used to perceive any object (e.g., pedestrians, traffic cones, buildings).

[0023] In some aspects, the perception system 104 may include one or more sensors associated with the sensor interface 106. In other examples, the perception system 104 is operatively linked to sensors that may be part of other systems of the vehicle 102. For example, the perception system may include a vision sensor coupled to the vision interface 106-1, a radar sensor coupled to the radar interface 106-2, and an ultrasonic sensor, lidar sensor, infrared sensor, or any other type of sensor coupled to other sensor interfaces 106-N. Information obtained from the sensor interface 106 may be used to calculate speed, distance, and azimuth as factors in assessing the probability of occlusion, for example, to facilitate radar tracking.

[0024] The perception system 104 can be used to detect and avoid collisions with another vehicle 112. For example, the perception system 104 can use information obtained from the visual interface 106-1 and the radar interface 106-2, which, when processed by the fusion module 108, can be used to determine the presence of the vehicle 112 and take preventative measures to avoid a collision with the vehicle 112.

[0025] In another example, perception system 104 may combine further information obtained from other sensor interfaces 106-n (e.g., lidar interface, ultrasonic interface) with information obtained from vision interface 106-1 and radar interface 106-2, so that fusion module 108 can determine the presence of one or more occluded objects on road 114. For example, vehicle 102 may be traveling along road 114 and may use perception system 104 to determine the presence of another vehicle 112 that is occluded outside the field of view and take precautions to avoid a collision with vehicle 112.

[0026] Typically, the manufacturer can mount the sensing system 104 to any mobile platform traveling on road 114. The sensing system 104 can project its field of view (FOV) from any external surface of the vehicle 102. For example, the vehicle manufacturer can integrate at least a portion of the sensing system 104 into a side mirror, bumper, roof, or any other internal or external location (where the FOV includes road 114 and moving or stationary objects in the vicinity of road 114). The manufacturer can design the location of the sensing system 104 to provide an FOV that fully covers road 114 on which the vehicle 102 may travel. In the depicted implementation, a portion of the sensing system 104 is mounted near the front of the vehicle 102.

[0027] In several ways, occlusion constraints are applied to the association between radar and visual tracking, located in the tracking association data store 110, to indicate the probability of occlusion. In other ways, vehicle 102 can avoid evaluating occluded radar and visual tracking collected by perception system 104. The occlusion probability is used to reduce the association probability of radar and visual tracking pairs with a high probability of occlusion and therefore useless for tracking. The improved data collected by perception system 104 can more rigorously represent multiple complex datasets of vehicle 102 used to prevent collisions with occluded objects while the vehicle operates in environment 100. When occlusion constraints are used to resolve tracking from multiple types of sensors, perception system 104 can process sensor data faster and with a higher degree of accuracy. By de-emphasizing some data from the application of occlusion constraints from the considerations, the functional performance and scalability of perception system 104 are increased.

[0028] Figure 2 An example system 200 is shown, in which occlusion constraints for resolving tracking from multiple types of sensors of vehicle 102 can be implemented. Figure 2The example system 200 shown includes a controller 202, which includes a processor 204-1 and a computer-readable storage medium (CRM) 206-1 (e.g., memory, long-term storage, short-term storage) storing instructions for the vehicle module 208. The controller 202 and the sensing system 104-1 can communicate via a link 212. The link can be wired or wireless, and in some cases includes a communication bus. The controller 202 performs operations based on information received via the link 212, such as indications that one or more objects are traveling on road 114.

[0029] The perception system 104-1 includes a vision interface 106-1, a radar interface 106-2, a processor 204-1, a processor 204-2, and a CRM 206-2. The CRM 206-2 can store instructions associated with the fusion module 108-1, which can be executed by the processor 204-2 to perform operations of the fusion module 108. In addition to the tracking-associated data storage 110-2 and the modified tracking-associated data storage 110-2, the fusion module 108-1 may also include an occlusion submodule 210-1 and a k-optimal pattern search submodule 210-2 (also simply referred to as the k-optimal submodule 210-2).

[0030] Processors 204-1 and 204-2 can be two separate processing units or a single processing unit (e.g., a microprocessor, multiple processors within a processing unit) or a pair of single on-chip systems of a computing device, controller, or control unit. Processors 204-1 and 204-2 execute computer-executable instructions stored within CRM 206-1 and CRM 206-2.

[0031] Typically, the perception system 104-1 executes the fusion module 108-1 to perform vehicle functions (e.g., activating the braking system), which may include using outputs from the vision interface 106-1 and the radar interface 106-2 to track objects (e.g., objects in environment 100). For example, when the fusion module 108-1 is executed by the processor 204-2, the fusion module 108-1 may receive indications of occluded objects detected by the occlusion submodule 210-1 to achieve more accurate object tracking. See below for reference. Figure 3-1 and Figure 3-2 The example scenarios 300-1 and 300-2 shown are used to further describe system 200 in detail.

[0032] Figure 3-1Example scenario 300-1 is illustrated, in which occlusion constraints for parsing tracking data from various types of sensors of vehicle 302 can be implemented. Vehicle 302 including system 200 is shown, system 200 including perception system 104-1. In scenario 300-1, perception system 104-1 is occluded by truck 304 and cannot visually detect a moving car 306. If the information obtained at the vision interface 106-1 of perception system 104-1 of vehicle 302 is modified, processor 204-2 may be unable to detect the moving car 306 using the information obtained from vision interface 106-1 (e.g., via a vision sensor). However, if the information obtained at the radar interface 106-2 of perception system 104-1 of vehicle 302 is modified, processor 204-2 can detect the moving car 306 using the information obtained from radar interface 106-2 (e.g., via a radar sensor).

[0033] For illustrative purposes, the vision interface 106-1 of the perception system 104-1 on vehicle 302 (e.g., using a vision sensor) detects visual tracking 308, while the radar interface 106-2 of the perception system 104-1 (e.g., using a radar sensor) detects first radar tracking 310, second radar tracking 312, third radar tracking 314, and fourth radar tracking 316. Visual tracking 308 cannot be used to detect car 306 because car 306 is obscured from view by truck 304. Second radar tracking 312, third radar tracking 314, and fourth radar tracking 316 can be used to detect car 306, but not truck 304. In various aspects, determining an object (e.g., a moving car 306) in the environment at a location derived from visual tracking 308 and radar tracking 310, 312, 314, 316 may include: determining the presence of a moving car 306 detected by radar sensors 312, 314, 316 and obscured from the view of vision sensor 308.

[0034] Figure 3-2 Further shown Figure 3-1In an example scenario, occlusion constraints for resolving tracking from multiple types of sensors on vehicle 302 can be implemented. Vehicle 302 also includes system 200, which includes perception system 104-1 that detects visual tracking 308-1, first radar tracking 310-1, second radar tracking 312-1, third radar tracking 314-1, and fourth radar tracking 316-1. A first distance 318 is shown between the positions indicated by visual tracking 308-1 and first radar tracking 310-1. A second distance 320 is shown between the positions indicated by visual tracking 308-1 and second radar tracking 312-1. A third distance 322 is shown between the positions indicated by visual tracking 308-1 and third radar tracking 314-1. A fourth distance 324 is shown between the positions indicated by visual tracking 308-1 and fourth radar tracking 316-1.

[0035] The perception system 104-2 utilizes a computer-readable storage medium 206-2 storing instructions for executing the fusion module 108-1 to calculate the distance (e.g., 318, 320, 322, 324) and occlusion probability between each visual tracking and radar tracking (referred to as "each tracking pair"). In various aspects, the fusion module 108-1 includes an occlusion submodule 210-1 and a k-optimal submodule 210-2, as well as a tracking association data store 110-1 containing associations between visual tracking (e.g., 308-1) and radar tracking (e.g., 310-1, 312-1, 314-1, 316-1). The occlusion submodule 210-1 implements occlusion constraints on the tracking association data within the tracking association data store 110-1 within a probabilistic framework, resulting in a modified tracking association data store 110-2. The k-optimal submodule 210-2 uses the modified tracking association data store 110-2 to obtain the optimal set of hypotheses / patterns. This helps to avoid running the k-optimal submodule 210-2 or any other pattern search algorithm again when the optimal pattern contains pairings that do not meet the occlusion condition.

[0036] When multiple radar tracks are competing candidates to be associated with a particular visual track, the occlusion submodule 210-1 can perform occlusion inference. In this case, it may occur that a radar track located closer to the detected object is completely or partially occluded by one or more radar tracks farther from the visual track but closer to the ego vehicle. In this scenario, the visual track is more likely to be associated with the occluding radar track than with the relatively closer occluded radar track. The occlusion probability between the visual track and the radar track can be derived from several factors, not just distance, including the total number of occluding radar tracks for a given radar track, the amount of overlap between the occluding radar tracks and the given track, the azimuth uncertainty of both the occluding and occluded radar tracks, and the probability that each occluding radar track is associated with other time tracks.

[0037] In each aspect, the occlusion probability of each in radar tracking and visual tracking is calculated using the cost matrix. For example, Figure 2 The tracking association data storage 110-1 may include associated visual radar tracking pairs (e.g., visual tracking 308 paired with radar tracking 310, visual tracking 308 paired with radar tracking 312, visual tracking 308 paired with radar tracking 314, and visual tracking 308 paired with radar tracking 316). The modified tracking association data storage 110-2 may include the association cost of the visual radar tracking pairs that has been modified according to occlusion constraints. In another aspect, the calculated occlusion probability may be modified based on the scene and the attributes of a set of sensors. For example, fusion modules 108 and 108-1 may also rely on other information to infer whether occlusion exists between the tracking pairs.

[0038] In some aspects, factors driving the occlusion probability include the amount of angular overlap. For example, the occlusion probability is determined based on the total number of radar tracks (e.g., 310-1, 312-1, 314-1, 316-1) that at least partially overlap with the occluded radar track in the azimuth angle for a particular visual track (e.g., 308-1) and the occluded radar track, and the amount of angular overlap associated with the total number of radar tracks that at least partially overlap with the occluded radar track in the azimuth angle. More occluded radar tracks make it increasingly difficult to perform feasible visual tracking matching. When obtaining the occlusion probability, the azimuth overlap between the occluded radar tracks (e.g., 312-1, 314-1, 316-1) and the occluded radar track (e.g., 310-1) is taken into account.

[0039] In the example, the calculated factor can include the extension angles of all obstructing radar tracks, and the portion overlapping with the extension angles of the obstructed radar tracks can be further calculated. This probability can then be combined with multiple obstructing radar tracks to determine the obstruction penalty through multiplication.

[0040] The perception system 104-1 communicates with the controller 202 via link 212. The controller 202 includes a processor 204-1 and a computer-readable storage medium 206-1, which stores instructions for executing vehicle module 208 capable of controlling or assisting in controlling vehicle 302 or enabling safety systems of vehicle 302. Figure 3-2 In the illustrated example, in order to enable the vehicle's safety system to avoid a collision with the obscured car 306, the sensing system 104-1 can transmit a modified tracking-associated data storage 110-2 to the controller 202 of the vehicle 302.

[0041] Figure 4 An example method 400 for resolving occlusion constraints from tracking data from multiple types of sensors, according to the techniques described herein, is illustrated. A perception system 104 receives sensor data associated with radar tracking data and visual tracking data 402. The perception system 104 generates an association based on distance proximity according to kinematic and attribute information between the radar tracking and visual tracking pairs 404, arranged as a cost matrix between the radar tracking data and the corresponding visual tracking data. Examples of kinematic information include the position of an object, the velocity of an object, the heading of an object, and the size of a particular tracking event. However, examples of attribute information include whether the object is classified as a pedestrian, vehicle, guardrail, stop sign, or some other object under consideration. More specifically, the cost matrix can be generated from the visual-radar tracking pairs located in the tracking association data storage 110-1. The perception system 104 is configured to calculate the occlusion probability for each associated radar tracking and visual tracking pair within the cost matrix 406. The occlusion probability can be calculated by the occlusion submodule 210-1. The occlusion submodule 210-1 can determine the occlusion probability based on the extension angle of the occlusion radar tracking and in combination with the number of occlusion radar tracks, so as to determine the occlusion penalty.

[0042] If the perception system 104 detects an occlusion probability higher than a threshold for the radar-to-visual tracking pair 408, then the perception system 104 applies an occlusion penalty to each occluded associated radar-to-visual tracking pair within the cost matrix 410. If the perception system 104 decides not to penalize the radar-to-visual tracking pair 408, it applies no penalty and retains the original cost in each occluded associated radar-to-visual tracking pair within the cost matrix 412. Including the occlusion penalty in the original cost reduces the probability that the radar-visual pair will be included in the optimal assumption generated by the k-optimal submodule 210-2. After modifying the associated cost of the radar-to-visual tracking pair according to the occlusion probability within the cost matrix, the perception system 104 can provide a revised cost matrix denoted by "A". The revised cost matrix can be stored in the modified tracking-associated data storage 110-2 and can be utilized by the controller 202 for implementing safety features of the vehicle.

[0043] Figure 5-1 An example 500 of a first feasibility matrix 502 is shown. The first feasibility matrix 502 includes a list of possible radar tracking candidates for each visual tracking 508 to indicate whether a pairing is feasible. The feasibility matrix can be represented two-dimensionally, with a first dimension for visual tracking 504 and a second dimension for radar tracking 506. The first feasibility matrix 502 includes rows corresponding to visual tracking 504 and columns corresponding to radar tracking 506. The perception system 104 determines whether an occlusion probability exists in the list of feasible radar candidates for visual tracking and decides whether to penalize the radar tracking and visual tracking pair 508. Based on the occlusion condition check, the perception system 104 can apply an additional penalty to the previously estimated cost of the pair and update the matrix.

[0044] Figure 5-2 Example 500-1 of cost matrix 512-1 is shown, representing the association between radar tracking and visual tracking. After incorporating an occlusion penalty, cost matrix 512-1 can be transformed 514 into a second cost matrix 516. In various aspects, transformation 514 includes applying occlusion constraints to the association maintained between visual tracking and radar tracking by including the occlusion penalty in specific elements of the cost matrix, since the association maintained between visual tracking and radar tracking is evidence of not using visual tracking to track objects in the environment.

[0045] For example, for a given visual track "j", one of its feasible radar candidates "i" is blocked by a number of other radar tracks of the Nocc value. Here, "K" is an adjustment parameter, and the penalty for blocking can be calculated as follows:

[0046]

[0047] in,

[0048]

[0049]

[0050]

[0051]

[0052] In some examples, this transformation can modify radar tracking and visual tracking matching with a high occlusion probability. Cost matrix 512-1 can utilize information from a lookup table (not shown). This lookup table maintains a list of possible occluded radar tracks, along with several occluded tracks for each radar track based on occlusion inference.

[0053] Figure 6 A method is shown, in which Figure 4 The revised cost matrix "A" is processed by the k-optimal algorithm. The k-optimal algorithm can begin a pattern search for "A", where n = 1606. The k-optimal algorithm searches for the next optimal pattern 608. A determination step 610 occurs, where if "n" is less than "k", then n = n + 1612, and the search for the next optimal pattern 608 is repeated. However, if "n" is not less than "k", the pattern search ends 614. The k-optimal algorithm can be performed in O(kN) time. 3 The computation takes time complexity , where "k" represents the number of optimal modes to be maintained and "N" represents the number of feasible radar and visual tracks. By using occlusion constraints and resolving multiple sets of tracks, the revised cost matrix is ​​obtained at "A" in Figure 5, such that... Figure 6 The execution time of the k-optimal algorithm shown remains constant, thus allowing the perception system to report objects faster and more accurately. It would be impossible to maintain a fixed execution time if the cost matrix were not revised / modified according to each occlusion constraint. In such cases, after each pattern is generated by the k-optimal pattern search method (608), occlusion constraints need to be detected for that pattern. And if the occlusion constraint is not satisfied, the pattern needs to be discarded and the pattern search method needs to be rerun.

[0054] For example, processing using the k-optimal algorithm increases the functional performance and scalability of data processing by reducing the time complexity of the cost matrix typically used for object tracking based on sensor fusion, even as the number of possible tracking pairs continues to increase.

[0055] Example Method

[0056] Figure 7An example method 700 is illustrated for applying occlusion constraints to parsing tracking from multiple types of sensors. In this example, the method determines whether occlusion conditions exist and penalizes radar and visual tracking matches with a high occlusion probability. This method can further process those radar and visual tracking matches penalized as a preprocessing step to reduce the processing time of the perception system 104.

[0057] At 702, the vehicle's perception system 104 determines whether occlusion conditions exist between the visual tracking and radar tracking generated for the environment. In one example, the perception system 104 collects radar and visual data to identify objects in the field of view. In another example, the perception system 104 determines the presence of occlusion conditions based on both visual and radar tracking.

[0058] At 704, in response to determining that an occlusion condition exists between visual tracking and radar tracking, an occlusion constraint is applied to the association maintained between visual tracking and radar tracking. This occlusion constraint effectively modifies the association cost maintained between visual tracking and radar tracking. For example, the feasibility of the association is formed when radar tracking and visual tracking meet some proximity criteria, and the aggregation of feasible radar tracking for each visual tracking is represented as a feasibility matrix. In another example, aggregations of radar tracking and visual tracking pairs with high occlusion probabilities are penalized based on the occlusion probability. In yet another example, the penalized radar tracking and visual tracking association cost is modified.

[0059] At 706, after occlusion constraints have been applied to the association costs maintained between visual and radar tracking, perception system 104 determines objects in the environment at locations derived from one visual track and another radar track, based on another association maintained between another visual track and another radar track. For example, the association between radar and visual track matches (e.g., cost matrices) has been diluted for those matches with high occlusion probabilities, and perception system 104 then processes the available revised data to identify occluded and undisturbed objects.

[0060] At 708, the perception system 104 outputs an indication of an object in the environment, which is used to track the object to control the vehicle or activate the vehicle's active safety features to prevent the vehicle from colliding with the object while operating in the environment. For example, the perception system 104 identifies an obstructed object on a trajectory that will collide with the vehicle, and thereby controls the vehicle to activate its braking system to prevent a collision with the obstructed object.

[0061] Typically, any of the components, modules, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Some operations of the example methods can be described in the general context of executable instructions stored on a computer-readable storage medium that is local and / or remote relative to a computer processing system, and implementations may include software applications, programs, functions, etc. Any functionality described herein may be performed at least in part by one or more hardware logic components, including but not limited to field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), etc.

[0062] Further examples

[0063] The following are some further examples:

[0064] Example 1: A method comprising: determining, by a perception system of a vehicle, whether an occlusion condition exists between visual tracking and radar tracking generated for an environment; in response to determining that an occlusion condition exists between the visual tracking and radar tracking, applying an occlusion constraint to an association maintained between the visual tracking and radar tracking, the occlusion constraint effectively modifying the association maintained between the visual tracking and radar tracking; after applying the occlusion constraint to the association maintained between the visual tracking and radar tracking, determining, by the perception system, an object in the environment at a location derived from another visual tracking and another radar tracking based on another association maintained between another visual tracking and another radar tracking; and outputting an indication of the object in the environment for tracking the object to control the vehicle or enable active safety features of the vehicle to prevent collisions with the object while the vehicle is operating in the environment.

[0065] Example 2: The method of the foregoing example, wherein determining whether an occlusion condition exists includes: using the azimuth uncertainty of the occlusion radar track and the occluded radar track to determine whether either of the occlusion radar tracks at least partially overlaps with the visual track and the occluded radar track in the azimuth; and in response to determining with sufficient azimuth uncertainty that at least one of the occlusion radar tracks overlaps with at least one of the visual track and the occluded radar track in the azimuth, determining that an occlusion condition exists between the visual track and the occluded radar track.

[0066] Example 3: Any of the methods in the foregoing examples further includes: determining an occlusion probability between radar tracking and visual tracking, the occlusion probability indicating the likelihood that visual tracking represents a different object from each radar tracking, wherein determining whether an occlusion condition exists between visual tracking and radar tracking includes: determining the occlusion probability between radar tracking and visual tracking as evidence that visual tracking is not used to track objects in the environment.

[0067] Example 4: Any of the methods in the foregoing examples further includes: determining the occlusion probability based on the total amount of occluded radar tracks that at least partially overlap with the occluded radar tracks and visual tracks in the azimuth angle and the amount of angular overlap associated with the total amount of radar tracks that at least partially overlap with the occluded radar tracks in the azimuth angle.

[0068] Example 5: Any of the methods in the foregoing examples further includes: generating a cost matrix, the cost matrix including associations maintained between visual tracking and radar tracking as specific elements of the cost matrix, each other element of the cost matrix representing another association between a unique pair of radar tracking and visual tracking; and determining occlusion penalties for radar tracking and visual tracking based on occlusion probabilities, wherein applying occlusion constraints to the associations maintained between visual tracking and radar tracking includes: adding occlusion penalties to specific elements of the cost matrix of the associations maintained between visual tracking and radar tracking to prevent the use of visual tracking to track objects in the environment.

[0069] Example 6: Any of the methods in the preceding examples, wherein applying occlusion constraints to the association maintained between visual tracking and radar tracking includes increasing the cost of that pair in the cost matrix to dilute the association maintained between the visual tracking and the radar tracking.

[0070] Example 7: A method from any of the preceding examples, where each element of the cost matrix represents the distance between the time state and the radar state associated with a unique pair of radar tracking and visual tracking, and an occlusion constraint is applied to penalize some elements.

[0071] Example 8: Any of the methods in the foregoing examples further includes: each element of the cost matrix represents the association between a specific visual tracking and a specific radar tracking as the distance between the specific visual tracking and the specific radar tracking; the distance between radar tracking and visual tracking corresponds to the cost value of a specific element in the cost matrix; and the distance between radar tracking and visual tracking affects the occlusion probability.

[0072] Example 9: The method of any of the foregoing examples further includes: each element of the cost matrix represents the association between a particular visual track and a particular radar track as the distance between the locations of the particular visual track and the particular radar track, and the particular radar track corresponds to the cost value of a particular element in the cost matrix; and generates an occlusion probability for each element of the cost matrix, the occlusion probability being based on one or more occluded radar tracks and the azimuth overlap between the occluded radar track and the occluded radar track.

[0073] Example 10: Any of the methods in the preceding examples, wherein determining an object in the environment at a location derived from another visual tracking and another radar tracking includes: identifying a moving vehicle that is occluded outside the field of view of the visual sensor; and the moving vehicle is detected by the radar sensor.

[0074] Example 11: Any of the methods in the foregoing examples further includes: modifying one or more elements, which represent radar tracking with corresponding one or more visual tracking; and providing a revised cost matrix.

[0075] Example 12: Any of the methods in the preceding examples, wherein modifying the cost matrix includes: reducing the complexity of the k-optimal algorithm data input to the k-optimal algorithm by penalizing the association cost of the pair under occlusion constraints, so that the pair appears in the k-optimal pattern with a lower probability; and the k-optimal algorithm has a time complexity of O(kN) 3 It is executed in time complexity, where "k" represents the number of optimal patterns to be maintained and "N" represents the number of feasible radar or visual tracks.

[0076] Example 13: Any of the methods in the preceding examples, wherein the modification from the cost matrix includes: reducing the complexity in the k-optimal algorithm so that the given execution time remains constant as the number of patterns increases.

[0077] Example 14: Any of the methods in the preceding examples, wherein providing the revised cost matrix includes: fading at least one track from a set of tracks in response to determining the occlusion probability of that set of tracks; and fading at least one track from a set of tracks includes: retaining at least one track for matching with different tracks from the set of tracks.

[0078] Example 15: A method of any of the preceding examples, where a lookup table maintains a list of possible occluded radar tracks, along with potential visual tracking pairs for each radar track based on occlusion inference.

[0079] Example 16: Any of the methods in the preceding examples, where the occlusion probability calculated by the vehicle's perception system can be modified based on the scene and the properties of a set of sensors.

[0080] Example 17: Any of the methods in the preceding examples, where a set of sensors is used to calculate the following tracking attributes: velocity; distance; and azimuth; where velocity, distance, and azimuth are factors used to evaluate the probability of occlusion.

[0081] Example 18: Any of the methods in the preceding examples, wherein a set of sensors includes one or more radar sensors, lidar sensors, or ultrasonic sensors, and a set of sensors includes one or more vision sensors.

[0082] Example 19: A system comprising: a processor configured to execute the methods described in the preceding example.

[0083] Example 20: A system comprising: means for performing any of the methods described in the preceding examples.

[0084] Conclusion

[0085] Although aspects of resolving occlusion constraints from tracking from multiple types of sensors have been described in language specific to certain features and / or methods, the subject matter of the appended claims is not necessarily limited to the specific features or methods described. Rather, specific features and methods are disclosed as example implementations of the stated methods for resolving occlusion constraints from tracking from multiple types of sensors, and other equivalent features and methods are also intended to fall within the scope of the appended claims. Furthermore, various aspects have been described, and it should be understood that each described aspect may be implemented independently or in combination with one or more other described aspects.

Claims

1. A method for a means of transport, the method comprising: The perception system of the vehicle determines, based on data received from radar and visual sensors, whether there is an occlusion condition between visual tracking and radar tracking generated for the environment, wherein determining whether an occlusion condition exists includes: The azimuth uncertainty of the obstructing radar track and the obstructed radar track is used to determine whether either of the obstructing radar tracks at least partially overlaps with one of the visual track and the obstructed radar track in azimuth, wherein the obstructing radar track corresponds to a first object also detected by the visual sensor, and the obstructed radar track corresponds to a second object obstructed from the visual sensor by the first object. In response to determining with sufficient azimuth angular certainty that at least one of the obstructing radar tracks overlaps azimuthally with one of the visual tracks and at least one of the obstructed radar tracks, it is determined that the obstruction condition exists between one of the visual tracks and at least one of the obstructed radar tracks, and Determine the occlusion probability of each of the radar tracks and one of the visual tracks, the occlusion probability indicating the probability that one of the visual tracks represents a different object from each of the radar tracks, and evidence that one of the visual tracks is not used to track objects in the environment, wherein the occlusion probability is based on the total number of occluded radar tracks that at least partially overlap with at least one of the occluded radar tracks and one of the visual tracks in azimuth angle, and the angular overlap associated with the total number of radar tracks that at least partially overlap with at least one of the occluded radar tracks in azimuth angle; In response to determining that an occlusion condition exists between one of the visual tracks and one of the radar tracks, an occlusion constraint is applied to the association maintained between the two tracks, which effectively modifies the association maintained between them. The application of the occlusion constraint includes: A cost matrix is ​​generated, comprising the correlation maintained as a specific element between one of the visual tracking and one of the radar tracking, wherein each other element of the cost matrix represents another correlation between the radar tracking and the visual tracking pair. Based on the occlusion probability, an occlusion penalty is determined for one of the radar tracks and one of the visual tracks, and The occlusion penalty is added to a specific element of the cost matrix used to maintain the association between one of the visual tracks and one of the radar tracks, to prevent the use of one of the visual tracks to track objects in the environment; After applying the occlusion constraint to the association maintained between one of the visual tracks and one of the radar tracks, the perception system determines, based on another association maintained between the other visual track and the other radar track, the object in the environment at a location derived from the other visual track and the other radar track; and Output an indication of the object in the environment, which is used to track the object to control the vehicle or enable the vehicle's active safety features to prevent the vehicle from colliding with the object while operating in the environment.

2. The method as described in claim 1, characterized in that Applying the occlusion constraint to the association maintained between one of the visual tracks and one of the radar tracks includes increasing the cost of the pair in the cost matrix to weaken the association maintained between the one of the visual tracks and one of the radar tracks.

3. The method as described in claim 1, characterized in that Each element of the cost matrix represents the distance between the time state and the radar state associated with the pair of radar tracking and visual tracking, and some elements are penalized due to the application of the occlusion constraint.

4. The method of claim 1, further comprising: Each element of the cost matrix represents the association between a specific visual tracking and a specific radar tracking as the distance between the states of each of the specific visual tracking and the specific radar tracking; The distance between one of the radar tracks and one of the visual tracks corresponds to the cost value of a specific element in the cost matrix; and The distance between one of the radar trackers and one of the visual trackers affects the occlusion probability.

5. The method of claim 1, further comprising: Each element of the cost matrix represents the association between a specific visual track and a specific radar track as the distance between the positions of each of the specific visual track and the specific radar track, and the specific radar track corresponds to the cost value of a specific element in the cost matrix; and The occlusion probability of each element of the cost matrix is ​​generated, the occlusion probability being based on one or more occlusion radar tracks and the azimuth overlap between the occlusion radar track and the occluded radar track.

6. The method of claim 1, wherein, Determining objects in the environment at locations derived from the other visual tracking and the other radar tracking includes: identifying moving vehicles obscured outside the field of view of the visual sensors; and The mobile vehicle is detected by the radar sensor.

7. The method of claim 1, further comprising: Modify one or more elements from the cost matrix, where the one or more elements represent radar tracking with corresponding one or more visual tracking; as well as Provides a revised cost matrix.

8. The method of claim 7, wherein, Modifying the cost matrix includes: reducing the complexity of the k-optimal algorithm data input into the k-optimal algorithm by penalizing the association cost of pairs under occlusion constraints, making the pairs appear in k-optimal patterns with a lower probability; and The k-optimal algorithm has a time complexity of O(kN) 3 It is executed in time complexity, where k represents the number of optimal patterns to be maintained, and N represents the number of feasible radar or visual tracks.

9. The method of claim 8, wherein, Modifications to the cost matrix include reducing the complexity of the k-optimal algorithm so that the given execution time remains constant as the number of patterns increases.

10. The method as described in claim 7, characterized in that, Providing the revised cost matrix includes: In response to determining the occlusion probability of a set of tracks, at least one track from said set of tracks is faded; and Diminishing the at least one track from a set of tracks includes: retaining the at least one track for matching with different tracks from the set of tracks.

11. The method as described in claim 7, characterized in that, The lookup table maintains a list of possible occlusion radar tracks, along with potential visual tracking pairs for each of the radar tracks based on occlusion inference.

12. The method as described in claim 11, characterized in that, The occlusion probability calculated by the perception system of the vehicle can be modified based on the scene and the attributes of a set of sensors.

13. The method as described in claim 12, characterized in that, The set of sensors was used to calculate the following tracking properties: speed; distance; Azimuth angle; and The speed, distance, and azimuth angle are used as factors in assessing the probability of occlusion.

14. The method as described in claim 13, characterized in that, The set of sensors includes one or more radar sensors, lidar sensors, or ultrasonic sensors, and the set of sensors includes one or more vision sensors.

15. A system for a vehicle, the system comprising a processor configured to: The perception system of the vehicle determines, based on data received from radar and visual sensors, whether there is an occlusion condition between visual tracking and radar tracking generated for the environment, wherein determining whether an occlusion condition exists includes: The azimuth uncertainty of the obstructing radar track and the obstructed radar track is used to determine whether either of the obstructing radar tracks at least partially overlaps with one of the visual track and the obstructed radar track in azimuth, wherein the obstructing radar track corresponds to a first object also detected by the visual sensor, and the obstructed radar track corresponds to a second object obstructed from the visual sensor by the first object. In response to determining with sufficient azimuth angular certainty that at least one of the obstructing radar tracks overlaps azimuthally with one of the visual tracks and at least one of the obstructed radar tracks, it is determined that the obstruction condition exists between one of the visual tracks and at least one of the obstructed radar tracks, and Determine the occlusion probability of each of the radar tracks and one of the visual tracks, the occlusion probability indicating the probability that one of the visual tracks represents a different object from each of the radar tracks, and evidence that one of the visual tracks is not used to track objects in the environment, wherein the occlusion probability is based on the total number of occluded radar tracks that at least partially overlap with at least one of the occluded radar tracks and one of the visual tracks in azimuth angle, and the angular overlap associated with the total number of radar tracks that at least partially overlap with at least one of the occluded radar tracks in azimuth angle; In response to determining that an occlusion condition exists between one of the visual tracks and one of the radar tracks, an occlusion constraint is applied to the association maintained between the two tracks, which effectively modifies the association maintained between them. The application of the occlusion constraint includes: A cost matrix is ​​generated, comprising the correlation maintained as a specific element between one of the visual tracking and one of the radar tracking, wherein each other element of the cost matrix represents another correlation between the radar tracking and the visual tracking pair. Based on the occlusion probability, an occlusion penalty is determined for one of the radar tracks and one of the visual tracks, and The occlusion penalty is added to a specific element of the cost matrix used to maintain the association between one of the visual tracks and one of the radar tracks, to prevent the use of one of the visual tracks to track objects in the environment; After the occlusion constraint has been applied to the association maintained between one of the visual tracks and one of the radar tracks, the perception system determines the object in the environment at a location derived from the other visual track and the other radar track, based on another association maintained between the other visual track and the other radar track; and Output an instruction for the object in the environment, the instruction being used to track the object to control the vehicle or to enable the vehicle's active safety features to prevent the vehicle from colliding with the object while operating in the environment.

16. A system for a vehicle, the system comprising: A means for a perception system of the vehicle to determine, based on data received from radar and visual sensors, whether there is an occlusion condition between visual tracking and radar tracking generated for the environment, wherein determining whether an occlusion condition exists includes: The azimuth uncertainty of the obstructing radar track and the obstructed radar track is used to determine whether either of the obstructing radar tracks at least partially overlaps with one of the visual track and the obstructed radar track in azimuth, wherein the obstructing radar track corresponds to a first object also detected by the visual sensor, and the obstructed radar track corresponds to a second object obstructed from the visual sensor by the first object. In response to determining with sufficient azimuth angular certainty that at least one of the obstructing radar tracks overlaps azimuthally with one of the visual tracks and at least one of the obstructed radar tracks, it is determined that the obstruction condition exists between one of the visual tracks and at least one of the obstructed radar tracks, and Determine the occlusion probability of each of the radar tracks and one of the visual tracks, the occlusion probability indicating the probability that one of the visual tracks represents a different object from each of the radar tracks, and evidence that one of the visual tracks is not used to track objects in the environment, wherein the occlusion probability is based on the total number of occluded radar tracks that at least partially overlap with at least one of the occluded radar tracks and one of the visual tracks in azimuth angle, and the angular overlap associated with the total number of radar tracks that at least partially overlap with at least one of the occluded radar tracks in azimuth angle; In response to determining that an occlusion condition exists between one in visual tracking and one in radar tracking, means for applying an occlusion constraint to an association maintained between the one in visual tracking and the one in radar tracking, the occlusion constraint effectively modifying the association maintained between the one in visual tracking and the one in radar tracking, wherein applying the occlusion constraint includes: A cost matrix is ​​generated, comprising the correlation maintained as a specific element between one of the visual tracking and one of the radar tracking, wherein each other element of the cost matrix represents another correlation between the radar tracking and the visual tracking pair. Based on the occlusion probability, an occlusion penalty is determined for one of the radar tracks and one of the visual tracks, and The occlusion penalty is added to a specific element of the cost matrix used to maintain the association between one of the visual tracks and one of the radar tracks, to prevent the use of one of the visual tracks to track objects in the environment; After applying the occlusion constraint to the association maintained between one of the visual tracks and one of the radar tracks, means for the perception system to determine, based on another association maintained between the other visual track and the other radar track, the location of an object in the environment derived from the other visual track and the other radar track; and A means for outputting an indication of an object in the environment, the indication being used to track the object to control the vehicle or to enable the vehicle's active safety features to prevent the vehicle from colliding with the object while operating in the environment.

Images