Radar tracking estimated using a model enhanced by radar detection.

Abstract

This document describes radar tracking that leverages model estimation enhanced by radar detection. Example tracker analysis uses information derived from radar detection to enhance radar tracking with object measurements estimated by directly analyzing a data cube using a model (e.g., a machine learning model). This model can quickly produce high-quality tracking that measures important objects. However, the model only estimates measurements of object categories it was trained or programmed to recognize. To improve the measurements estimated from the model, or even, in some cases, to convey additional object categories, the tracker analyzes the detections separately. Detections consistently aligned with objects recognized by the model can update the model-derived measurements initially conveyed in the tracking. Consistently observed detections that are not aligned with existing tracking can be used to build new tracking that conveys more object categories than the model was able to recognize.

Description

BACKGROUND

[0001] Perception systems for vehicles (e.g., advanced safety or autonomous driving systems) can rely on the output of radar trackers. Radar trackers for multiple-input-multiple-output (MIMO) radar systems can output radar tracks to objects derived from analysis of complex radar data (e.g., a datacube or a list of detections) that characterizes radar returns detected within multiple frames using a MIMO array. To improve responsiveness and speed of reporting tracks, radar trackers can evaluate only a portion of radar data and ignore the rest. For example, model-based trackers can quickly identify a particular class of objects (e.g., cars, trucks) by focusing only on certain portions of a datacube to only look for particular object types. However, even though some detections are not considered (e.g., eliminated as noise), conventional trackers can analyze detections inferred from most, if not all, of a datacube. In many common driving scenarios, perception systems that rely on radar trackers that are tuned for speed or that only identify a particular class of objects can fail to provide a vehicle with sufficient situational awareness, where there are often other classes of objects that are not reported and / or tracked. SUMMARY

[0002] This document describes techniques and systems for radar tracking with model estimates augmented by radar detections. In one example, a method includes establishing, by a radar system, a track for an object in an environment using information acquired from a model, the information acquired from the model including object measurements estimated from a radar datacube representation of signals received from the environment; determining, in addition to the information acquired from the model, whether a radar detection observed from the signals received from the environment corresponds to the same object as the track established using the information acquired from the model; and in response to determining that the radar detection corresponds to the same object as the track established using the information acquired from the model, augmenting a portion of the track initialized from the object measurements acquired from the model using additional information derived from the radar detection to improve accuracy or detail associated with the portion of the track.

[0003] In some examples, a method includes maintaining a primary track layer using information acquired from a model, the primary track layer including a track for an object established using the information acquired from the model in addition to any other tracks established using the information acquired from the model;

[0004] and in addition to using information obtained from the model, maintain a secondary tracking layer using additional information derived from a radar detection, additional information derived from another radar detection, and additional information derived from any other radar detection, the secondary tracking layer including: a track established using the additional information derived from the radar detection, another track established using the additional information derived from the another radar detection, and any other track established using the additional information derived from any other radar detection. In another example, a method includes: using a combination of a model-based radar tracker and a detection-based radar tracker to track objects in an environment of a vehicle based on signals received from the environment, the tracking including: maintaining a primary tracking layer including tracks established using information obtained from a model, the information including: object measurements estimated from a radar data-cube representation of the signals received from the environment; and maintaining a secondary tracking layer including: tracks established using additional information derived from radar detections identified from the signals received from the environment other than from the model and refraining from outputting any of the tracks from the secondary tracking layer until the additional information derived from the radar detections satisfies a quality threshold for performing detection-based augmentation of a portion of the tracks in the primary tracking layer; outputting the tracks in the primary tracking layer to enable the vehicle to avoid the objects while driving in the environment.

[0005] By implementing these or other examples contemplated by the present invention, effective and accurate radar processing can be achieved to track a wide variety of objects than can be tracked using other radar tracking techniques. This overview presents a brief concept of radar tracking with model estimates augmented by radar detections, such as examples of vehicles (e.g., trucks, cars) equipped with radar tracking to support driving and other object tracking, as explained further by the DETAILED DESCRIPTION and the accompanying drawings. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it used to determine or limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0006] Details of radar tracking with model estimates augmented by radar detections are described in this document with reference to the accompanying drawings, which can use identical reference numbers to refer to like features and components in different drawings, and which use a hyphenated number to indicate a variation of a feature or component from another instance of similar

[0007] Figure 1 An example environment of radar tracking with model estimates augmented by radar detections is shown;

[0008] Figure 2 An example vehicle including a system configured to perform radar tracking with model estimates augmented by radar detections is shown;

[0009] Figures 3-1 to 3-4 An example system is shown that is configured for radar tracking with model estimation augmented by radar detections;

[0010] Figure 4 An example system is shown that is configured for maintaining multiple tracking layers to facilitate radar tracking in order to support radar tracking with model estimation augmented by radar detections;

[0011] Figure 5 A flow diagram of an example process for radar tracking with model estimation augmented by radar detections is shown.

[0012] Figure 6-1 And Figure 6-2 An additional flow diagram is shown that shows in more detail portions of an example process for radar tracking with model estimation augmented by radar detections;

[0013] Figure 7 A flow diagram of an example process for maintaining multiple tracking layers to facilitate radar tracking in order to support radar tracking with model estimation augmented by radar detections is shown; and

[0014] Figure 8 A flow diagram of an example process for operating a vehicle using radar tracking with model estimation augmented by radar detections is shown. DETAILED DESCRIPTION

[0015] INTRODUCTION

[0016] A perception system can rely on a radar tracker to provide situational awareness and detect objects of interest in an environment of a vehicle. A radar system includes a multiple-input multiple-output (MIMO) antenna array interfaced with a monolithic microwave integrated circuit (MMIC) that obtains radar returns from MIMO detections of objects and transmits these radar returns as radar data. From the radar data for a frame output from the MMIC, a radar processor can identify a detection list for the frame. Each detection contains information used to derive a location and a range rate for a radar reflected portion of an object.

[0017] A detection-based radar tracker can derive object measurements (e.g., position, size, orientation, velocity, acceleration) for an entire object based on detections of small portions of the object. Some detections can be discarded (e.g., deemed to be noise or deemed to be too low in energy) to reduce the amount of radar data that is processed; this can include applying thresholds and / or applying non-maximum suppression of energy attributed to detections, e.g., by finding peaks in the Doppler plane. This can improve tracking speed and cause tracks of objects with object measurements to appear more quickly in the radar output; this is a result of focusing on the remaining detections. Periodic sequences of radar tracks, including object measurements, are output in response to grouped detections that can be attributed to the same object. Each of the radar tracks represents a different object from a group of objects that are observable by grouping detections. Each radar track has many fields for conveying object measurements; some fields can not be used until object measurements associated with the object can be inferred from subsequent detection frames.

[0018] Some radar processors convert the output of the MMIC to a more detailed representation of the radar returns than is represented with a list of detections. The output of the MMIC can be converted to a data structure called a data cube. The data cube organizes the radar data for each frame obtained from the MIMO array into a two-dimensional (2D) data cube representation of that radar data, with a distance-Doppler plane for each receive channel of the MIMO array. This enables the formation of a three-dimensional (3D) data cube representation of the MMIC output into a one-dimensional effective receive array, or a four-dimensional (4D) data cube representation of the MMIC output into a 2D effective receive array. Some radar processors can perform additional processing on the receive channels for the additional inclusion of 1D angle response measurements (e.g., azimuth or elevation) or 2D angle response measurements (e.g., azimuth and elevation) into the data cube. In addition to forming the data cube, detections can also be determined, and in some cases, the detections are derived from the formation of the data cube. In either case, this is a pre-processing step that is required before traditional detection-based tracking can occur. With advances in computing technology, model-based trackers can instead directly analyze these complex data cube collections to identify specific classes of objects and can estimate some object measurements more quickly than can be estimated from object measurements estimated using detections.

[0019] Model-based trackers can track objects by directly processing data cubes without the need to analyze individual detections as detection-based trackers do. For example, a model-based tracker can include a machine learning network trained to analyze a sequence of data cubes or a sparse subset thereof for identifying objects of a particular type or class in the environment, which is only a subset of all potential types and classes. Thus, the subset includes a number of objects that belong to the particular object class that the network is trained to recognize. The model-based tracker reports a periodic list of modeled measurements (e.g., the machine learning network periodically reports a list of machine learning measurements) that contain estimates of certain properties of the entire object discovered by the machine learning network by directly analyzing the data cubes. Some fields of the track generated by the model-based tracker can contain modeled measurements estimated by the machine learning network.

[0020] As more emphasis is placed on tracking directly from data cubes, detections are often ignored to ensure fast tracking, otherwise these detections can be used to support track generation and measurement estimation. Detections are discarded or ignored as during detection-based tracking, even when sophisticated models are used, objects of interest for safe driving in some environments can not be tracked and reported in the radar output. To have high tracking speed, this reduces situational awareness, accuracy, and safety. Invariantly, existing model-based and detection-based trackers discard or ignore portions of radar returns that can have useful information for radar tracking. Thus, some existing trackers fail to convey information that can otherwise be derived from the portions of radar returns that are ignored. If an autonomous driving or advanced safety application relies on the existing tracker, then an inaccurate or incomplete set of radar tracks can lead to unsafe, uncomfortable, or unstable driving behavior when the vehicle system relies on the tracker.

[0021] SUMMARY

[0022] This document describes radar tracking with model estimates augmented by radar detections. An example tracker analyzes information derived using radar detections to augment radar tracking with object measurements estimated by directly analyzing a datacube using a model (e.g., a machine learning model). Using the model can quickly produce high-quality tracking that measures important objects. However, the model only estimates measurements for object classes that its training or programming enables it to recognize. To improve the measurements estimated from the model, or even in some cases, to convey additional object classes, the tracker separately analyzes detections. Detections that align with objects recognized by the model can update model-derived measurements originally conveyed in the tracking. Detections that are observed to align with existing tracking can be used to establish new tracking to convey more object classes than the model is able to recognize.

[0023] In this way, an example radar tracker can perform efficient radar tracking to convey tracking of a wide variety of objects with high accuracy and speed than can be reported using other radar tracking techniques. Relying on the model’s ability to quickly report objects from a datacube, the model estimate augmentation quickly reports model-identifiable objects (e.g., vehicles, trucks, buses, motorcycles, or bicycles) that are most interesting in driving. Moreover, without introducing a delay in this initial object reporting, the model estimate augmentation facilitates more accurate object measurement reporting (e.g., speed) and / or additional object reporting. This can result in making more safe driving decisions than relying on model-based radar tracking or detection-based radar tracking alone. This high-speed improved situational awareness can prevent vehicles from colliding with almost any type of radar-detectable object appearing on or near a road, thereby improving driving safety in a wide variety of driving environments and situations where a wide variety of objects need to be avoided.

[0024] Example Environment

[0025] Figure 1 An example environment 100 is shown that utilizes radar tracking with model estimates augmented by radar detections. In the depicted environment 100, a vehicle 102 is driving on a road on which or near which a wide variety of objects are freely moving and impacting driving. Although shown as a passenger car, the vehicle 102 can represent other types of motor vehicles (e.g., a car, a motorcycle, a bus, a tractor, a semi-truck), non-motor vehicles (e.g., a bicycle), rail vehicles (e.g., a train), watercraft (e.g., a boat), aircraft (e.g., an airplane), or spacecraft (e.g., a satellite), etc. The vehicle 102 relies on a perception system, including a radar system 104, for avoiding objects in the environment 100 when driving the vehicle 102 among the objects.

[0026] The radar system 104 outputs radar data that includes an indication of an object detected in the field of view 106. For example, the radar data can alert an advanced safety or autonomous driving system of the vehicle 102 that there is an object 108 in the driving path of the vehicle 102. The radar system 104 can be mounted on, mounted to, or integrated with any portion of the vehicle 102, such as a portion of the front, back, top, bottom, or side of the vehicle 102, a bumper, a side mirror, a headlight, and / or a taillight, or at any other internal or external location of the vehicle 102 where object detection using radar is desired. The radar system 104 can include multiple radar devices or multiple radar systems that coordinate with each other to act as a single provider of the field of view 106. The radar system 104 is primarily described as a MIMO radar system, however, the radar tracking techniques described herein are not necessarily limited to MIMO radar systems alone, but can apply to radar tracking in other kinds of radar systems.

[0027] The radar system 104 can include a combination of hardware components and software components executing thereon. For example, a computer-readable storage medium (CRM) of the radar system 104 can store machine-executable instructions that, when executed by a processor, cause the radar system 104 to output information about an object detected in the field of view 106. One example combination of hardware components and software components of the radar system 104 includes the MMIC 112, the at least one processor 114, the object detector model 116, the multi-layer tracker 118, and the output interface 120. The radar system 104 can include other components not shown. For example, for simplicity of the drawing, an antenna of the radar system 104 is not shown, however, the radar system 104 is configured to couple or include an antenna, such as a MIMO antenna array.

[0028] The MMIC 112 processes radar signals that can include one or more of: signal mixing to upconvert an analog signal to a radar frequency, power amplification of the upconverted signal, and routing of the signal to an antenna (not shown in FIG. 1) for transmission (e.g., radar signal 110-1). Additionally or alternatively, the MMIC 112 receives a reflection (e.g., radar signal 110-2) through an antenna, downconverts the signal to an intermediate frequency (IF) signal, and routes the IF signal to other components (e.g., radar processor 114) for further processing and analysis. Figure 1

[0029] ​The MMIC 112 of the radar system 104 includes an interface to an appropriate antenna, and during each frame time, the MMIC 112 transmits a radar signal 110-1 via the antenna to radiate objects in the field of view 106. The MMIC 112 receives, via the antenna, radar returns 110-2 that are reflected versions of the radar signal 110-1 transmitted for the frame. The MMIC 112 converts the radar returns 110-2 into a digital format to enable the radar system 104 to quickly establish tracking of objects in the field of view 106.

[0030] The processor 114, object detector model 116, multi-layer tracker 118, and output interface 120 are configured to process the radar returns 110-2 digitized by the MMIC 112 into various forms of other radar data, some of which are maintained by the radar system 104 for internal use, such as a datacube 122, detections 126, and data objects 128. Other forms of radar data are generated for use by other systems of the vehicle 102 (e.g., other than the radar system 104), such as one or more tracks 124. Each of the tracks 124 corresponds to a unique object tracked from the radar returns 110-2 obtained in the field of view 106.

[0031] For clarity, the environment 100 shows the radar processor 114 as a single processor, but various implementations can include multiple processors. To illustrate, the radar processor 114 can include a signal processor (SP) and a data processor (DP) for different processing tasks. As one example, the SP performs low-level processing such as fast Fourier transforms (FFTs), identifying targets of interest for the DP, generating range-Doppler plots, etc., while the DP performs high-level processing such as tracking, display updates, user interface generation, etc. The radar processor 114 processes digital microwave and / or radar signals (e.g., IF signals). This can include scheduling and configuring radar signal transmissions to create a field of view (e.g., the field of view 106) having a particular size, shape, and / or depth. For example, the radar processor 114 generates a digital representation of a signal transmission that is fed into a digital-to-analog converter (DAC) and upconverted to a radar frequency for transmission. In aspects, the DAC (not shown) is part of the radar system 104, but in other aspects, the DAC is part of the radar processor 114. Figure 1The MMIC 112 can be integrated with the radar processor 114 as part of a SoC, or included in the radar system 104 as a standalone component. Additionally or alternatively, the radar processor 114 receives and processes digitized versions of the reflected radar signals through an analog-to-digital converter (ADC), such as the digitized IF signals generated by the MMIC 112 from the radar signals 11-2. In a similar manner to the DAC, the ADC can be integrated with the MMIC 112, as part of a SoC with the radar processor 114, or included in the radar system 104 as a standalone component.

[0032] The processor 114 generates a datacube 122 representing a set of measurements related to the digitized radar returns 110-2 obtained from the MMIC 112 during a frame. In this MIMO example, the receive channels of the radar system 104 are processed into a 2D (two-dimensional) datacube 122 per channel including a range-Doppler plane, which results in a 3D (three-dimensional) datacube 122 for a ID (one-dimensional) effective receive antenna array, or a 4D (four-dimensional) datacube 122 for a 2D (two-dimensional) effective receive antenna array. The datacube 122 can also contain other information obtained from further processing, such as information about specific ID or 2D angular responses (e.g., elevation, azimuth, a combination of elevation and azimuth).

[0033] Further, based on the datacube 122 or directly based on the digitized radar returns 110-2 obtained from the MMIC 112, the processor 114 generates detections 126. Individual detections or groups of detections 126 can indicate a radial distance change rate of an effective radar return 110-2 (scatter) on an object 108 in the environment 100. The detections 126 can be in error in location (e.g., range, azimuth), however, many detections 126 with accurate and high-quality radial distance change rates can be used to support enhanced object measurements conveyed in some tracks 124.

[0034] Data objects 128 are maintained internally by the radar system 104, which are produced through analysis of detections 126 and separate analysis of data cubes 122. Data objects 128 include two types of data objects, referred to as model-based data objects and detection-based data objects. Data cubes 122 are parsed into model-based data objects 128 by executing object detector models 116. Detections 126 are parsed into detection-based data objects 128 by executing multi-layer trackers 118. Data objects 128 are maintained internally by the radar system 104 to enable subsequent generation of tracks 124. Data objects 128 are distinct from tracks 124, which are highly accurate model-based object representations augmented by radar detections. In contrast, data objects 128 refer to internal representations of objects (or portions of objects) and their respective measurements, which are maintained by the radar system 104 until they can be used to establish or augment tracks 124.

[0035] Data objects 128, which are model-based data objects, are generated by object detector models 116 directly from data cubes 122 using computer models (e.g., machine learning networks). Object detector models 116 can be any type of computer model programmed or trained to apply machine learning to identify various classes of objects from data cubes. Machine learning models, such as artificial neural networks, are a type of computer model that can quickly identify specific classes of objects from complex, multi-dimensional data sets (e.g., data cubes 122) and estimate measurements for the objects for establishing tracks (e.g., tracks 124). Relying on models, object detector models 116 are configured (e.g., trained or programmed) to estimate object measurements for specific classes of objects, however, relying on the model can hinder object detector models 116 from detecting many objects encountered in driving that the model is not able to directly identify from data cubes 122. The model of object detector models 116 can analyze a subset of data cubes 122 in its analysis, ignoring other information that can be derived from portions of data cubes 122 outside of the analyzed subset for the sake of speed.

[0036] In some examples, model-based data objects 128 are generated by object detector models 116 in various different ways, including using other types of models. Object detector models 116 can be any component or functionality configured to operate directly on radar data to accurately estimate object measurements, for example, producing estimates of location and extent of objects. For example, extent can be a directional bounding box (e.g., as Figure 4The oriented bounding box (as shown) is preferentially selected for depicting non-articulated vehicle objects in the driving scene. In some cases, the object detector model 116 can operate directly on the demodulated radar returns 110-2 received from the MMIC 112, or on a datacube 122 generated by the processor 114 after further processing of the demodulated radar returns 110-2 (e.g., after performing range-doppler FFT processing, after performing range-doppler FFT processing and angle processing). The object detector model 116 can operate on one or more radar devices of the radar system 104, and across one or more time steps, frames, durations, or scans of multiple radar devices of the radar system 104.

[0037] The multi-layer tracker 118 produces data objects 128 that are detection-based data objects. Derived from the detections 126, e.g., from grouping and analyzing the detections 126 using detection-based object tracking techniques, the multi-layer tracker 118 adds one or more detection-based data objects to the data objects 128 in addition to using the object detector model 116. Some of the detections 126 can be discarded (e.g., false detections, noise) to reduce the complexity / number of detections 126 for final processing. Some of the detections 126 can be discarded by applying a threshold and / or by applying non-maximum suppression of energy attributed to the detections (e.g., by finding peaks in the Doppler plane). The multi-layer tracker 118 can also include some of the detections 126 in its detection analysis that are ignored by the object detector model 116, of portions of the datacube 122. Each of the detection-based data objects 128 can correspond to any radar-detectable portion of an object in the field of view 106, while each of the model-based data objects correspond to an entire object of a particular class or type associated with the model. As such, the data objects 128 can include many more detection-based data objects than model-based data objects.

[0038] As the name implies, the multi-layer tracker 118 conceptually manages the data objects 128 as multi-layered internal tracks. One layer includes internal tracks referred to as model tracks for each of the model-based data objects 128. Other layers include internal tracks referred to as detection tracks for each of the detection-based data objects 128. These internal tracks are not exported from the radar system 104 and are not yet ready for use in establishing the final output tracks 124. However, each internal track includes information for populating fields in the tracks 124, including object measurements such as position, movement, or other characteristics. The multi-layered internal tracks are provided to the output interface 120, which resolves the model-based tracks with the detection-based tracks into the tracks 124. As described throughout this disclosure, while described as being maintained as layers, the internal tracks can be maintained using any suitable data structure to facilitate future alignment and position comparisons.

[0039] The output interface 120 receives the model-based tracks and the detection-based tracks maintained by the multi-layer tracker 118 to resolve the two types of internal tracks and generate the tracks 124 exported from the radar system 104. The tracks 124 define positions and other information about objects in the environment 100. As the radar system 104 captures radar returns 110-2 within subsequent frames, the tracks 124 are updated. In general, the output interface 120 causes each of the tracks 124 to include object-level information corresponding to an entire radar-reflective object in the environment 100. Each of the tracks 124 can have a structure including many fields containing various properties estimated for a particular object.

[0040] Some of the tracks 124 can be defined based solely on model estimates determined directly from the data cubes 122. However, ultimately, some of the tracks 124 are augmented to define objects using further information derived from the detections 126 used to improve the accuracy or detail of some of the model estimates. For example, the output interface 120 initially outputs the tracks 124 to include one track for each model-based track and avoid using any of the detection tracks maintained internally. When the model-based track and / or information derived from the detections 126 satisfy a quality threshold, the detection information is used to change information contained in fields of the tracks 124 initially established using model estimates alone. By augmenting the tracks 124 based on the detections 126, the accuracy and / or detail of the tracks 124 can be improved.

[0041] In some cases, the tracks 124 include transmitted objects that are only inferred from the detections 126. In some cases, the quality of the tracked based on the detections is sufficient for the output interface 120 to include a new track of an object that was not previously included in the model-based data objects 128 or in the existing tracks 124. The tracks 124 that include the enhanced or new tracks are output from the output interface 120, which enables the vehicle 102 to avoid objects including the objects 108 when driving in the environment 100.

[0042] In this way, the tracks 124 can have accurate and precise tracking of objects identified by the model, which can also be enhanced by analyzing the detections 126, by rate of change of distance, or other information inferred from other than the model. The multi-layer tracker 118 can use detection-based radar tracker techniques to track many objects that are not reported from the object detector model 116. This can prevent latency in providing radar tracking, which can then be augmented with additional information from the detector to increase the radar tracking to accurately, quickly, and transmit a wide variety of objects. Improving the accuracy and fidelity of object tracking in this way can result in safety of the vehicle 102 as well as other vehicles and pedestrians traveling in or near the roadway.

[0043] Example vehicle configuration

[0044] Figure 2 An example vehicle 102-1 is shown that includes a system configured to perform radar tracking with model estimates augmented by radar detections. The vehicle 102-1 is an example of the vehicle 102.

[0045] Included in the vehicle 102-1 is a radar system 104-1, which is an example of the radar system 104 shown in more detail. The vehicle 102-1 also includes a vehicle-based system 200 that is operatively and / or communicatively coupled to the radar system 104-1 via a link 202, which can be one or more wired and / or wireless links including a vehicle-based network communication for interconnecting components of the vehicle 102-1. In some examples, the link 202 is a vehicle communication bus.

[0046] The vehicle-based system 200 uses vehicle data, including radar data provided by the radar system 104-1 on the link 202, to perform vehicle-based functions, which can include, among other functions, functions for vehicle control. The vehicle-based system 200 can include any conceivable device, apparatus, component, module, part, subsystem, routine, circuit, processor, controller, etc. that uses the radar data to represent the vehicle 102-1. As some non-limiting examples, the vehicle-based system 200 can include a system for autonomous control 206-1, a system for safety 206-2, a system for positioning 206-3, a system for vehicle-to-vehicle communication 206-4, a system for as a passenger interface 206-5, and a system for as a multi-sensor tracker 206-6. Upon receiving the radar data, the functions provided by the vehicle-based system 200 use a portion of the radar data, including estimated measurements of the objects detected in the field of view 106, to configure the vehicle 102-1 to safely drive without colliding with the detected objects.

[0047] For example, the tracks 124 are examples of radar data output to the vehicle-based system 200 on the link 202. One of the tracks 124 can include information about the movement of the object 108, such as speed, position, etc., so that the vehicle-based system 200 can control or assist braking, steering, and / or acceleration of the vehicle 102-1 to avoid a collision with the object 108. The system for autonomous control 206-1 can use the tracks 124 received via the link 202 to autonomously or semi-autonomously safely drive the vehicle 102-1 on the roadway. The system for as a passenger interface 206-5 can use the information in the tracks 124 to allow an operator or passenger to have situational awareness to make driving decisions or provide operator input to a controller for providing more buffer to avoid the object. The system for vehicle-to-vehicle communication 206-4 can be used to provide the tracks 124 or information contained therein to other vehicles to allow operators, passengers, or controllers of the other vehicles to also avoid the tracked object or have confidence that the vehicle 102-1 knows of their existence based on receiving the tracks 124. By improving situational awareness of the vehicle 102-1 and other vehicles in the environment 100, the vehicle 102-1 can drive in a safer manner under manual, autonomous, or semi-autonomous control.

[0048] The radar system 104-1 includes a MMIC 112-1 (as an example of a MMIC 112). The MMIC 112-1 includes a transmitter / receiver element 210, a timing / control element 212, and an analog-to-digital converter 214. For simplicity of the figures, only one MMIC 112-1 is shown in the radar system 104-1, but it is understood that there can be multiple MMICs 112-1 in the radar system 104-1. Figure 2The mid-probe omits an antenna array (e.g., a MIMO antenna array), which is also part of the radar system 104-1 and is operatively coupled to the transmitter / receiver element 210.

[0049] The transmitter / receiver element 210 is configured to transmit EM signals using one or more components for transmitting electromagnetic (EM) energy. Using one or more components for receiving EM energy, the transmitter / receiver element is configured to receive reflections in response to objects reflecting the transmitted EM energy. The transmitter / receiver element 210 can be configured as a transceiver configured as a single component (e.g., a chip) to perform both transmission and reception. For example, the components for transmitting EM energy enable irradiation of the field of view 106 by transmitting the radar signals 110-1. The radar returns 110-2 represent reflections of the EM energy irradiated with the radar signals 110-1; the components for receiving EM energy enable reception of the radar returns 110-2, which can enable detection and tracking of objects in the field of view 106.

[0050] The timing / control element 212 performs operations that adjust characteristics (e.g., frequency, gain, phase, period) of the transmitted radar signals 110-1, or performs operations that receive the radar returns 110-2 in a manner that is effective for radar tracking using a particular antenna and radar design. For example, the timing / control element 212 causes the transmitter / receiver element 210 to adjust the size, shape, antenna pattern, or other characteristics of the radar system 104-1 to transmit the radar signals 110-1 using multiple transmitters and to capture more information from the radar returns 110-2 during each frame using multiple receivers, thereby enabling high-resolution radar tracking.

[0051] The analog-to-digital converter 214 converts the radar returns 110-2 acquired from the transmitter / receiver element 210 into a digital format that can be used to generate digital cubes, detect, and implement other radar processing. The MMIC 112-1 outputs the digitized radar returns 110-2 over a link 208, which represents an internal communication link between components of the radar system 104-1. The link 208 can be wired or wireless, and enables internal representation of radar data (e.g., data cubes 122, detections 126) to be exchanged within the radar system 104-1 before being presented as the tracking 124 output over the link 202.

[0052] The radar system 104-1, which is operatively coupled to the MMIC 112-1 and the link 208, also includes at least one processor 114-1, which is an example of the processor 114. Some examples of the processor 114-1 include a controller, control circuitry, a microprocessor, a chip, a system, a system on a chip, a device, a processing unit, a digital signal processing unit, a graphics processing unit, and a central processing unit. The processor 114-1 can be any component configured to process frames of digitized versions of the radar returns 110-2 acquired from the MMIC 112-1 to convey objects in the environment 100 as information presented in the track 124. The processor 114-1 can include multiple processors, one or more cores, embedded memory storing software or firmware, cache, or any other computer elements that enable the processor 114-1 to execute machine-readable instructions for generating the track 124.

[0053] Machine-readable instructions executed by the processor 114-1 can be stored by a computer-readable medium (CRM) 204 of the radar system 104-1. The CRM 204 can also be used to store data managed by the processor 114-1 during instruction execution. In some examples, the CRM 204 and the processor 114-1 are a single component, such as a system on a chip that includes the CRM 204 configured as a dedicated memory of the processor 114-1. In some examples, access to the CRM 204 is shared by other components of the radar system 104-1 (e.g., the MMIC 112-1) connected to the CRM 204, e.g., via the link 208. The processor 114-1 acquires instructions from the CRM 204; execution of the instructions configures the processor 114-1 for performing radar operations, such as radar tracking, that result in transmission of the track 124 to the vehicle-based system 200 and other components of the vehicle 102-1 through the link 202.

[0054] In this example, the CRM 204 includes instructions to configure the processor 114-1 for generating the datacube 122, generating the detections 126, and resolving the datacube 122 and the detections 126 into the track 124 by using information derived from the detections 126 to enhance model estimates that define object measurements conveyed by the track 124. For example, the CRM 204 includes instructions to execute the object detector module 116-1, which is an example of the object detector module 116. Also included in the CRM 204 are instruction sets that, when executed by the processor 114-1, implement the multi-layer tracker 118-1 and the output interface 120-1, which are examples of the multi-layer tracker 118 and the output interface 120, respectively.

[0055] When executed by processor 114-1, object detector module 116-1 receives a sequence of data cubes, including data cube 122. Object detector model 116-1 outputs model-based data objects 128-1, which are a subset of data objects 128. Model-based data objects 128-1 can be presented on link 208 as internal data managed by processor 114-1 and stored on CRM 204. Model-based data objects 128-1 are each of a particular object class or object type that object detector model 116-1 is specifically trained or programmed to recognize. Model-based data objects 128-1 can include detailed information indicating object class, location, size, and shape; with lower accuracy, information about heading and velocity (e.g., speed) is included in model-based data objects 128-1, which is estimated by object detector model 116-1. Although object detector model 116-1 can not be able to recognize some types of objects, information about recognized model-based data objects 128-1 is maintained by CRM 204 for use by multi-layer tracker 118-1, facilitating output interface 120-1 to generate tracks 124.

[0056] Multi-layer tracker 118-1 can output two internal sets of tracks, which are resolved by output interface 120-1 to produce tracks 124. The two sets of tracks output from multi-layer tracker 118-1 can include model tracks 216 and detection tracks 218. Each of model-based data objects 128-1 corresponds to one of model tracks 216. Each of detection tracks 218 corresponds to one of data objects 128 (referred to as detection-based data objects 128-2) that are produced internally by multi-layer tracker.

[0057] Detection-based data objects 128-2 are formed by grouping and analyzing detections 126, which are derived from other than object detector model 116-1 and data cubes 122. Typically, at least one of detection-based data objects 128-2 corresponds to one of model-based data objects 128-1. However, some detection-based data objects 128-1 convey additional objects that can be detected from radar returns 110-2 beyond model-based data objects 128-1 that are recognizable by the model of object detector model 116-1. Since detection tracks 218 correspond to each of detection-based data objects 128-2, the number of detection-based data objects 128-2 can be greater than model-based data objects 128-1, and thus detection tracks 218 can include substantially more tracks than model tracks 216.

[0058] The detection track 218 can point to a detection-based data object 128-2, which can be any object type or any object class (with a radar-reflective surface), while the model track 216 can point to a model-based data object 128-1, which is limited to object types and object classes that are model-recognizable by the object detector model 116-1. Additional details that can be derived from the detection 126 are included in the detection-based data object 128-2 and are communicated in the detection track 218, and thus the detection track 218 can include information that improves the measurements estimated for the model-based data object 128-1 that have been communicated by the model track 216.

[0059] The output interface 120-1 receives the model tracks 216 and the detection tracks 218 from the multi-layer tracker 118-1. In some examples, the detection 126 is also an input to the output interface 120-1. By comparing the set of model tracks 216 and the set of detection tracks 218, the output interface 120-1 determines whether to use any information from the detection tracks 218 to augment the information in any of the model tracks 216, or whether to form additional tracks between the tracks 124 based on the detection tracks 218 to communicate additional objects beyond those communicated by the model tracks 216 (even if augmented by the detection 126).

[0060] Details of how the output interface 120-1 resolves the model tracks 216 and the detection tracks 218 into the tracks 124 are described in further detail below. The output interface 120-1 employs an overlay approach to resolve the set of model tracks 216 and the set of detection tracks 218. The basis for this overlay approach is two-fold.

[0061] First, the detection 126 and the detection-based data object 128-2 used to support the detection tracks 218 generally have these characteristics: the detection 126 can report any type of radar-reflective object, multiple detections 126 can report a single object, multiple detections 126 can report any location on the object, and sometimes report outside of the object but close to the object. As such, the detection-based data object 128-2 can have significant range or angle (e.g., azimuth) errors, resulting in location errors attributed to the detection tracks 218. When based on the detection 126, the rate of change of range associated with the detection-based data object 128-2 is generally of high quality.

[0062] Second, the model-based data objects 128-1 generally include measurements with the following properties estimated by the object detector model 116-1: only objects of a particular object type or class are reported (e.g., pedestrians, bicycles, cars, trucks). However, the measurements from the datacube 122 estimated by the model are generally high quality in terms of location and size measurements and determinations about object class. To degrade the quality, the orientation estimates from the model are reported, as well as inaccurate velocity vectors for the identified objects. Given this dual basis, the output interface 120-1 is configured to identify those desirable aspects of the detection tracks 218 that, when overlaid or corresponding to a portion of the existing model tracks 216, can be used to augment the existing model tracks 216, thereby improving the accuracy and resolution of the tracks 124.

[0063] In tracking the model-based data objects 128-1, the high-quality capabilities of the model to process the datacube 122 to determine object class, location, size, and orientation are fully leveraged. The high-quality distance change rate of the detection-based data objects 128-2 can be used to augment or reinforce the tracks 124 produced from the model-based data objects 128-1, for example, to improve the velocity vectors or other low-quality measurements estimated by the model. In some cases, this dual-coverage approach enables the output interface 120-1 to convey all types of radar-detectable objects in the field of view 106 (not just those that the model can identify) by selectively promoting one or more of the detection tracks 218 to be included in the tracks 124 on the link 202. By maintaining the model tracks 216 separate from the detection tracks 218, the multi-layer tracker 118-1 enables the output interface 120-1 to quickly resolve the tracks 124 into a more accurate representation of potential obstacles in the field of view 106.

[0064] Example Radar Architecture

[0065] Figures 3-1 to 3-4 A further example of a system is shown that is configured to perform radar tracking with model estimates augmented by radar detections. The techniques described herein for augmenting model estimates with detections can be implemented using a variety of computer architectures, including using one or more processor components (e.g., processors 114, processors 114-1) distributed among multiple components or portions of a radar system circuit. An example radar architecture is shown in FIG. 2B. Figures 3-1 to 3-4Radar systems 104-2, 104-3, 104-4, and 104-5, respectively, each of which is an example architecture for radar system 104 and radar system 104-1. Radar systems 104-2 through 104-5 use identically numbered components to implement the same or similar functionality as radar system 104 and radar system 104-1, some of which are shown with additional details, arranged in various ways to illustrate different example architectures.

[0066] Radar system 104-2 is a dual-processor architecture that includes processor 114-2 and processor 114-3 in addition to antenna 300 (e.g., an antenna array) and MMIC 112. Processors 114-2 and 114-3 are configured to perform the same or similar operations described with respect to processor 114 and processor 114-1, however, with more than two processors, radar system 104-2 can share or allocate the performance of operations between two different processors 114-2 and 114-3, which themselves can be two different processor types.

[0067] Processor 114-2 is a radar processor that receives digitized radar returns 302, which are acquired from MMIC 112 via antenna 300, and uses digital signal transformer 306 to generate datacube 122 and detections 126 at a sufficient speed to achieve a frame rate for implementing object recognition for driving. Processor 114-2 performs digital signal transformer 306 and TX / RX control component 308.

[0068] TX / RX control component 308 can control MMIC 112 to cause MMIC 112 to transmit radar signal 110-1 and receive radar returns 110-2. TX / RX control component 308 can specify a frequency, a frame rate, a gain, or any other parameter or input to enable MMIC 112 to output radar signal 110-1 and receive radar returns 110-2 for each frame.

[0069] Digital signal converter 306 can use known radar processing techniques to transform digitized radar echoes 302 into data cubes 122 and detections 126. For example, some digitized radar echoes 302 can undergo fast time processing, while others can undergo slow time processing, which can ultimately undergo integration and thresholding as described above to represent the transformed digitized radar echoes 302 as data cubes 122 that can be evaluated for tracking. Detections 126 can be generated from digital signal converter 306 performing peak estimation and detection techniques (including direction of arrival calculation). For each frame of digitized radar echoes 302, digital signal converter 306 can cause processor 114-2 to output detections 126 and data cubes 122.

[0070] Processor 114-3 differs from processor 114-2 and can be considered a domain controller. As a domain controller, processor 114-3 can perform a number of functions, including being configured at least to generate tracking 124 based on model estimates derived from data cube 122 and, in some cases, enhanced by detection 126. Other functions of the domain controller may be possible, including other operations performed based on data cube 122, detection 126, and / or tracking 124. As some examples, the domain controller can perform tracking functions for performing conventional types of radar tracking, including detection-based radar tracking using detection 126 alone, or model-based radar tracking using data cube 122 alone. The domain controller can manage other types of object tracking. For example, in addition to using detection 126, data cube 122, or other radar data, the domain controller can also combine aspects of tracking 124 with tracking of other aspects generated by other sensors to perform multi-sensor type object tracking (e.g., sensor fusion tracking). In some cases, the domain controller performs operations for scene processing that can supplement the object tracking function to output information that conveys more details about environment 100, including objects other than those associated with tracking 124.

[0071] To perform radar tracking using detection-enhanced model estimation, object detector model 116-2 (which is an example of object detector model 116-1) acquires data cube 122 and outputs model-based data object 128-1 by directly analyzing data cube 122, relying on a model trained or programmed to estimate measurements of model-based data object 128-1. Processor 114-3 may be a preferred type of processing unit for executing machine learning models, such as a graphics processing unit (GPU). Processor 114-3 may include multiple cores or execution pipelines, for example, for executing the model while performing other functions of radar system 104-2 that are not yet executed on processor 114-2.

[0072] Then, as an example of multi-layer tracker 118-1, multi-layer tracker 118-2 reports detection track 218 and model track 216, which can be derived from model-based data object 128-1 and detection-based data object 128-2 generated from detection 126. Model tracker 310 and detection tracker 312 are included within multi-layer tracker 118-2.

[0073] Model tracker 310 is configured to process model-based data objects 128-1 acquired over multiple frames to derive model tracking 216. Model tracker 310 is a radar tracker configured to generate model tracking 216 based on its input to data cube 122 received over time. It uses a model that can be trained using machine learning or otherwise programmed to estimate object measurements directly for specific categories of objects directly from data cube 122. Model tracker 310 may not report some types of objects (as explained later) directly identified by detection tracker 312 from detection 126. The output from model tracker 310 includes at least one of the model tracking 216 corresponding to each of the model-based data objects 128-1.

[0074] Detection tracker 312 is a radar tracker configured to generate detection tracks 218 by analyzing detections 126 acquired over time. By performing a process called thresholding and non-maximum suppression on the EM received energy using techniques (such as finding peaks within the EM received energy using specific isolation techniques for each plane) in one or more planes (e.g., range-Doppler plane, range-angle plane, Doppler-angle plane), usable groups of detections 126 can be isolated from erroneous or unusable groups of detections 126 (including detections 126 presented as noise). The output from detection tracker 312 includes at least one detection track 218 corresponding to at least one of the detection-based data objects 128-2. Detection tracker 312 is configured to process detections 126 over multiple frames to derive detection-based data objects 128-2 used to form detection tracks 218.

[0075] Finally, processor 114-3 executes output interface 120-1, which is an example of output interface 120. Output interface 120-1 includes a tracking resolver 314 configured to receive detection tracking 218 and model tracking 216 as inputs and derive tracking 124 from these inputs, which is output as radar data on link 202. At least one of tracking 124 is established for each of model tracking 216. As the consistency and / or accuracy of detection tracking 218 increases over multiple frames, tracking resolver 314 can enhance the initially established tracking 124, or tracking resolver 314 can establish one or more new trackings based on detection tracking 218 and add them to tracking 124. Detection 126 can be received as further input for tracking resolver 314 when generating tracking 124; detection 126 can be obtained directly from digital signal converter 306 or inferred from detection-based data object 128-2 or detection tracking 218.

[0076] and Figure 3-2 To create a contrast, Figure 3-2An example architecture for implementing the technology described herein is shown, including an antenna 300, an MMIC 112, a digital signal converter 306, a TX / RX control unit 308, an object detector model 116-2, a multilayer tracker 118-2, and an output interface 120-1, all on a single processor or component. Unlike the dual-processor architecture of radar system 104-2, radar system 104-3 includes a processor 114-4, referred to as a system-on-a-chip, incorporating the functions of processors 114-2 and 114-3. Processor 114-4 may include separate execution blocks; however, detection 126, data cube 122, and tracking 124 are generated from a single logic block, chip, or integrated circuit.

[0077] Figure 3-3 Including radar system 104-4, Figure 3-4 Radar system 104-5 is shown, each illustrating a further example of a multiprocessor architecture that can be used to implement radar system 104-2 using more than two processors. In radar system 104-4, in addition to processor 114-2, a combination of three separate processors 114-5, 114-6, and 114-7 is included, instead of processor 114-3. Similarly, in addition to processor 114-2, radar system 104-5 includes processors 114-5 and 114-8, instead of processor 114-3 or a combination of processors 114-5, 114-6, and 114-7.

[0078] Besides reference Figures 3-1 to 3-4 In addition to the architectures shown and described, other architectures are also possible, including those involving more or fewer components than those shown. Each architecture may include a unique combination of processor types to implement radar systems 104-2, 104-3, 104-4, and 104-5 to obtain certain computational benefits (e.g., faster execution time, faster reporting of tracking 124). Certain types of processors are superior to other types of processors in quickly processing specific data types or performing specific types of operations. Specific types of processors used in radar systems 104-2, 104-3, 104-4, and 104-5 may be uniquely suited for performing specific operations related to radar tracking using detection-enhanced model estimations. Processors 114-2, 114-3, 114-4, 114-5, 114-6, 114-7, and 114-8 may include any type of processor used in combination to perform radar tracking according to the described techniques.

[0079] The concept of multi-level methods

[0080] Figure 4System 400 is shown, configured to maintain multiple tracking layers to facilitate radar tracking using model estimation enhanced by radar detection. System 400 represents a conceptual view of tracking resolver 314-1, which is derived from... Figure 3-1 Example of a tracer parser 314.

[0081] Tracking resolver 314-1 is configured to receive input including model tracking 216 and detection tracking 218 (e.g., provided by multi-layer tracker 118-2) to derive a single set of tracking 124-1, as an example of tracking 124 output on link 202. Tracking resolver 314-1 relies on an overlay approach to manage model tracking 216 and detection tracking 218, maintaining a background layer 404 for each detection tracking 218 and a foreground layer 402 for each model tracking 216. The foreground layer 402 may also be referred to as the first tracking layer or primary tracking layer; similarly, the background layer may be alternatively described as the second tracking layer or secondary tracking layer. By observing the alignment between the portions of the detection tracking 218 on the background layer 404 and the portions of the model tracking 216 on the foreground layer 402, the tracking parser 314-1 can output tracking 124-1 in such a way that objects in the environment 100 can be transmitted more accurately compared to being transmitted separately from model tracking 216 or detection tracking 218.

[0082] When generated from model tracking 216, the foreground layer 402 includes model tracking 216-1 and model tracking 216-2 derived from model tracker 310 (e.g., using tracking techniques applied by model tracker 310 to locate measurements of model-based data object 128-1). These model tracking 216-1 and 216-2 benefit from high-quality estimates of position, size, orientation, and object category provided by object detector model 116-2. In contrast to the foreground layer 402, the background layer 404 is generated from detection tracking 218, which includes detection tracking 218-1 to 218-6 derived from detection tracker 312 (e.g., using tracking techniques applied by detection tracker 312 to estimate the position and distance change rate of detection-based data object 128-2). Background layer 404 is kept separate from foreground layer 402, so that tracking 218-1 to 218-6 does not depend on model tracking 216 or model-based data object 128-1 output from object detector model 116-2.

[0083] Tracking parser 314-1 enables output interface 120-1 to perform a track-down selection scheme. In this regard, either a model track 216 maintained on the foreground layer and a detection track 218 maintained on the background layer 404 represents a track pairing for the same object in the field of view 106. In other words, some model tracks 216 in the foreground layer 402 can occlude some of their paired detection tracks 218 in the background layer 404. Detection tracks 218 paired with one of the model tracks 216 are not output within tracking 124-1. Instead, detection tracks 218 are maintained in the background layer 404 until the information derived from the detection tracks 218 has sufficient quality to enhance one of the model tracks 216 pointing to a common object.

[0084] As shown by the temporal changes of tracking 124-1 transmitted on link 202 from time T1 to time T2 and then to time T3, tracking resolver 314-1 enables radar system 104 to track the category of model-based data object 128-1 identified by object detector model 116-2. This category includes high-quality estimates of position, velocity, acceleration, yaw rate, trajectory curvature, size, orientation, and object category. Initially, good tracking can be achieved by using position measurements from model tracking 216 to transmit at least some of the radar-detectable objects in field of view 106. However, at each subsequent timestamp, by evaluating detection tracking 218 relative to model tracking 216 at each time step, a more accurate detection-based rate of change of position and range associated with the detection tracking 218 aligned with the corresponding one in model tracking 216 is obtained to quickly correct any errors in the estimated velocity, acceleration, yaw rate, and curvature previously transmitted in tracking 124-1.

[0085] For example, turn to trace 124-1 shown at different times T1, T2, and T3. Initially, trace parser 314-1 renders trace 124-1 on link 202 at time T1, including traces of each of model traces 216-1 and 216-2 corresponding to foreground layer 402. These are high-quality traces that transmit model estimates of object measurements derived from model-based data object 128-1.

[0086] Subsequently, at time T2, the tracking resolver 314-1 enhances or updates tracking 124-1 to appear on link 202 differently at time T2 compared to time T1. In some cases, model tracking 216-1 and 216-2 can benefit from the high-quality position and distance change rates of detection 126 (e.g., for rapidly enhanced estimates of velocity, yaw rate, and trajectory curvature). For example, when any of the detection tracking 218 of background layer 404 is in an accurate state and / or aligned in position with one of the model tracking 216, they can be used to update object measurements transmitted in model tracking 216-1 and 216-2. For example, this could include updating the velocity estimate using more accurate information derived from detection 126. In other words, although the tracking 124-1 at time T2 still includes the tracking of each of the model trackings 216-1 and 216-2 corresponding to the foreground layer 402, the model tracking 216-1 is updated at time T2 to include new velocity or other new information derived from the detection trackings 218-1 and 218-2 that are aligned with the model tracking 216-1 in position. Figure 4 The diagram shows an enhanced version of model tracking 216-1, as one of tracking 124-1 at time T2, which includes a label abbreviated as "216-1&(218-1,218-2)", where "&" indicates an enhancement operation applied to the model tracking, as specified by the variable to the left of "&". The enhancement operation depends on information derived from one or more detection tracks, as specified by a series of variables to the right of "&".

[0087] Figure 4 The diagram also shows that, at a further time, at time T3, and after further radar data has been collected by radar system 104, track 124-1 can subsequently be presented on link 202, where model track 216-2 is updated to include new velocity or other new information derived from detection track 218-4, which is aligned in position with model track 216-2. This is in addition to the output of model track 216-1, which has been updated or further updated using information derived from detection tracks 218-1 and 218-2. This is Figure 4 An enhanced version of model tracking 216-2 is shown, as one of tracking 124-1 at time T3, which includes the label “216-2 & 218-4”.

[0088] Example process - in general

[0089] Figure 5 A flowchart of an example process 500 for radar tracking using model estimation enhanced by radar detection is shown. For ease of description, process 500 is primarily described using radar system 104 and its various examples (including radar systems 104-1 to 104-6, including...). Figure 4The context in which the trace parser 314 is executed is described. In this example, the operations (also called steps) of process 500 are numbered from 502 to 520. However, this numbering does not necessarily imply a specific order of operations. The steps of process 500 can be related to... Figure 5 The diagram shows different ways to rearrange, skip, repeat, or execute specific methods.

[0090] At step 502, object measurements of the environment are acquired from the radar cube-based model, and at step 504, in addition to the object measurements, detections of the environment are also acquired. For example, model-based data object 128-1 is generated by object detector model 116-1 based on radar data cube 122. Detection 126 is received by multilayer tracker 118-1, and detection-based data object 128-2 is derived from detection 126 in a manner separate from the operations performed by object detector model 116-1 when analyzing data cube 122.

[0091] At 506, tracking of the object is established and updated based on object measurements estimated by the model. For example, multilayer tracker 118-1 generates model tracking 216 to include a corresponding one of model tracking 216 for each of the model-based data objects 128-1. Each of the model tracking 216 may include multiple fields of information derived from the object measurements estimated by object detector model 116-1 for a unique one of the model-based data objects 128-1. In addition to establishing tracking, step 506 may also include updating the established tracking based on updated object measurements estimated by the model. For example, at each frame, data cube 122 is input to object detector model 116-1, which may result in an update of the previously estimated object measurements and enable output interface 120-1 to perform an update of one or more fields of tracking 124 established from model tracking 216 received from the previous frame. An example of step 506 is shown in more detail below. Figure 6-1 Step 506-1 in the process.

[0092] At step 508, the traces are output for use by the vehicle-based system. For example, output interface 120-1 may receive model traces 216 and initially output each of the model traces as a set of traces 124 sent on link 202 to the vehicle-based system 200.

[0093] At step 510, it is determined whether any of the radar detections observed from signals received from the environment corresponds to the same object as the tracking established using information obtained from the model. In response to determining that at least one radar detection corresponds to the same object as the tracking established using information obtained from the model, step 512 is executed after the "Yes" output from step 510. For any of the remaining radar detections that do not correspond to the same object as the tracking established using information obtained from the model, step 516 is executed after the "No" output from step 510. An example of step 510 is shown in more detail below. Figure 6-1 Step 510-1 in the process.

[0094] For example, multi-layer tracker 118-1 generates detection traces 218 to include one detection trace for each of the detection-based data objects 128-2 derived from detection 126. Each of the detection traces 218 may include information derived from detection 126 for a unique instance of the detection-based data objects 128-2. In response to receiving model traces 216 and detection traces 218, output interface 120-1 may determine whether any of the detection traces 218 is aligned with or related to an object in any of the model traces 216. If the location of a subset of the detection traces 218 corresponds to the location of a transmitted object in model trace 216, then at step 512, the model trace is considered a candidate for enhancement using the additional information contained in the corresponding subset of the detection traces 218. However, if none of the detection traces 218 corresponds to the location of a transmitted object in model trace 216, then the model trace is not considered a candidate for enhancement using the additional information contained in any of the detection traces 218. Model trace 216 may be updated in response to subsequent data cubes acquired.

[0095] At 512, in response to determining that at least one radar detection corresponds to the same object as the tracking established using information obtained from the model, additional information derived from the radar detection is used to enhance at least a portion of the tracking initialized based on object measurements obtained from the model, to improve the accuracy or detail associated with that portion of the tracking. For example, when it is determined at 512 that the tracking output at 508 is a candidate for enhancement, the tracking output at 508 can be enhanced based on additional information transmitted in some detection tracks 218 aligned with that tracking. In addition to using detection 126, this may also include adjusting or updating the orientation, velocity, rate, size, or other information estimated by the object detector model 116-1. An example of step 512 is shown in more detail below. Figure 6-1 Step 512-1 in the process.

[0096] At 514, an update to the tracking established or updated in step 508 is output, including the enhanced portion transmitted in the update, thereby improving the accuracy or detail associated with object tracking compared to estimates relying solely on measurements derived from model-based data cube analysis. In other words, prior to step 512 (where enhancement of part of the tracking may occur), a tracking established or updated using information obtained from object detector model 116-1 within one or more frames and information obtained from the occurrence of step 508 is output as one of tracking 124. After considering the tracking as a candidate for enhancement at step 510, the tracking is updated at step 512, transmitting that portion of the tracking in a manner different from how the portion of the tracking prior to enhancement was transmitted in tracking 124.

[0097] Even at 510, if radar system 104-1 determines that detection 126 cannot be used to update an existing track established from model tracking 216, detection 126 may still be useful for radar system 104-1 to transmit other aspects of environment 100 that object detector model 116-1 cannot recognize. Referring back to 510, for any radar detection that does not correspond to the same object as the track established using information obtained from the model, process 500 then follows up with a "No" from the output of step 510 to execute step 516.

[0098] At 516, it is determined whether any radar detection observed from signals received from the environment corresponds to another object different from the tracked object established at 506 using information obtained from the model. This may include determining whether another radar detection observed from signals received from the environment corresponds to the same object as the track established using information obtained from the model. For example, for each of the detection-based data objects 128-2 derived from detection 126 using detection track 218, output interface 120-1 may determine whether any additional information other than that observed by object detector model 116-1 from data cube analysis can be derived from detection 126 and / or detection track 218. Output interface 120-1 can resolve detection track 218 and model track 216 by updating one or more of tracks 124 with this additional information to improve the accuracy of object measurements transmitted by track 124 when compared with previous outputs of track 124.

[0099] If neither radar detection nor detection tracking 218 can be used to support radar tracking of another object, then a "No" is output from step 516. For any detection tracking 218 or detection 126 whose quality is insufficient to support enhancement or object tracking itself, radar system 104-1 may retain detection tracking 218 (even if unused) for future consideration in subsequent frames or periods of process 500. This may include: in response to determining that another radar detection observed from signals received from the environment does not correspond to the same object as the tracking established using information obtained from the model, thereby avoiding the use of additional information derived from the other radar detection to enhance any part of the tracking initially set up based on object measurements obtained from the model.

[0100] However, at 516, in response to determining that at least one radar detection corresponds to another object, at 518 additional information derived from the radar detection is used to establish or update another track. That is, in response to identifying at least one radar detection that corresponds to another object and is not currently captured by track 124, step 518 is executed after the "Yes" output in step 516.

[0101] At 518, output interface 120-1 can identify one of the detection tracks 218: although it is not aligned with any model track 216, its quality is sufficient to be reported and tracked in track 124. Output interface 120-1 can include a new track in track 124, or update one of the following in track 124: that corresponds to an object or object category not reported by object detector model 116-1 but otherwise observable from radar detection. An example of step 518 is shown in more detail below. Figure 6-2 Step 518-1 in the process.

[0102] At 520, an update occurs to track 124 on link 202 to include any combination of the following tracks: A) a track established or updated at 508 based on object measurements estimated from object detector model 116-1; B) a track established or updated at 508 that is enhanced at 512 using additional information from detection 126 to improve the accuracy or detail associated with the parts of the track; or C) a track established or updated at 518 based on additional information from detection 126 that has not yet been used to enhance one of the existing tracks 124. For example, output interface 120-1 causes track 124 to reach vehicle system 200 with improved accuracy over time. Initially, track 124 points to objects recognizable by object detector model 116-1. Over time, track 124 can be updated by detection-based enhancements to update information in track 124 and / or transmit object categories beyond those object categories recognizable by the data cube feedback model.

[0103] Example Process - Further Details

[0104] Figure 6-1 and Figure 6-2 Additional flowcharts are shown, including Figure 5 The detailed example of process 500 or its variants is provided for radar tracking using model estimation enhanced by radar detection. Process 600-1 is shown in... Figure 6-1 The process includes steps 506-1, 510-1, and 512-1. Process 600-1 is an example of enhancing tracking 124 based on detection 126 and / or detection tracking 218, which is initially established for each of the model tracking 216 identified by object detector model 116-1. As a supplement to process 600-1, process 600-2 includes step 518-1 and provides another example of enhancing detection-based tracking 124, this time including additional tracking of objects that object detector model 116-1 cannot identify from data cube 122.

[0105] Model estimation enhancement

[0106] First, proceed to step 506-1 of process 600-1, where steps 602, 604, and 606 can be executed to establish or update object tracking based on object measurements obtained from the model (e.g., object tracker model 116-1). Execution of process 600-1 can lead to the execution of step 516. That is, tracking resolver 314 can use model tracking 216 to update or establish tracking 124 sent by output interface 120-1 to vehicle system 200. To do this, at 602, it is determined whether each of the model tracking 216 is associated with any tracking 124 already established by or output from radar system 104-1. If one of the model tracking 216 is already associated with one of the previously established tracking 124, the "yes" branch flows from 602 to step 604. Alternatively, in response to determining that none of the previously established tracking 124 corresponds to the same object as one of the model tracking 216, the "no" branch flows from 602 to step 606.

[0107] At 604, one of the traces 124 is updated based on the one in model trace 216 that corresponds to the same object. This occurs without using information from detection trace 218 or detection 126, but instead relies on information output from object detector model 116-1.

[0108] At 606, a new trace is created to be added to trace 124 based on one of the model traces 216 that has not yet been represented by the previously created trace 124. Similar to step 604, step 606 occurs without using information from detection trace 218 or detection 126, but instead relies on information output from object detector model 116-1 captured in the form of model-based data objects 128-1 and model trace 216.

[0109] For example, the tracking resolver 314 can use a tracking filter to fuse information contained in the model-based object measurements with detection 126 or detection tracking 218. The tracking filter can use an approximate coordinated turning constant acceleration (CTCA) motion model to model the dynamic behavior of the object being tracked 124. Any other suitable tracking filter model, such as Cartesian constant acceleration, coordinated turning, etc., can be used to implement any suitable tracking filter technique, including extended Kalman filtering, unscented Kalman filtering, PHD filtering, particle filtering, etc. Tracking filter state variables can include X and Y position, heading, trajectory curvature, velocity, and tangential acceleration, etc. State variables can indicate the quality of a partial tracking established using information from the model. For example, initially, the velocity or reported position may be less accurate, but over time, the quality of the object measurements estimated from the model-based tracking can improve.

[0110] In some cases, establishing or updating one of the tracks 124 at 604 and 606 does not necessarily include outputting that track until the object detector model 116-1 has processed enough frames of data cube 122. During initialization, some of the tracks 124 are suppressed from output and are only reported after a period of time; the quality of the tracks 124 improves over time, so that when output to the vehicle system 200 on link 202, they include accurate information. At the end of the interval, a curve fitting process can be used to provide initial velocity estimates; for example, the curve fitting process can be used to initialize the velocity state (or velocity and heading state) of the tracking filter used to report the tracks 124.

[0111] Proceeding to the next step 510-1, as tracking 124 is established or updated based on model tracking 216, process 600-1 flows out of step 506-1 and into step 510-1 to consider whether to perform detection-based enhancement. At this point, each tracking 124 ready for output is individually supported by model tracking 216 and model object measurements (e.g., position, orientation, velocity, size). Before detection 126 and / or detection tracking 218 can be used to enhance the object measurements of tracking 124, the object measurement state (specifically, velocity and related states) derived from the tracking filter is analyzed for tracking 124 to determine whether tracking 124 contains sufficiently accurate information for enhancement by additional information that can be derived from detection 126 and / or detection tracking 218.

[0112] Additional information derived from detection 126 and / or detection tracking 218 (which is aligned with tracking 124 established for each of model tracking 216) can be used to enhance or reinforce object measurements to improve tracking quality. For example, updating the velocity estimate based on detection 126 or detection tracking 218 can further improve the estimated velocity provided by object detector model 116-1 and the employed tracking filter, enabling radar system 104-1 to exceed any limitations of the tracking filter or object detector model 116-1 regarding accuracy. This is particularly beneficial for improving the responsiveness of tracking 124 so as to transmit accurate measurements even in the event of sudden maneuvering or turning.

[0113] The velocity states managed by the tracking filter or reported from the object detector model 116-1 may contain pervasive errors. For example, a sudden maneuver of vehicle 102 may take some time to be observed in the model estimates of object measurements (such as the object's centroid). Detection-based enhancements can reduce errors in the velocity states and other velocity-related states (e.g., heading, curvature, and acceleration), especially during these sudden maneuvers or turns of vehicle 102 or the tracked object. Therefore, before additional information can be used to enhance tracking 124, tracking 124 can be performed over multiple frames or cycles (e.g., once, twice, three times, four times, five times) before the velocity states of tracking 124 are stable, consistent, or otherwise of sufficient quality to be considered for detection-based enhancements.

[0114] Steps 608 and 610 of steps 510-1 can be performed to determine whether any radar detection or detection tracking corresponds to any of the objects in tracking 124 established or updated based on object measurements obtained from the model. Tracking resolver 314 or tracking filter can take into account the quality of tracking 124 and model tracking 216 to determine whether any tracking is in a state that can be used for detection-based enhancement.

[0115] At 608, it is determined whether the model tracking is in a state where detection-based enhancement is being considered. That is, in response to determining that the quality of these portions of the tracking meets the quality threshold for performing detection-based enhancement, a portion of the tracking 124, initialized based on object measurements obtained from object detector model 116-1, can be enhanced or updated. For example, if a velocity state corresponding to one of the model tracking 216 or the corresponding one of the tracking 124 has sufficient quality to consider detection-based enhancement, the "yes" branch of 608 flows to step 610. Alternatively, in response to determining that a corresponding one of the model tracking 216 or the previously established tracking 124 is not in a state for enhancement (e.g., because the velocity state does not have sufficient quality to consider using detection-based enhancement), the "no" branch of 608 flows to step 614.

[0116] At 610, it is determined whether the detected track or detection defines the same object as the model track in the augmentation consideration state. This includes determining whether another radar detection observed from signals received from the environment corresponds to the same object as the track established using information obtained from the model. For example, a range rate gating procedure is used to not emphasize or eliminate detection 126 and detection track 218 with velocity rate of change that deviates too far from the tracking filter state used for model tracking in the augmentation consideration. If either detection track 218 or detection 126 is reported to be at or near a location of one of the model tracks 216 in the augmentation state, the "yes" branch of step 610 flows to step 612. Alternatively, in response to determining that model track 216 and detection 126 do not define the same object as the model track in the augmentation consideration state, the "no" branch from 602 flows to step 516.

[0117] Step 512-1 includes steps 612 and 614. At step 614, a track 124 not established or updated using detection 126 and / or detection track 218 is sent for output on link 202 by output interface 120-1. At step 612, the track 124 established or updated based on model track 216 in an augmentation state is updated using information derived from the corresponding detection track 218 or detection 126. For example, heading, speed, rate of change of speed, distance change rate, or other object measurements can be updated using detection-based measurements of the same characteristics, which become more accurate over time than machine learning model estimates. In this way, a more complete set of radar-detectable information associated with the tracked object is used, rather than relying solely on data cube 122 or a limited portion of the data cube analyzed by object detector model 116-1.

[0118] The result of executing procedure 600-1 is that, before any part of the tracking is enhanced to transmit additional information derived from radar detection for rapid object transmission, the tracking from the radar system's output can initially include tracking built using information obtained from the model. The tracking is then enhanced using information derived from radar detection to transmit more accurate or more up-to-date object measurements (e.g., velocity-related) compared to those that the model can estimate alone from the data cube.

[0119] Enhanced detection and tracking

[0120] Figure 6-2 Process 600-2 demonstrates a combination of detection-based radar tracking and model-based tracking that considers the use of radar detection to enhance it. Also known as detection-tracking enhancement, the execution of process 600-2 can occur in response to the execution of step 516 and can lead to the execution of step 520. The tracking resolver 314 or tracking filter can consider the quality of the detected tracking 218 and / or detection 126 to determine whether any detection is in a state of detection-based enhancement for tracking 124, specifically, in a state that supports new tracking in tracking 124 output from radar system 104-1 or supports updating existing tracking therein.

[0121] At 616, it is determined whether the detection tracking is in a state considering detection-based enhancement. This may include determining whether the quality of the tracking established using additional information derived from radar detection meets a quality threshold for performing detection-based enhancement. The quality of the tracking established using the additional information derived from radar detection may initially be set below the quality threshold, and the quality of the tracking established using the additional information derived from radar detection may be improved as more radar detections are received during a frame or subsequent interval.

[0122] For example, if any detection tracks 218 are reported at a location far from model tracks 216, they may not be used to enhance model tracks 216, but instead, they may be used to support new tracks. In response to determining that the radar detections observed from signals received from the environment do not correspond to the same object as the tracks established using information obtained from the model, step 616 is performed to determine whether, in addition to using information obtained from the model, additional information derived from the radar detections is used to establish another track for a different object to support object tracking of the additional object, rather than object tracking of the object reported by the model from the radar data cube.

[0123] In response to determining that detection traces 218 are not yet in a state to be considered for use in detection-based enhancements, the "No" branch of step 616 flows to step 618. At 618, detection traces 218 not yet ready for object tracking are maintained for future consideration. In some examples, this maintenance involves estimating the degree of uncertainty regarding velocity and velocity-related states. Alternatively, if any detection trace 218 is in a state to be considered for detection-based enhancements, the "Yes" branch of step 616 flows to step 620. When a detection trace passes the check of step 616, any subsequent checks on that detection trace may be skipped during subsequent execution of step 616, or detection traces that have proven to be of sufficient quality may be automatically considered for detection-based tracking at step 620 (if that detection trace exists).

[0124] In some examples, before any detection tracking 218 includes stable and accurate object measurements (e.g., velocity state) for supporting object tracking and before the "yes" branch flows from 616 to 620, steps 616 and 618 may be repeated for a period of time (e.g., several frames, several seconds) for the curve fitting process. During this interval, the position measurements (e.g., velocity, velocity state) associated with detection tracking 218 stabilize to a sufficient quality or consistent value to be considered for object tracking.

[0125] Uncertainty-based decision-making can be performed instead of repeating the curve-fitting process for a period of time or using a fixed number of trackers in a loop. For example, detection track 218, which identifies and tracks objects not represented by track 124, can be queued for output to the vehicle-based system 200, while also outputting track 124 supported by model track 216. Track 124 can quickly transmit not only objects associated with model track 216, but also other detection-based objects represented by any track 124. This allows as many radar-detectable objects as possible within the field of view to be tracked and reported with minimal latency.

[0126] At 620, it is determined whether the detected tracking has been associated with the tracking output from the radar system. For example, if the quality of the detected tracking is sufficient to support object tracking, the tracking resolver 314 can promote the detected tracking to one of the tracks 124 output by the radar system 104-1. If no "No" branch is taken at 620, the detected tracking can be used to establish a new tracking at 622, and the detected tracking can be sent at steps 624 and 626 for outputting an update to one of the existing detection-based tracks 124.

[0127] At 622, information derived from radar detection is used to establish tracking for output. At 624, tracking is updated using information derived from radar detection, and then sent at 626 for output, so that it can be output in tracking 124.

[0128] The trace established for output at 622 or sent for output at 626 can be classified into an object category different from the object category identified by the model. The trace can therefore be an object category different from the object category of any trace established using information obtained from the model. In other words, the model (e.g., object detector model 116-1) may not be able to identify objects outside of a specific category or multiple categories from the data cube 122. Therefore, the trace established in step 612 is an object category different from the specific object category of the trace established using information obtained from the model.

[0129] The result of performing process 600-2 in conjunction with process 600-1 is that, in addition to some tracking with radar detection enhancements and other tracking established solely using radar detection, the tracking output from the radar system can include tracking established using information obtained from the model. In this way, most (if not all) categories of radar-detectable objects are trackable, not just those categories that the model cannot identify from the data cube.

[0130] Multi-layer object tracking

[0131] Figure 7 A flowchart of an example process is shown, which is used to maintain multiple tracking layers to facilitate radar tracking in order to support radar tracking using model estimation enhanced by radar detection. For ease of description, process 700 is mainly described by radar system 104 and its various examples (including radar systems 104-1 to 104-6, including...). Figure 4 The context described is as follows (the aspect of the trace parser 314 in the example). In this example, the operations (also called steps) of process 700 are numbered from 702 to 720. However, this numbering does not necessarily imply a specific order of operations. The steps of process 700 can be related to... Figure 7 The diagram shows different ways to rearrange, skip, repeat, or execute specific methods.

[0132] At 702, object measurements of the environment are obtained from the radar data cube-based model. At 704, radar detections of the environment are obtained, in addition to object measurements. At 706, the main tracking layer is maintained using information obtained from the model. The main tracking layer includes object tracking established using information obtained from the model. At 708, a secondary tracking layer is maintained using additional information derived from radar detections. The secondary tracking layer includes object tracking established using additional information derived from any other radar detections, in addition to information obtained from the model.

[0133] In steps 706 and 708, the primary and secondary tracing layers can be maintained by assigning an identifier to each trace in the primary tracing layer, such that the identifier is unique for each object in the secondary tracing layer. In this way, there are two sets of trace identifiers, one for the primary layer and one for the secondary layer, which facilitates the organization and resolution of traces 124 into a single set.

[0134] At 710, the tracking from the main tracking layer is output so that the vehicle can avoid objects while driving in the environment. That is, any tracking maintained in the secondary tracking layer is avoided before enhancing the tracking built using information obtained from the model. However, every tracking maintained in the main tracking layer is output.

[0135] At 712, it is determined whether any tracking in the secondary tracking layer meets the quality thresholds required to support tracking or enhancement. For example, it is determined whether the quality of the detected tracking is sufficient to support object tracking or enhancement. In response to determining that the quality of tracking in the secondary tracking layer is sufficient, the "Yes" branch of step 712 leads to steps 716 and / or 718. In response to determining that the quality of tracking in the secondary tracking layer is insufficient, the "No" branch of step 712 leads to step 714.

[0136] At step 714, the trace parser 314 can prevent or enhance trace 124 from being output using any trace of insufficient quality from the secondary trace layer.

[0137] At step 716, the tracking resolver 314 can enhance portions of the tracking from the primary tracking layer using additional information derived from quality detection tracking on the secondary tracking layer. In some examples, to support enhancement of existing tracking in the primary tracking layer by promoting tracking from the secondary tracking layer to the primary tracking layer, the identifier of the tracking established using information derived from radar detection maintained in the secondary tracking layer can be unassigned. Unassigning this identifier eliminates the consideration of the object as separate from the object in the primary tracking layer. The unassigned identifier can be applied to new tracking maintained in the primary tracking layer or new tracking maintained in the secondary tracking layer.

[0138] At step 718, the tracking resolver can elevate the detected tracking to one of the tracking 124 output by radar system 104-1, and subsequently include it in the main tracking layer. This may include removing tracking established using additional information derived from another radar detection from the secondary tracking layer in response to enhancing tracking of objects established using information obtained from the model. If the detected tracking cannot be used to enhance the portion of tracking from the main tracking layer at step 716, the detected tracking can be used at 718 to establish a new tracking, and in this case, the detected tracking can be elevated to the main tracking layer. By performing step 718, tracking 124 transmits additional objects that need to be avoided while driving in the environment, beyond those objects transmitted by object measurements obtained from the model.

[0139] At 720, tracking is established using information derived from radar detection for output. At 624, tracking is updated using information derived from radar detection, and then sent at 626 for output, to be output in tracking 124. This can include enhanced tracking output from the main tracking layer, enabling vehicles to avoid objects more accurately compared to driving based on unenhanced tracking output from the main tracking layer.

[0140] By executing process 700, the radar system can maintain two layers or two sets of tracking, which, when intelligently and systematically parsed according to the aforementioned technology, can transmit a highly accurate tracking representation of the environment without delay. This is achieved by utilizing high-performance and fast object recognition performed from some machine learning model, and by utilizing detection-based object tracking technology to provide a more accurate and higher-fidelity representation of the radar system's field of view. With a more accurate representation of the environment, the vehicle subsystem can make safer driving decisions, resulting in a more comfortable and safer experience for occupants, other vehicles, and other people or objects sharing the road.

[0141] The results of process 700 can be used by radar system 104 to output a series of tracks for operating vehicle 102 in environment 100. These tracks include each of the tracks from the primary tracking layer and at least one other output from a secondary tracking layer that does not correspond to any tracked object output from the primary tracking layer.

[0142] Example use cases

[0143] Figure 8 A flowchart of an example process 800 for operating a vehicle using radar tracking estimated by a model enhanced by radar detection is shown. For ease of description, it is mainly described in the form of... Figure 2Process 800 is described in the context of a vehicle-based system 200, for example, to facilitate the execution of processes 500, 600-1, 600-2, and / or 700 by radar system 104. Other devices and systems that act as receivers of radar data to track objects in the environment may also execute process 800. The operation of process 800 may be rearranged, skipped, repeated, or executed in ways different from the specific flow shown in the figure.

[0144] At 802, radar data, including radar tracking, is received. Radar tracking is generated based on model estimation enhanced by radar detection. In addition to tracking other radar-detectable objects that the model cannot identify from the data cube, radar tracking also includes tracking of objects that the model directly identifies from the data cube. For example, vehicle-based system 200 may receive tracking 124 output from radar system 104-1. Within tracking 124, objects associated with each model tracking 216 are transmitted via output interface 120-1, in some cases, fields of enhanced information based on one or more corresponding detection tracking 218. Output interface 120-1 may further output one or more tracking of objects within tracking 124, which correspond to any detection tracking 215 that can be tracked individually or in combination as radar-detectable objects that object detector model 116-1 cannot detect.

[0145] At 804, the vehicle is driven by avoiding objects transmitted in radar tracking. For example, tracking 124 can be processed by vehicle-based system 200 to enable vehicle 102-1 to perform driving maneuvers and avoid objects in environment 100 when they are reported in tracking 124. Tracking 124 can indicate object size, object type, speed, position, rate of change of distance, etc., not only for object types that can be detected by object detector model 116, but also for additional radar-detectable objects not reported by object detector model 116 but inferred from detection 126. Even if tracking 124 is for one or more of the objects estimated by object detector model 116-1, tracking 124 can include updates or more accurate measurements for some attributes by deriving more accurate estimates (e.g., speed) from parts of data cube 122 that are otherwise ignored by object detector model 116-1.

[0146] Further examples

[0147] Example 1. A method comprising: establishing, by a radar system, a tracking of an object in an environment using information derived from a model, the information derived from the model including estimated object measurements from a radar data cube representation of signals received from the environment; determining, in addition to the information derived from the model, whether a radar detection observed from signals received from the environment corresponds to the same object as the tracking established using the information derived from the model; and, in response to determining that the radar detection corresponds to the same object as the tracking established using the information derived from the model, using additional information derived from the radar detection to enhance a portion of the tracking initialized based on the object measurements derived from the model, to improve the accuracy or detail associated with said portion of the tracking.

[0148] Example 2. The method of any of the preceding examples further includes: outputting a tracking established using information obtained from the model before the portion of the enhanced tracking is performed; and updating the tracking output to transmit the portion of the enhanced tracking using additional information derived from radar detection.

[0149] Example 3. The method of any of the preceding examples further includes: determining whether another radar detection observed from signals received from the environment corresponds to the same object as a track established using information obtained from the model; and in response to determining that another radar detection observed from signals received from the environment does not correspond to the same object as a track established using information obtained from the model, avoiding the use of additional information derived from the other radar detection to enhance any part of the track initially set up based on object measurements obtained from the model.

[0150] Example 4. As in any of the preceding examples, the method further includes: determining whether another radar detection observed from signals received from the environment corresponds to the same object as a track established using information obtained from the model; and in response to determining that the other radar detection observed from signals received from the environment does not correspond to the same object as a track established using information obtained from the model, establishing another track for the other object using additional information derived from the other radar detection, in addition to the information obtained from the model, to support object tracking for additional objects besides those reported by the model in response to analyzing the radar data cube.

[0151] Example 5. The method of any of the preceding examples further includes: before using additional information derived from radar detection to enhance the portion of the track to transmit the portion of the track, outputting a track built using information obtained from the model; and outputting another track built using information derived from another radar detection to transmit objects of additional object categories that the model cannot identify from the data cube.

[0152] Example 6. As in any of the preceding examples, where the tracked object established using information obtained from the model corresponds to a specific object category, and another tracked object established using information derived from another radar detection corresponds to an object category different from the specific object category of the tracked object established using information obtained from the model.

[0153] Example 7. As in any of the previous examples, where the model cannot identify any object from the radar data cube that corresponds to the following object category: the object category is different from the specific object category of the tracked object established using information obtained from the model.

[0154] Example 8. As in any of the preceding examples, wherein enhancing the portion of the tracking initialized based on object measurements obtained from the model to improve the accuracy or detail associated with the tracked portion is further in response to: determining that the quality of the tracking established using additional information derived from radar detection meets a quality threshold for performing detection-based enhancement.

[0155] Example 9. As in any of the previous examples, where the quality of the tracking established using additional information derived from radar detection is initially set below a quality threshold, and the quality of the tracking established using additional information derived from radar detection improves as more signals are received by radar detection.

[0156] Example 10. The method as in any of the preceding examples further includes: determining the quality of said portion of the trace built using information from the model; and

[0157] In response to determining that the quality of the tracked portion meets another quality threshold for performing detection-based enhancement, the tracked portion is enhanced based on object measurements obtained from the model.

[0158] Example 11. The method of any of the preceding examples further includes: maintaining a primary tracking layer using information obtained from the model, the primary tracking layer including, in addition to any other tracking established using information obtained from the model, tracking of objects established using information obtained from the model; and maintaining a secondary tracking layer using, in addition to information obtained from the model, additional information derived from the radar detection, additional information derived from the other radar detection, and additional information derived from any other radar detection, the secondary tracking layer including: tracking established using additional information derived from the radar detection, the other tracking established using additional information derived from the other radar detection, and any other tracking established using additional information derived from any other radar detection.

[0159] Example 12. The method of any of the preceding examples further includes: avoiding any traces whose output is maintained in a secondary tracking layer before enhancing the tracking of the object built using information obtained from the model, while the output is maintained in each trace in the main tracking layer.

[0160] Example 13. The method of any of the preceding examples further includes: in response to enhancing the tracking of an object built using information obtained from the model, removing the tracking built using additional information derived from the other radar detection from the secondary tracking layer.

[0161] Example 14. The method of any of the preceding examples further includes: outputting a series of traces for operating a vehicle in an environment, the series of traces including: each of the traces from a primary tracing layer, and at least one other trace from a secondary tracing layer that does not correspond to any of the traces output from the primary tracing layer.

[0162] Example 15. As in any of the preceding examples, where maintaining the primary tracing layer and maintaining the secondary tracing layer includes: assigning an identifier to each trace in the primary tracing layer, the identifier being unique for each trace assigned to the secondary tracing layer.

[0163] Example 16. The method as in any of the preceding examples, wherein maintaining the primary tracking layer and maintaining the secondary tracking layer includes: deassigning the identifier to a tracking established using additional information derived from the radar detection and maintained in the secondary tracking layer, to allow the deassigned identifier to be applied to a new tracking maintained in the primary tracking layer or a new tracking maintained in the secondary tracking layer.

[0164] Example 17. A method comprising: tracking objects in an environment based on signals received from the environment of a vehicle using a combination of a model-based radar tracker and a detection-based radar tracker, the tracking comprising: maintaining a primary tracking layer including tracking established using information obtained from a model, the information including estimated object measurements based on a radar data cube representation of the signals received from the environment; and maintaining a secondary tracking layer including: tracking established using additional information derived from radar detection identified in addition to the signals received from the environment; and avoiding outputting any of the tracking from the secondary tracking layer until the additional information derived from radar detection satisfies a quality threshold for performing detection-based enhancement of a portion of the tracking in the primary tracking layer; and outputting the tracking in the primary tracking layer such that the vehicle can avoid objects while driving in the environment.

[0165] Example 18. The method of any of the preceding examples further includes: in response to determining that additional information derived from radar detection satisfies a quality threshold for performing detection-based enhancement of a portion of the tracking in the main tracking layer, using the additional information derived from radar detection to enhance the portion of the tracking in the main layer to improve the accuracy or detail associated with that portion of the tracking, compared to an initial setting based on object measurements obtained from the model.

[0166] Example 19. A radar system including at least one processor configured to: establish tracking of an object in an environment using information obtained from a model, the information obtained from the model including: object measurements estimated based on a radar data cube representation of signals received from the environment; determining, in addition to the information obtained from the model, whether a radar detection observed from signals received from the environment corresponds to the same object as the tracking established using the information obtained from the model; and in response to determining that the radar detection corresponds to the same object as the tracking established using the information obtained from the model, using additional information derived from the radar detection to enhance a portion of the tracking initialized based on the object measurements obtained from the model, to improve the accuracy or detail associated with that portion of the tracking.

[0167] Example 20. A radar system of any of the preceding examples, wherein at least one processor is further configured to: maintain a primary track, which, in addition to any other track established using information obtained from a model, includes: a track of an object established using information obtained from a model; maintain a secondary tracking layer, which, in addition to any other track established using additional information obtained from, but not from, any other radar detection, besides the model, includes: a track established using additional information derived from the radar detection; further, in response to determining that the radar detection and the track established using information obtained from the model correspond to the same object, remove the track established using the additional information derived from the radar detection from the secondary tracking layer; and output a radar track for operating a vehicle in an environment, the radar track including: each of the tracks from the primary tracking layer, and at least one other track from the secondary tracking layer that does not correspond to any of the tracks output from the primary tracking layer.

[0168] Example 21. A radar system of any of the preceding examples, wherein at least one processor is configured to perform any of the methods in the preceding examples.

[0169] Example 22. A system including means for performing any method in the method of the preceding example.

[0170] Example 23. A computer-readable storage medium including instructions that, when executed, cause at least one processor to perform any of the methods as in the previous examples.

[0171] Conclusion

[0172] While various embodiments of the present disclosure have been described in the foregoing description and illustrated in the accompanying drawings, it should be understood that the present disclosure is not limited thereto, but can be practiced in various ways within the scope of the following claims. It will be apparent from the foregoing description that various modifications can be made without departing from the scope of the present disclosure as defined by the appended claims. In addition to radar systems, problems related to object recognition can arise in other systems (e.g., imaging systems, lidar systems, ultrasonic systems) that identify and process tracking from various sensors. Therefore, although described as improving radar tracking, the techniques described above can be adapted and applied to other problems to effectively detect and track objects in a scene using other types of object trackers.

[0173] Unless the context clearly specifies otherwise, the use of "or" and grammatically related terms indicates an unrestricted, non-exclusive alternative. As used herein, the phrase referring to "at least one" of a list of items means any combination of those items, including a single member. For example, "at least one of a, b, or c" is intended to cover: a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

Claims

1. A method of radar tracking, the method comprising: establishing, by a radar system, a track of an object in an environment using information acquired from a model, the information acquired from the model including object measurements estimated from a radar datacube representation of signals received from the environment; determining, in addition to the information acquired from the model, whether a radar detection observed from the signals received from the environment corresponds to the same object as the track established using the information acquired from the model; responsive to determining that the radar detection corresponds to the same object as the track established using the information acquired from the model, augmenting a portion of the track initialized from the object measurements acquired from the model using additional information derived from the radar detection to improve accuracy or detail associated with the portion of the track; determining whether another radar detection observed from the signals received from the environment corresponds to the same object as the track established using the information acquired from the model; and responsive to determining that the other radar detection observed from the signals received from the environment does not correspond to the same object as the track established using the information acquired from the model, establishing, in addition to using the information acquired from the model, another track of another object using additional information derived from the other radar detection to support object tracking of additional objects other than objects reported by the model responsive to analyzing the radar datacube.

2. The method of claim 1, wherein, further comprising: outputting the track established using the information acquired from the model prior to augmenting the portion of the track; and updating the track output to convey the portion of the track augmented using the additional information derived from the radar detection.

3. The method of claim 1, wherein, further comprising: further responsive to determining that the other radar detection observed from the signals received from the environment does not correspond to the same object as the track established using the information acquired from the model, refraining from augmenting any portion of the track initially set from the object measurements acquired from the model using additional information derived from the other radar detection.

4. The method of claim 1, wherein, further comprising: outputting the track established using the information acquired from the model prior to augmenting the portion of the track using the additional information derived from the radar detection to convey the portion of the track; and outputting the other track established using the information derived from the other radar detection to convey objects of additional object classes not identifiable by the model from the datacube.

5. The method of claim 1, wherein, the object of the track established using the information acquired from the model corresponds to a particular object class, and the other object of the other track established using the information derived from the other radar detection corresponds to an object class different from the particular object class of the object of the track established using the information acquired from the model.

6. The method of claim 5, wherein, The model fails to identify any objects from the radar datacube that correspond to an object class that is different from the particular object class of the object for which the track is established using the information obtained from the model.

7. The method of claim 1, wherein, Enhancing the portion of the track initialized from the object measurements obtained from the model to improve accuracy or detail associated with the portion of the track is further responsive to determining that a quality of the track established using the additional information derived from the radar detections satisfies a quality threshold for performing detection-based enhancement.

8. The method of claim 7, wherein, The quality of the track established using the additional information derived from the radar detections is initially set below the quality threshold, and the quality of the track established using the additional information derived from the radar detections improves as more signals are received by the radar detections.

9. The method of claim 7, wherein, Further comprising: determining a quality of the portion of the track established using the information from the model; and enhancing the portion of the track initialized from the object measurements obtained from the model in response to determining that the quality of the portion of the track satisfies another quality threshold for performing detection-based enhancement.

10. The method of claim 1, wherein, Further comprising: maintaining a primary tracking layer using the information obtained from the model, the primary tracking layer including, in addition to any other tracks established using the information obtained from the model, the track of the object established using the information obtained from the model; and maintaining a secondary tracking layer using, in addition to the information obtained from the model, additional information derived from the radar detections, additional information derived from the other radar detections, and additional information derived from any other radar detections, the secondary tracking layer including tracks established using additional information derived from the radar detections, the other track established using additional information derived from the other radar detections, and any other tracks established using additional information derived from any other radar detections.

11. The method of claim 10, wherein, Further comprising: avoiding outputting any tracks maintained in the secondary tracking layer while each track maintained in the primary tracking layer is outputted prior to enhancing the track of the object established using the information obtained from the model.

12. The method of claim 10, wherein, Further comprising: removing the track established using the additional information derived from the other radar detections from the secondary tracking layer in response to enhancing the track of the object established using the information obtained from the model.

13. The method of claim 10, wherein, Further comprising: outputting a succession of tracks for operating a vehicle in the environment, the succession of tracks including each of the tracks from the primary tracking layer, and at least one other track from the secondary tracking layer that does not correspond to an object of any of the tracks outputted from the primary tracking layer.

14. The method of claim 10, wherein, Maintaining the primary tracking layer and maintaining the secondary tracking layer includes assigning an identifier to each track in the primary tracking layer, the identifier being unique from an identifier assigned to each track in the secondary tracking layer.

15. The method of claim 14, wherein, Maintaining the primary tracking layer and maintaining the secondary tracking layer includes de-assigning the identifier from the track maintained in the secondary tracking layer established using the additional information derived from the radar detection to allow the de-assigned identifier to be applied to a new track maintained in the primary tracking layer or a new track maintained in the secondary tracking layer.

16. A method of radar tracking, the method comprising: tracking objects in an environment of a vehicle based on signals received from the environment using a combination of a model-based radar tracker and a detection-based radar tracker, the tracking including: maintaining a primary tracking layer including tracks established using information obtained from a model, the information including: object measurements estimated from a radar data-cube representation of the signals received from the environment; and maintaining a secondary tracking layer including: tracks established using additional information derived from radar detections identified from the signals received from the environment other than from the model; and avoiding outputting any of the tracks from the secondary tracking layer until the additional information derived from radar detections satisfies a quality threshold for performing detection-based augmentation of a portion of the tracks in the primary tracking layer; outputting the tracks in the primary tracking layer to enable the vehicle to avoid the objects while driving in the environment.

17. The method of claim 16, wherein, further comprising: in response to determining that the additional information derived from radar detections satisfies a quality threshold for performing detection-based augmentation of a portion of the tracks in the primary tracking layer, augmenting a portion of the tracks in the primary tracking layer using the additional information derived from the radar detections to improve accuracy or detail associated with the portion of the tracks, the portion of the tracks in the primary tracking layer being improved compared to an initial setting based on the object measurements obtained from the model.

18. A radar system comprising at least one processor configured to: establishing a tracking of objects in the environment using information obtained from a model, the information obtained from the model including: object measurements estimated from a radar data-cube representation of signals received from the environment; determining, other than from the information obtained from the model, whether a radar detection observed from the signals received from the environment corresponds to a same object as the track established using the information obtained from the model; in response to determining that the radar detection corresponds to a same object as the track established using the information obtained from the model, augmenting a portion of the track initialized from the object measurements obtained from the model using additional information derived from the radar detection to improve accuracy or detail associated with the portion of the track; determining whether another radar detection observed from the signals received from the environment corresponds to the same object as the tracking established using the information obtained from the model; and in response to determining that the other radar detection observed from the signals received from the environment does not correspond to the same object as the tracking established using the information obtained from the model, establishing another tracking of another object using additional information derived from the other radar detection in addition to using the information obtained from the model to support object tracking of additional objects in addition to the objects reported by the model in response to analyzing the radar datacube.

19. The radar system of claim 18, wherein, The at least one processor is further configured to: maintain a primary tracking layer that includes the tracking of the object established using the information obtained from the model in addition to any other tracking established using the information obtained from the model; maintain a secondary tracking layer that includes tracking established using additional information derived from radar detections other than the model in addition to any other tracking established using the additional information obtained from the radar detections other than the model; in further response to determining that the radar detection corresponds to the same object as the tracking established using the information obtained from the model, remove from the secondary tracking layer the tracking established using the additional information derived from the radar detection; and output tracking for use in operating a vehicle in the environment, the tracking including each of the tracking from the primary tracking layer and at least one other tracking from the secondary tracking layer that does not correspond to any of the tracking output from the primary tracking layer.

Images