Methods and devices for multi-radar, multi-frame ego-motion estimation

Abstract

A system may include a control circuit including couple to a plurality of sensor circuits of a vehicle to receive unsynchronized Ego-motion data including data indicative of objects in a view area of one or more of the sensor circuits. The system may include processor circuits configured to concatenate the unsynchronized Ego-motion data from at least some of the sensor circuits and select a subset of the unsynchronized Ego-motion data. The processor circuits may use a deep neural network to fuse the subset and determine acceleration, Kalman gain, and dynamic weight estimates based on the fused subset. The processors may compensate the subset based on the acceleration estimates and filter and smooth the subset and the compensated subset using the Kalman gain estimates and the dynamic weight estimates to produce vehicle Ego-motion data corresponding to transitional motion and rotational motion of the vehicle and relative motion of the objects.

Description

METHODS AND DEVICES FOR MULTI-RADAR, MULTI-FRAME EGO-MOTIONESTIMATIONFIELD OF USE

[0001] The present disclosure generally relates to systems that are configured to determine moving and stationary objects in an egocentric coordinate system where the system is moving, and more particularly, to systems, methods, and devices for multi-radar, multi-frame Ego-motion estimation.BACKGROUND

[0002] Motion estimation refers to object detection and vehicle motion (ego-motion), which is important for self-driving vehicles and for driver-assist functionality, such as adaptive cruise control, emergency braking, and other functionality. Automotive radar plays an important role as a sensing modality for providing environmental perception capability to enable safe-driving functions. In the field of radar, Ego-motion estimation relates to the determination of the position and motion of a radar apparatus based on data captured by the radar apparatus.BRIEF DESCRIPTION OF THE DRAWINGS

[0001] The detailed description is set forth with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures and in the detailed description indicates similar or identical items or features.

[0003] FIG. 1 depicts a system including a vehicle with multiple sensors configured to monitor multiple view areas surrounding the vehicle to determine objects, in accordance with certain embodiments of this disclosure.

[0004] FIG. 2 depicts a block diagram of a portion of the system including Ego-motion estimators and a temporal fusion and smoothing component configured to determine Ego-motion of the vehicle based on sensor data, in accordance with certain embodiments of this disclosure.

[0005] FIG. 3 depicts a block diagram of a system including one or more processor circuits coupled to sensor circuits and actuator circuits, in accordance with certain embodiments of this disclosure.

[0006] FIG. 4 depicts a block diagram of an embodiment of one of the Ego-motion estimators of FIG. 2, in accordance with certain embodiments of this disclosure.

[0007] FIG. 5 depicts a block diagram of an embodiment of the temporal fusion and smoothing component of FIG. 2, in accordance with certain embodiments of this disclosure.

[0008] FIG. 6 depicts a graph of the translational velocity (in meters / second) versus time (in seconds) determined by the systems of FIGs. 1-5, in accordance with certain embodiments of this disclosure.

[0009] FIG. 7 depicts a graph of the radial velocity (in meters / second) versus angle of arrival (in radians) determined by the systems of FIGs. 1-5, in accordance with certain embodiments of this disclosure.

[0010] FIG. 8 depicts a flow diagram of a method of determining Ego-motion estimates with timestamps, in accordance with certain embodiments of this disclosure.

[0011] FIG. 9 depicts a flow diagram of a method of determining Ego-motion of a vehicle based on a plurality of Ego-motion estimates with timestamps, in accordance with certain embodiments of this disclosure.

[0012] While implementations are described in this disclosure by way of example, those skilled in the art will recognize that the implementations are not limited to the examples or figures described. Rather, the figures and detailed description thereto are not intended to limit implementations to the form disclosed, but instead the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope as defined by the appended claims. The headings used in this disclosure are for organizational purposes only and are not meant to limit the scope of the description or the claims. As used throughout this application, the word “may” is used in a permissive sense (in other words, the term “may” is intended to mean “having the potential to”) instead of in a mandatory sense (as in “must”). Similarly, the terms “include,” “including,” and “includes” mean “including, but not limited to.”DETAILED DESCRIPTION

[0013] Embodiments of systems, methods, and devices are described herein that include a multi-radar, multi-frame Ego-motion estimation technology configured to use data from a radar sensor network, which may be asynchronous. The system may be configured to temporally fuseand smooth radar data from multiple radar platforms to enhance overall accuracy and robustness, achieving performance that can match dedicated sensors. The system may provide a number of advantages over conventional Ego-motion estimation systems. First, the system may operate with unsynchronized radar sensors. Second, there is no limitation of radar configurations (location, field of view, resolution, etc.), because the system can operate on unsynchronized radar sensor data and because the system is configured to smooth and fuse such data from various sources. Third, the ego-motion parameters are estimated with temporal information, enabling the system to work under adverse conditions: such as sensor data loss, sensor failure, sensor network out-of-sync, and other problematic conditions. Fourth, unlike other traditional multi-frame ego- estimation technologies, there is no frame aggregation or data association required. Instead, estimated Egomotion parameters are temporally fused, filtered, and smoothed, and the resource requirement and latency are comparable to single-frame methods. Another advantage may include using deep neural network (DNN) estimators that can be integrated into the radar sensor circuits. Such DNN estimators may learn or may be configured to correct misestimation (over-estimation or underestimation) of Doppler measurements due to changes in the vehicle’s velocity over time (i.e., acceleration or deceleration).

[0014] Automotive radars are common on modern vehicles. It is getting more and more popular that multiple radar sensors are used for different applications, such as adaptive cruise control, cross traffic alert, blind spot detection, and numerous other applications. Radar sensors can operate in extreme weather conditions; are less sensitive to lighting conditions than some other types of sensors (such as optical sensors or cameras); and can illuminate targets that are outside of the direct line of sight.

[0015] Embodiments of systems, methods, and devices are described below that may include multiple sensors including multiple radar circuits, each of which may be configured to capture data indicative of objects in a respective view area. Each of the multiple radar circuits may include a deep neural network (DNN) Ego-motion estimator that is configured to determine Ego-motion estimates with timestamps for the radar circuit. Each of the radar circuits may output the determined Ego-motion estimates with time stamps, which may be unsynchronized, and which may be presented as a sequence of radar data. A temporal fusion and smoothing component may process subsets of the sequence to determine Ego-motion data for the vehicle, which may be used for various system applications, such as emergency braking, adaptive cruise control, blind spotwarning signals, cross-traffic alerts, lane correction, or other applications. In one or more embodiments, the Ego-motion data may be used for autonomous vehicle control or for driverassistance applications. An example of a system including multiple radar sensors and other sensors is described below with respect to FIG. 1.

[0016] FIG. 1 depicts a system 100 including a vehicle 102 with multiple sensor circuits 106 configured to monitor multiple view areas surrounding the vehicle to determine objects, in accordance with certain embodiments of this disclosure. The vehicle 102 is an example of an embodiment of a system 100, which may include multiple sensor circuits 106, such as multiple radar circuits, optical sensor circuits, light-detection and ranging (LIDAR) circuits, ultrasound or ultrasonic sensor circuits, other sensor circuits, or any combination thereof. The sensor circuits 106 may be configured to determine objects associated with the vehicle’s immediate surroundings, objects within the vehicles path (forward, backward, or turning), cross-traffic, objects, and so on. Sensor circuits 106 may also include other types of sensors, such as temperature sensors, pressure sensors, gyroscopic sensors, motion sensors, orientation sensors, other sensors, or any combination thereof.

[0017] In the illustrated example, the view areas of some of the sensor circuits 106 are shown, including a long-range radar view area 108 that corresponds to a physical area that may be sensed by a long-range radar circuit based on various parameters, such as its orientation, frequency range, sensor sensitivity, other parameters, or any combination thereof. The long-range radar circuit may include one or more transmitters configured to transmit electromagnetic energy in short pulses in a selected direction and at one or more selected frequencies. The short pulses may be reflected by objects in the path of the short pulses. The long-range circuit may include one or more sensors or receivers configured to receive the reflected signals and to determine objects in the long-range radar view area 108 based on the reflected signals. The objects detected within the long-range radar view area 108 may include other vehicles that may be stopped or may be moving slower than the vehicle 102, and data related to one or more detected objects in the long-range radar view area 108 may be used for various functions, such as adaptive cruise control, early braking, driver alerting applications, collision avoidance (e.g., object on the road), and so on.

[0018] The view areas of the sensor circuits 106 may include a light-detection and ranging (LIDAR) view area 110 that corresponds to a physical area that may be sensed by a LIDAR circuitbased on various parameters, such as orientation, light-emitting frequencies, sensor sensitivity, other parameters, or any combination thereof. The objects detected within the LIDAR view area 110 may include other vehicles that may be stopped or may be moving slower than the vehicle 102, pedestrians, and so on. Data related to one or more detected objects in the LIDAR view area 110 may be used for various functions, such as emergency braking, pedestrian detection, collision avoidance, and so on.

[0019] The view areas of the sensor circuits 106 may include one or more optical sensors view areas 112. In the illustrated example, the system 100 may include an optical sensor front view area 112(1), optical sensor side view areas 112(2) and 112(3), and optical sensors rear view area 112(4). In one or more embodiments, the optical sensors view areas 112 may correspond to a physical area that may be sensed or viewed by sensors of one or more optical sensor circuits. One or more of the optical sensor circuits may be forward-facing and may capture optical data associated with one or more optical sensors front view area 112(1), which data may may include traffic signs, lane markings, and other optical data that may be used for traffic sign recognition, lane departure warnings, steering assistance, and other functionality. One or more of the optical sensor circuits may be side-facing and may capture optical data associated with one or more optical sensors side view areas 112(2) and 112(3). One or more optical sensor circuits may be rear-facing and may capture optical data associated with optical sensors rear view area 112(4), which data may be used for parking assistance and at least a portion of which may be presented on a display within the vehicle 102 when backing up.

[0020] The view areas of the sensor circuits 106 may include short-medium range radar view areas 114. In the illustrated example, the system 100 may include a short-medium range radar front view area 114(1), short-medium range radar side view areas 114(2) and 114(3), and a shortmedium range radar rear view area 114(4). In one or more embodiments, the short-medium range view areas 114 may correspond to a physical area that may be sensed or viewed by sensors of one or more short-medium range radar circuits. In one or more embodiments, the short-medium range radar view areas 114 may be configured to determine cross-traffic data that can be used to generate an alert, to determine objects in blind spots (locations along the sides and behind the vehicle 102 that cannot be seen with the vehicle’s side mirrors or rear view mirror), to determine an object about to collide with the rear of the vehicle 102 to provide a rear collision warning, or to generate data for other purposes.

[0021] The view areas of the sensor circuits 106 may include one or more ultrasonic sensors front view area 116(1) and ultrasonic sensors rear view area 116(2), which may be sensed or viewed by sensors of one or more ultrasonic sensor circuits. The ultrasonic sensors may provide data indicative of objects, which data may be used for driver assistance, such as parking assistance or other near-object detection functionality.

[0022] In one or more embodiments, the vehicle 102 may include circuitry 104 that may include or that may be coupled to the one or more sensor circuits 106. The circuitry 104 may include processing circuitry, which may include logic circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), one or more processors, artificial intelligence (Al) accelerators, other circuitry, or any combination thereof. In one or more embodiments, the processing circuitry may be configured to receive sensor data from the multiple sensor circuits 106 and to process the sensor data to determine Ego-motion data for the vehicle 102. In one or more embodiments, the circuitry 104 may include a memory configured to store processor-executable instructions and data.

[0023] In one or more embodiments, the circuitry 104 may be configured to utilize an artificial intelligence (Al)-based Ego-motion estimator configured to temporally fuse and smooth unsynchronized radar data from multiple radar sources to determine the Ego-motion data for the vehicle 102. The circuitry 104 may receive sensor data from the multiple sensor circuits 106, which may have different sampling rates, different resolutions, different ranges, and so on, and which may be received at different times. In one or more embodiments, one or more of the sensor circuits 106 may include a deep neural network (DNN) Ego-estimator that is integrated with the sensor platform to produce Ego-estimates with time stamps from each of the one or more sensors. The circuitry 104 may include a DNN Ego-estimator that is trained using data from all of the radar circuits to translate the cloud points from the radar circuits to objects projected to vehicle coordinates. The DNN Ego-estimator may run on the processor of each radar, on a zonal / bridge processor circuit, or both.

[0024] The DNN Ego-estimator may generate point-wise weights and point-wise offsets for each radar point-cloud. To reduce computational cost and the impact of data outliers, the DNN Ego-estimator may select only the k points with the highest weights, along with their original features (such as the radar cross section (RCS), doppler estimates, and other features), and providethe selected data to a weighted least square (WLS) regression component to determine estimated Ego-motion data, which may be provided as an Ego-motion estimate with a timestamp and a radar identifier (ID). The estimated Ego-motion estimate with the timestamp and the radar ID from each radar circuit may be concatenated to form a feature vector represented by a sequence of Egomotion estimates.

[0025] The circuitry 104 may include a temporal fusion and smoothing circuit, which may be configured to select a subset of the sequence of Ego-motion estimates using an adjustable window of N-frames or a selected time interval T and to process the subset. The temporal fusion and smoothing circuit may project features from the Ego-motion estimates to a vehicle-coordinate feature space, extract fused temporal features from the Ego-motion estimation sequence and predict parameters (acceleration, gain, weights, etc.) for a Kalman filter. The temporal fusion and smoothing circuit may compensate the subset using the predicted acceleration parameters and filter the extracted fused temporal features and the compensated subset using a Kalman filter configured using the predicted gain parameters. The temporal fusion and smoothing circuit may process the output of the Kalman filter using the predicted weights to produce the Ego-motion data for the vehicle 102.

[0026] Fusing the output data from the multiple radar circuits (e.g., long-range radar circuits, short-medium range radar circuits) and other sensors (e.g., the LIDAR circuits, optical sensor circuits, and ultrasound circuits) may enhance object discrimination capabilities, reduce computational costs, and increase the reliability of the object detection functionality. Embodiments of systems, methods, and circuits (such as the circuitry 104) are described herein that may be configured to fuse, smooth, and filter unsynchronized Ego-motion estimates from multiple radar circuits to determine Ego-motion data for the vehicle 102.

[0027] FIG. 2 depicts a block diagram of a portion of the system 200 including deep neural network (DNN) Ego-motion estimators 222 and a temporal fusion and smoothing component 210 configured to determine Ego-motion of the vehicle 102 based on sensor data, in accordance with certain embodiments of this disclosure. The temporal fusion and smoothing component 210 may be implemented as a circuit, may be implemented in processor-readable instructions that may be executed by a processor circuit, or any combination thereof. The system 200 includes a plurality of radar platforms, including a first sensor platform (Rl) 202(1) and an n-th sensor platform202(n). Each sensor platform 202 may include a front-end 214, which may be configured to receive signals from a sensor or antenna. In the following discussion, the sensor platforms 202 are assumed to be radar platforms that may be coupled to one or more antennas. However, it should be understood that one or more of the sensor platforms 202 may include a LIDAR platform, a radar platforms, or both.

[0028] The radar front-end 214 may include a transmit path and a receive path. The transmit path may be configured to receive a signal from a radar controller (not shown), which may be received at an input of a chirp generator circuit. The chirp generator circuit may generate one or more signal pulses or chirps, which may be provided a radio-frequency (RF) conditioning block. The RF conditioning block may condition the signal pulses or chirps by performing one or more of a filtering operation, a phase shift operation, a frequency operation, or an amplitude scaling operation on the pulses or chirps. The RF conditioning block may provide the conditioned signal to a power amplifier, which may amplify the condition signal and which may provide the amplified signal to an antenna for transmission.

[0029] The radar front-end 214 may include the receive path configured to receive an analog antenna signal corresponding to reflections from the transmission field, which reflections may be indicative of objects in the view area from an antenna, which may be the same antenna used to transmit the signal or a different antenna. The receive path may include a low-noise amplifier (LNA) including an input coupled to the antenna to receive the analog antenna signal, to amplify the received signal, and to provide the amplified signal to a Deramp mixer, which may mix the transmit signal with the amplified signal from the output of the LNA. The Deramp mixer may produce an output signal including one or more unique frequency components representative of one or more objects in the view area of the transmitted radar signal. Each unique frequency component may depend on the time of arrival of the reflected signal.

[0030] The radar front-end 214 may include a high-pass filter including an input coupled to the output of the Deramp mixer and including an output. The radar front-end circuit 214 may include a programmable amplifier including an input coupled to the output of the high-pass filter and including an output. The radar front-end circuit 214 may include a low-pass filter including an input coupled to the output of the programmable amplifier and including an output coupled toan input of an analog-to-digital converter (ADC), which provides a digital output indicative of the filtered signal.

[0031] Each radar platform 202 may include a radar receive-path microprocessor (MCP) 216 configured to process the received digital signals from the radar front-end 214 to determine fasttime range spectrum data and slow-time Doppler spectrum data. In one or more embodiments, the radar MCP may detect one or more range-Doppler cells and may construct a multi-input multioutput (MIMO) array including an array measurement vector. In one or more embodiments, the radar MCU 216 may be configured to determine direction of arrival information for one or more objects the view area based on the array measurement vector data.

[0032] Each radar platform 202 may include a pre-processing unit 218 configured to add a time-stamp and radar identifier to the data determined by the radar MCU 216 to produce a radar point cloud 220, including a plurality of data indicative of objects in a field of view of the radar platform 202. The radar point cloud P is a multi-dimensional point cloudwhere J is the number of detected points, n is the radar index, and AT is the number of features (local and global) of each point of the cloud.

[0033] For the vehicle 102, the direction of the X-axis corresponds to the down-range motion of the vehicle 102, and the direction of the y-axis coincides with the cross-range motion of the vehicle 102. Therefore, the two-dimensional motion state (Ego-motion) of the vehicle 102 may be described as follows:

[0034] where e is the Ego-motion, uj02is the down- range velocity of the vehicle 102, Vy02is the cross-range velocity of the vehicle 102, and w102is the rotation velocity. The radar platform 202(1) may be mounted at a position {%202(I)' 7202(1)' ^202(1)}’is the distance of the radar platform 202(1) from the x-axis, where y 202(1) is the distance of the radar platform 202(1) to the y-axis, and where ^202(1) isthe mounting angle of the radar platform 202(1) relative to the x-axis. Since the radar platform 202 measures the relative motion between itself and one or more detected objects, the Ego-motion coordinates from the radar’s coordinate system to the vehicle’s coordinate system. In one or more embodiments, two or more features (M > 2) may be used to determine the Doppler and Direction of Arrival (DoA) or Angle of Arrival (AoA) measurements,the Doppler velocity dj1determined from the n-th radar platform 202(n) and the j-th detection point pj in the radar point cloud can be expressed as follows:A similar derivation may be added if elevation (z-axis dimension) is added. Since the Doppler velocity represents a radial component of the relative motion between the radar platform 202 and a detected object, the relationship between the radar motion state and the Doppler measurements is a vector including x and y components.

[0035] The radar point cloud 220 does not include the motion state of objects indicated by data points within the radar point cloud 220. The motion state of objects is information that specifies whether the object represented by one of the data points is stationary or moving. If the object is moving, it may be advantageous to determine the velocity vector of the object, the velocity magnitude, or at least whether the object is moving in the same direction or the opposite direction of the sensor circuitry 106 or the vehicle 102 to which the circuitry 104 may be mounted. Such information may be important to the autonomous driving functions or the advanced driver assistance systems.

[0036] Embodiments of the circuitry 104, including a deep neural network (DNN) Ego-motion estimator 222 and a DNN 212 described in FIG. 2 may be configured to determine the Ego-motion of the vehicle 102 and the relative motion of objects detected by the sensor circuits 106 based on the radar point cloud 220 without relying on other external sensors, such as a global positioning satellite (GPS) sensor, an inertial measurement unit, or other external measurements. The DNN Ego-motion estimator 222 and the DNN 212 may determine Ego-motion solely from the radar point clouds 220 from the radar platforms 202. Thus, if other sensors or data links become unreliable, radar-based perception of the Ego-motion may still function correctly.

[0037] Each radar platform 202 may include a DNN Ego-motion estimator 222 configured to process the radar point cloud to extract the relevant features from the radar point cloud (a multidimensional point cloud including J points and M features) to predict pointwise weights and pointwise offsets, and to determine Ego-motion estimates with timestamps 204 based on a weighted least square (WLS) method.

[0038] The Ego-motion estimates with timestamps 204 may be a multi-dimensional radar point cloud with J points and M features. Each DNN Ego-motion estimator 222 may include multiple upscaling and downscaling layers configured to achieve hierarchical feature extraction, including both local and global features. These features may be included in the Ego-motion estimates produced by each radar platform 202 and may be aggregated with Ego-motion estimates from each of the radar platforms 202 to produce a plurality of Ego-motion estimates with timestamps 204. Though timestamps are included with the estimates, it should be appreciated that the timestamps are not necessarily synchronized, since they were applied by each of the radar platforms 202 independently. Accordingly, the timestamps may not be synchronized to a system-wide clock signal, and the resulting timestamped Ego-motion estimates are therefore unsynchronized sensor data.

[0039] In one or more embodiments, the Ego-motion estimates with timestamps 204 from each of the radar platforms 202 may be concatenated into an Ego-motion estimate sequence 206. The Ego-motion estimate sequence 206 may include a random sequence of Ego-motion estimates that continues to accumulate new Ego-motion estimates as they are produced by the radar platforms 202, such that the sequence of Ego-motion estimates is assembled along a time frame from time ti to time tj. In one or more embodiments, a subset of the Ego-motion estimate sequence 206 are selected by an Ego-motion estimation window 208. In the illustrated example, the size of the Egomotion estimation window 208 determines the number of frames selected for the subset. The size of the Ego-motion estimation window 208 may be varied. In an example, size of the Ego-motion estimation window 208 may be varied with the speed of the vehicle 102. In one or more embodiments, as the speed of the vehicle 102 increases, the size of the Ego-motion estimation window 208 may be increased to process more of the Ego-motion estimates in parallel. In one or more embodiments, as the speed of the vehicle 102 decreases, the size of the Ego-motion estimation window 208 may be decreased to process fewer of the Ego-motion estimates in parallel. In one or more embodiments, the size of the Ego-motion estimation window 208 may be predefined and may operate independent of the speed of the vehicle 102. In the illustrative, nonlimiting example, the Ego-motion estimation window 208 selects four Ego-motion estimates at a time to produce a subset for further processing. The Ego-motion estimation window 208 is a moving window that moves from time ti toward time tj producing time- varying subsets of the Egomotion estimate sequence 206.

[0040] The system 200 may include a temporal fusion and smoothing component 210, which may include a deep neural network (DNN) 212, which may be configured to receive the subset from the Ego-motion estimation window 208 and to project the local and global features from the subset to a new feature space corresponding to the geophysical area surrounding the vehicle 102. The DNN 212 may be configured to extract fused temporal features from the subset and to predict parameters, such as acceleration parameters, gain parameters, and weight parameters. The DNN 212 may be configured to apply the acceleration estimates to the subset to produce a compensated subset.

[0041] The temporal fusion and smoothing component 210 may include a second order Kalman filter 224, which may be configured to receive the subset, the compensated subset, and the gain parameters. The second order Kalman filter 224 may be configured to apply the gain parameters to the subset and the compensated subset to produce a filtered subset. The temporal fusion and smoothing component 210 may include a weighted moving average (WMA) filter 226, which may be configured to apply the weight parameters (dynamic weight estimates) to the filtered subset to determine the vehicle Ego-motion data 228. The vehicle Ego-motion data 228 may be the output of the temporal fusion and smoothing component 210 and may be indicative of the motion of the vehicle 102 as well as the relative motion of objects in the view areas of the radar platforms 202.

[0042] In one or more embodiments, the DNN Ego-motion estimators 222 and the DNN 212 of the temporal fusion and smoothing component 210 may be trained together from end-to-end. For example, the system 200 may be exposed to known configurations of objects and conditions to train the DNN 212 and the DNN Ego-motion estimators 222.

[0043] In one or more embodiments of system 200, the hierarchical DNN Ego-motion estimator 222 processes the point cloud data for its radar platform 202. The DNN Ego-motion estimators 222 process the data from each radar platform 202 before fusion of the data. The system 200 may simplify training of multiple radar platforms 202, facilitating model deployment and future updates, because the radar platforms 202 and the temporal fusion and smoothing component 210 can be trained together, instead of separately. In one or more embodiments, the DNN Egomotion estimators 222 and the DNN 212 may be configured to not only extract spatial features of the input radar point clouds but may also recognize local patterns, such as slow-moving objects.In one or more embodiments, the DNN Ego-motion estimators 222 and the DNN 212 may be trained together to handle a large number of trainable parameters. In one or more embodiments, the DNN Ego-motion estimators 222 and the DNN 212 may be trained to manage sparse or empty view areas and to overcome sensor failure. In one or more embodiments, the DNN 212 may be trained to determine the weights for motion and Doppler effects, reducing manual settings. In one or more embodiments, the DNN Ego-motion estimators 222 and the DNN 212 may be trained to compensate automatically for mis-estimation (over or under estimation) of Doppler measurements due to acceleration, deceleration, or both due to steering of the vehicle 102.

[0044] In one or more embodiments, a sensor platform 202 may be implemented as a LIDAR platform in which the front-end 214 may be coupled to one or more optical sensors to receive signals indicative of reflected light from one or more objects in the associated view area. In this example, the MCP 216 and pre-processing 218 may operate on the optical data to produce a point cloud 220 of optical data. The LIDAR sensor platform 202 may include a DNN Ego-motion estimator 222 that may process the point cloud 220 to produce Ego-motion estimates with timestamps 204’, which may be concatenated with the Ego-motion estimates with timestamps 204 from the radar platforms 202 within the Ego-motion estimate sequence 206.

[0045] In one or more embodiments, the training of the DNN Ego-motion estimators 222 may be performed using data from all the sensor platforms 202, with points from the sensor platforms 202 projected to geophysical coordinates of the vehicle 102. In one or more embodiments, the training model may be executed on the DNN Ego-motion estimator 222 of each radar platform 202 or on a zonal / bridge processor, such as the DNN 212 of the temporal fusion and smoothing component 210 or a processor configured to concatenate the Ego-motion estimates with timestamps 204 into the Ego-motion estimate sequence 206.

[0046] In the example of the system 200 of FIG. 2, the temporal fusion and smoothing component 210 is implemented as a programmable circuit, which may include processors, a memory, and other circuitry. It should be understood that the functionality of the temporal fusion and smoothing component 210 may be implemented as processor-readable instructions that may be executed by one or more processors. In one or more embodiments, the circuitry 104 may include processor circuits, a non-volatile memory that can be programmed and optionally reprogrammedto update the processor-readable instructions. An example of a system including processing circuitry and a memory is described below with respect to FIG. 3.

[0047] FIG. 3 depicts a block diagram of a system 300 including control circuitry 302 coupled to sensor circuits 106 and actuator circuits 304, in accordance with certain embodiments of this disclosure. The system 300 may be an embodiment of a portion of the system 100 of FIG. 1 including the circuitry 104 and the sensor circuits 106. In one or more embodiments, the control circuitry 302 may be implemented as part of the circuitry 104. It should be understood that the circuitry 104 may be distributed throughout the vehicle 102 in FIG. 1. In one or more embodiments, the system 300 may be implemented as one or more application specific integrated circuits, one or more computing systems, other data processing systems, or any combination thereof, which may include or may be coupled to one or more sensor circuits 106 and one or more actuator circuits 304. The control circuit 302 in FIG. 3 may be coupled to the sensor circuits 106 and to the actuator circuits 304.

[0048] In one or more embodiments, the sensor circuits 106 may include one or more long- range radar circuits 306, one or more LIDAR circuits 308, one or more optical sensor circuits 310 (such as one or more cameras), one or more short-medium range radar circuits 312, one or more ultrasonic sensor circuits 314, one or more other sensor circuits 316, or any combination thereof. Each of the radar circuits 306 and 312 are configured to generate of radar signals (radio frequency signals) that are transmitted toward a view area (e.g., the long-range radar view area 108 and the short-medium range radar view areas 114), which may include one or more objects (not shown). The radar circuits 306 and 312 may receive reflected radar signals, which may be processed by the radar circuits 306 and 312 to produce radar-point-cloud data for each radar circuit 306 or 312. The radar-point-cloud data may include a plurality of data points representative of reflections received from one of the view areas 108 or 114 and may be indicative of one or more objects in the view area at a single time point. Subsequent instances of the radar-point-cloud information may represent later time points. Each point of the plurality of points may be characterized by a plurality of features including one or more of a doppler velocity measurement indicative of the velocity of the point, an angle-of-arrival measurement indicative of the angle to the point from a reference direction of the radar circuit 306 or 312 (i.e., the direction faced by the transmitter / receiver of the radar circuit 306 or 312). In one or more embodiments, each radar circuit 306 and 312 may include a deep neural network (DNN) component configured to determine a plurality of features thatcharacterize each point. The features may include one or more of spatial dimension coordinates (e.g. cartesian coordinates), the angle-of-arrival measurement including azimuth and elevation (available, if the radar circuit 306 or 312 has a planar antenna array), a range or distance between the radar circuit 306 or 312 and the object to which the point corresponds, a reflected signal power measurement, or other data. In one or more embodiments, the DNN component may also determine information that may characterize each point or groups of points based on learned feature extractors to determine Ego-motion estimates for each radar. Each of the radar circuits 306 and 312 may be configured to produce an output including Ego-motion estimates with timestamps and an identifier indicative of the radar circuit 306 or 312.

[0049] Similarly, the one or more LIDAR circuits 308 and the ultrasonic sensor circuits 314 may determine point-cloud data indicative of signal reflections from objects in associated view areas (e.g., the LIDAR view area 110 or the ultrasonic sensors view areas 116). In one or more embodiments, the LIDAR circuit 308 and the ultrasonic sensor circuits 314 may include a DNN component configured to characterize each point or groups of points based on learned feature extractors to determine Ego-motion estimates for each LIDAR circuit 308 or each ultrasonic sensor circuit 314. Each LIDAR circuit 308 may be configured to produce an output including Egomotion estimates with timestamps and an identifier indicative of the LIDAR circuit 308.

[0050] The optical sensor circuits 310 may capture image data, which may include optical data indicative of objects in the associated view areas (e.g., the optical sensors view areas 112). In one or more embodiments, the optical sensor circuits 310 may generate optical point-cloud data indicative of objects in the view areas. In one or more embodiments, each optic sensor circuit 310 may include a DNN component configured to characterize the optical point-cloud data based on learned feature extractors to determine Ego-motion estimates for each optical sensor circuit 310. Each optical sensor circuit 310 may be configured to produce an output including Ego-motion estimates with timestamps and an identifier indicative of the optical sensor circuit 310.

[0051] The other sensor circuits 316 may include other types of object detection circuits. Each of the object detection circuits may also include a DNN component configured to determine Egomotion estimates and each may be configured to produce an output including Ego-motion estimates with timestamps and an identifier indicative of associated object detection circuit. The other sensor circuits 316 may also include other types of sensors, such as temperature sensors,moisture sensors, a global positioning satellite (GPS) circuit, an inertial measurement unit (IMU), other sensors, or any combination thereof.

[0052] In one or more embodiments, the sensor circuits 106 may provide sensor data, including Ego-motion estimates with timestamps and an identifier, to the control circuitry 302. The control circuitry 302 may process the Ego-motion estimates to determine Ego-motion data, which may be used to generate control signals, which may be provided to one or more of the actuator circuits 304 to control various functions.

[0053] The actuator circuits 304 may include acceleration actuators 318 (e.g., electric motor driver circuits, fuel injection systems, or other types of actuators), steering actuators 320, braking actuators 322, other actuators 324, or any combination thereof. In one or more embodiments, the control circuitry 302 may be configured to receive sensor data from one or more of the sensor circuits 106, determine one or more parameters based on the sensor data, and send a control signal to one or more of the actuator circuits 304 based on the determined one or more parameters.

[0054] In one or more embodiments, the control circuit 302 may include one or more input / output (I / O) interfaces 326, which may be configured to couple to the sensor circuits 106, the actuator circuits 304, other circuits, or any combination thereof. In one or more embodiments, the I / O interfaces 326 may include wired or wireless transceivers configured to communicate with the sensor circuits 106, the actuator circuits 304, or other circuits to send signals, to receive Egomotion estimate data, to receive sensor data or other data, or any combination thereof.

[0055] In one or more embodiments, the control circuitry 302 may include one or more processor circuits 328, which may be coupled to the one or more I / O interfaces 326 to provide data or to receive data. In one or more embodiments, the one or more processor circuits 328 may include logic circuits, microprocessors, processors, or other circuits configured to execute instructions that cause the processor circuits 328 to perform one or more operations. The control circuitry 302 may include one or more transceivers 329, which may be configured to send and receive data through a wired connection (such as an Ethernet connection) or a wireless connection (such as a radio frequency communications link). The control circuitry 302 may include or may be coupled to a memory 330, which may include a non-volatile memory device configured to store data and processor-executable instructions.

[0056] In one or more embodiments, the memory 330 may include operating system instructions (not shown) that may be executed by the one or more processor circuits 328 to enable communication with the hardware associated with the system 300 and to enable various services, such as Ethernet communication and other communication services, hardware controls, and other common services that may be used by instructions executed by the processor circuits 328.

[0057] In one or more embodiments, the memory 330 may include one or more sensor control instructions 332 that may cause the one or more processor circuits 328 to receive data from the one or more sensor circuits 106 and to store the sensor data 334 in the memory 330. In one or more embodiments, the sensor data 334 may include Ego-motion estimates with timestamps and associated identifiers that indicate the sensor circuit 106 that generated the Ego-motion estimate. In one or more embodiments, the sensor control instructions 332 may cause the one or more processor circuits 328 to poll selected sensor circuits 106 to retrieve sensor data. In one or more embodiments, the sensor control instructions 332 may cause the one or more processor circuits 328 to communicate with the sensor circuits 106 to initiate a data sensing operation.

[0058] In one or more embodiments, the memory 330 may include actuator control instructions 336 that may cause the one or more processor circuits 328 to generate one or more control signals and to send the control signals to the one or more actuator circuits 304. In one or more embodiments, one or more of the processor circuits 328 may use the actuator control instructions 336 to generate a control signal to provide the control signal to one of the actuator circuits 304 to trigger an action, such as emergency braking when the control signal is provided to the one or more braking actuators 322. In one or more embodiments, the actuator control instructions 336 may cause the one or more processor circuits 328 to store actuator data 338 in the memory 330. The actuator data 338 may include an actuator identifier (ID), data indicative of the state of the actuator, other data, or any combination thereof.

[0059] In one or more embodiments, the memory 330 may include system controller instructions 340 that may cause the one or more processor circuits 328 to control operation of various components of the system 300. In one or more embodiments, the system controller instructions 340 may be configured to receive Ego-motion data determined using temporal fusion and smoothing instructions 342 and to provide control signals to various actuator circuits 304, indicators (such as warning lights or other indicators, not shown), or other components.

[0060] In one or more embodiments, the memory 330 may include the temporal fusion and smoothing instructions 342 that, when executed, may cause the one or more processor circuits 328 to receive data from the sensor circuits 106 and to determine Ego-motion data corresponding to the vehicle 102 based on the received data. The memory 330 may include Ego-motion estimate handling instructions 344 that may cause the one or more processor circuits 328 to concatenate the Ego-motion estimates from the one or more long-range radar circuits 306 and the short-medium range radar circuits 312 and optionally from one or more of the one or more LIDAR circuits 308, the one or more ultrasonic sensor circuits 314, or the other sensor circuits 316. In one or more embodiments, each of the one or more long-range radar circuits 306 and the short-medium range radar circuits 312 (and optionally the LIDAR circuits 308) may include a deep neural network (DNN) estimator configured to receive a point cloud 220 from a single sensor (radar or LIDAR). The DNN may include multiple up-scaling and down-scaling layers configured to achieve hierarchical feature extraction, enabling both local and global feature extraction. The DNN may extract spatial features of the input radar point clouds and may recognize local patterns (e.g., slow moving objects)

[0061] The temporal fusion and smoothing instructions 342 may include estimate window instructions 346 that, when executed, cause the one or more processor circuits 328 to iteratively select subsets of the concatenated Ego-motion estimates for processing. The estimate window instructions 346 may determine a selected number of frames A or a selected time period T defining a size of the window, which determines the selected number of Ego-motion estimates defining the subset.

[0062] The temporal fusion and smoothing instructions 342 may include a deep neural network (DNN) 348 configured to process selected subsets of a sequence of Ego-motion estimates with timestamps from each of the radar circuits 306 and 312, and optionally from one or more of the LIDAR circuits 308. The DNN 348 may include feature projection instructions 350 that, when executed, may cause the one or more processor circuits 328 to extract a plurality of features from the selected subset and to project the features to a new feature space corresponding to the vehicle 102. In one or more embodiments, the new feature space may represent objects detected within a physical area surrounding the system 300 and mapped to a geophysical space such that the data may be used for real-time functionality related to Ego-motion of the system 300.

[0063] The DNN 348 may include sequence model instructions 352 that, when executed, may cause the one or more processor circuits 328 to extract fused temporal features form the selected subset of the Ego-motion estimate sequence and to predict acceleration data (acceleration estimates 354), Kalman filter gain estimates 356, and dynamic weight estimates 358 based on the selected subset. The sequence model instructions 352 may include a bi-directional gate recurrent unit (GRU) instructions, a recurrent neural network (RNN) instructions, a long short-term memory (LSTM) instructions, transformer instructions (for example, a self-contained deep learning model), other sequence model instructions, or any combination thereof.

[0064] In one or more embodiments, the temporal fusion and smoothing instructions 342 may include Ego-motion compensation unit instructions 360 that, when executed, may cause the one or more processor circuits 328 to compensate data from the selected subset using the acceleration estimates 354 to produce compensated Ego-motion estimates. In one or more embodiments, the temporal fusion and smoothing instructions 342 may include Kalman filter update unit instructions 362 configured to use the Kalman gain estimates 356 to adjust one or more settings and to apply the updated settings to filter the compensated Ego-motion estimates to produce filtered and compensated Ego-motion estimates.

[0065] In one or more embodiments, the temporal fusion and smoothing instructions 342 may include weighted moving average estimation instructions 364 that may cause the one or more processor circuits 328 to apply the dynamic weight estimates 358 to the filtered and compensated Ego-motion estimates to apply weighted moving averages to fuse, smooth and filter the Egomotion estimations from different radars and times to determine the Ego-motion data for the system 300.

[0066] In one or more embodiments, the temporal fusion and smoothing instructions 342 may include other instructions 366, which may perform additional functions. In one or more embodiments, the instructions stored in the memory 330 may be updated from an external system, such as by a software update.

[0067] FIG. 4 depicts a block diagram of an embodiment of a system 400 including one of the DNN Ego-motion estimators 222 of FIG. 2, in accordance with certain embodiments of this disclosure. The DNN Ego-motion estimator 222 may include one or more inputs to receive a radar point cloud 401, which may include a matrix of object- related data points including Doppler datafor each point and Angle of Arrival (Ao A) data for each point. The point cloud 401 (from a radar or a LIDAR sensor) may include a number of features M (local and global) and a number of data points J. The DNN Ego-motion estimator 222 may include a deep neural network (DNN) 402, which may be configured to determine point- wise weights 428 and point- wise offsets 430 based on the radar point cloud 401.

[0068] The DNN 402 may include includes an encoding component 404 configured to receive the point cloud matrix 401, encode the point cloud 401, and provide the encoded point cloud 401 to a T-net 406. The T-net 406 may process the encoded radar point cloud 401 to determine learned features, and to provide the features to a multi-layer perceptron (MLP) 408.

[0069] In one or more embodiments, the MLP 408 may be configured to encode a selected subset of the features M of the point cloud 401 and to project the selected subset onto a different feature space, such as the vehicle feature space. The MLP 408 may operate pointwise on the radar point cloud 401 from features A / =l to 64 and may provide the projected subset to a scaling layer 418. The MLP 408 may also provide the data points J=1 to 64 to the scaling layer 410.

[0070] The scaling layer 410 may scale the features and data points and provide them to a channel mixing layer 412, which may mix the features A / =l to 64 and the data points J= to 64 with the features and data points from 64 to 128. The mixed output may be provided to a scaling layer 418, which may be configured to scale the mixed output and the feature and data points from 16 to 64. The mixed output of the channel mixing layer 412 may also be provided to a scaling layer 414, which may scale the features and data points from 64 to 16 and the mixed output. The mixed output of the scaling layer 414 may be provided to a channel mixing layer 416, which may mix the features and data points from the mixed output with the features and data points from 128 to 256.

[0071] The output of the channel mixing layer 416 may be provided to the scaling layer 418, which may scale the features and data points from 16 to 64 received from the channel mixing layer 416 and the channel mixing layer 412. The output may be provided to a channel mixing layer 420, which may channel mix the features and data points from 256 to 128.

[0072] The mixed output may be provided to a feature drop-out layer 422, which may drop out one or more features from the mixed channel data based on various parameters. The resultingoutput may be provided to a scaling layer 424, which may scale the remaining features and data points from the feature drop-out layer 422 and the features and data points from the MLP 408.

[0073] The output from the scaling layer 424 may be provided to a channel mixing layer 426 configured to mix the features and data points from 128 to 64. The channel mixing layer 426 may produce point-wise weights 428 and point-wise offsets 430, which may be the outputs of the DNN 402.

[0074] The DNN Ego-motion estimator 222 may include a weight selector 432, which may be configured to receive the point-wise weights 428. To reduce the computational cost and impact of outliers, the weight selector 432 may be configured to select the k data points with the highest weights along with their original features, such as the radar cross-section (RCS), Doppler, and other features. The selected k data points and their original features may be provided to a weighted least square regression component 434, which may process the k data points and their associated features to determine the Ego motion estimates with timestamps 204 (in FIG. 2).

[0075] FIG. 5 depicts a block diagram of an embodiment of a system 500 including the temporal fusion and smoothing component 210 of FIG. 2, in accordance with certain embodiments of this disclosure. As previously discussed, an Ego-motion estimate sequence 206 may be produced by concatenating the Ego-motion estimates with timestamps 204 in FIG. 2. The Egomotion estimation window 208 represents a moving window that may be used to select subsets of the Ego-motion estimate sequence 206 for processing.

[0076] The selected subset 501 may be provided to the DNN 212 of the temporal fusion and smoothing component 210. The DNN 212 may include a feature projection sub-network 502, which may be configured to determine features from the selected subset 501 and to project the determined features to a new feature space, which may correspond to the structure to which the radar platforms 202 are mounted, such as the vehicle 102.

[0077] In one or more embodiments, the projected features may be provided to a sequence model 504. The sequence model 504 may be implemented in various architectures, such as a bidirectional gate recurrent unit (GRU), a recurrent neural network (RNN), a long short-term memory (LSTM), transformers (a self-contained deep learning model), other sequence models, or any combination thereof.

[0078] The sequence model 504 may be configured to extract fused temporal features from the selected subset 501 and the projected features. The sequence model 504 may be configured to determine a plurality of estimates based on the extracted fused temporal features. The estimates may include acceleration estimates 506, Kalman gain estimates 508, and dynamic weight estimates 510.

[0079] The temporal fusion and smoothing component 210 may include an Ego-motion compensation unit 512 configured to receive the acceleration estimates 506 and the selected subset 501 of the Ego-motion estimate sequence 206. The Ego-motion compensation unit 512 may apply the acceleration estimates 506 to produce compensated Ego-motion estimates. The acceleration estimates 506 may be used to compensate the Doppler measurements caused by acceleration of the vehicle 102.

[0080] The temporal fusion and smoothing component 210 may include a Kalman filter 224, which may be a second order Kalman filter. The Kalman filter 224 may be configured to receive the Kalman gain estimates 508, the compensated Ego-motion estimates from the Ego-motion compensation unit 512, and the selected subset 501 of the Ego-motion estimate sequence 206. The Kalman gain estimates 508 may provide a weight to measurements and current-state estimates of the Kalman filter 224, which can be tuned to provide a selected performance. The Kalman filter 224 may use the Kalman gain estimates 508 to generate filtered Ego-motion estimates based on the selected subset 501, which may include Ego-motion estimates 304 from different radars and different times.

[0081] The temporal fusion and smoothing component 210 may include a weighted moving average (WMA) unit 226, which may be configured to receive the filtered Ego-motion estimates from the Kalman filter 224 and which may receive the dynamic weight estimates 510 from the DNN 212. The WMA unit 226 may be configured to use the dynamic weight estimates 510 from the DNN 212 to fuse and smooth the filtered Ego-motion estimates to produce vehicle Ego-motion data 228. In one or more embodiments, the vehicle Ego-motion data 228 may include translation velocity (transitional motion) and yaw rate (rotational motion) of the vehicle 102. In the context of the systems of FIGs. 1-4, the vehicle Ego-motion data 228 may be used to enable functionality, such as emergency braking, collision avoidance, and other functionality. In one or more embodiments, one or more of the acceleration estimates 506, the Kalman gain estimates 508, andthe dynamic weight estimates 510 may be provided as an output that may be used in some downstream tasks, such as mapping or other tasks.

[0082] In one or more embodiments, the input to the temporal fusion and smoothing component 210 includes Ego-motion estimates with timestamps 204 from multiple radar platforms 202, each with its own identifier, which Ego-motion estimates with timestamps 204 are concatenated into the Ego-motion estimate sequence 506 to form a feature vector. The features may be aggregated in a queue with a configurable length of N Ego-motion estimates or of a time T. The Ego-motion estimate sequence 506 may be fed to the DNN 212 and the Kalman filter 224 and the WMA unit 226 of the temporal fusion and smoothing component 210 for acceleration compensation, fusion, and smoothing.

[0083] The DNN 212 may fuse the selected subset 501, which may include results from different radar platforms 202 at different times, and then predict estimates (acceleration estimates 506, Kalman gain estimates 508, and weighted moving average estimates 510) for the Kalman filter 224 and WMA unit 226. The DNN 212 may automatically adjust the parameters of the Kalman filter 224 and the WMA unit 226 during operation, enhancing both the accuracy and robustness of the system relative to a fixed parameter implementation or relative to a manually adjustable system.

[0084] While the discussion above has focused on Ego-motion of the vehicle 102, it should be appreciated that the temporal fusion and smoothing component 210 may process Ego-motion estimates from the multiple sensor platforms 202 for other purposes. In an example, the Egomotion estimates may be processed to determine odometry functions. In one or more embodiments, the temporal fusion and smoothing component 210 may be configured to overcome failure of one or more sensor platforms 202. In one or more embodiments, the accuracy of the Ego-motion data 228 may be used to generate maps with smoother curbs and edges and with moving objects removed from the map.

[0085] In one or more embodiments, inclusion of the DNN Ego-motion estimators 222 in each of the radar platforms 202 enables determine of local patterns (e.g., slow moving objects) at the sensor platform 202. The temporal fusion and smoothing component 210 may enable a multiple radar implementation where the DNN 212 and the DNN Ego-motion estimators 222 may be trainedtogether on data from each of the radar platforms 202, facilitating model deployment and future updates.

[0086] In one or more embodiments, the temporal fusion and smoothing component 210 may be configured to take advantage of the high sampling rate of the sensor platforms 202 and the temporal correlation of the motion of the vehicle 102 using a Kalman filter 224 that can be dynamically updated by the DNN 212 to adjust for process noise and observation noise to enhance the accuracy of the Ego-motion data 228.

[0087] In one or more embodiments, the DNN Ego-motion estimator 222 and the DNN 212 of the temporal fusion and smoothing component 210 may be trained together to be insensitive to sparse / empty scenes or sensor failure, and the weights for the motion loss and Doppler loss may be dynamically compensated, providing for enhanced Ego-motion determination capability. In one or more embodiments, the DNN 212 may be configured to compensate for over-estimation or under-estimation of Doppler measurements caused by acceleration or deceleration of the vehicle 102.

[0088] FIG. 6 depicts a graph 600 of the translational velocity (in meters / second) versus time (in seconds) determined by the systems of FIGs. 1-5, in accordance with certain embodiments of this disclosure. In this example, single frame estimation may cause the system to deviate from the actual data by a small margin during acceleration. The temporal fusion and smoothing component 210 may enable accurate Doppler compensation during acceleration, which may be difficult with fixed or manually compensation schemes. When observations from multiple sensor platforms 202 with different fields-of-view (e.g., radar platforms 202 associated with long-range radar view area 108 and short-medium range radar view areas 114 in FIG. 1, LIDAR platforms associated with the LIDAR view area 110 in FIG. 1 , other sensor platforms, or any combination thereof) are fed into the temporal and smoothing component 210, the temporal fusion and smoothing component may be configured to accurately identify accelerations, either longitudinal or lateral, and to determine acceleration estimates as described with respect to the acceleration estimates 354.

[0089] In one or more embodiments, the DNN 212 may be configured to fuse the selected subset 501 (in FIG. 5), which may include results from different radar platforms 202 at different times. The DNN 212 may predict parameters (such as acceleration estimates 508, Kalman gain estimates 510, and dynamic weight estimates 512) to compensate for acceleration, to filter and tosmooth the selected subset 501 to produce the vehicle Ego-motion data 228, including translational velocity and yaw rate of the vehicle 102.

[0090] FIG. 7 depicts a graph 700 of the radial velocity (in meters / second) versus angle of arrival (in radians) determined by the systems of FIGs. 1-5, in accordance with certain embodiments of this disclosure. In this example, Doppler measurements are largely grouped along a curved path, with a plurality of outliers that are spaced apart from the majority. The temporal fusion and smoothing component 210 may be configured to compensate, filter, and smooth the Ego-motion estimates to produce vehicle Ego-motion data 228 that is accurate even under acceleration or deceleration of the vehicle 102. The temporal fusion and smoothing component 210 may compensate for over-estimation and under-estimation of Doppler measurements due to acceleration and deceleration of the vehicle 102. The temporal fusion and smoothing component 210 may also readily discard outliers, enabling accurate object detection.

[0091] FIG. 8 depicts a flow diagram of a method 800 of determining Ego-motion estimates with timestamps 204, in accordance with certain embodiments of this disclosure. At 800, the method 800 may include receiving one or more signals at a front-end 214 of a sensor platform 202. In one or more embodiments, the system may include a plurality of sensor platforms 202, which may include radar sensor platforms, LIDAR sensor platforms, other sensor platforms, or any combination thereof. The front-end 214 may include one or more mixers, one or more high-pass filters, one or more programmable amplifiers, one or more low-pass filters, and an analog-to- digital converter (ADC) to process received analog signals and to produce a digital output indicative of the received signals.

[0092] At 804, the method 800 may include determining data from the one or more signals. The data may be determined by an MCP 216 configured to process the received digital signals from the front-end 214 to determine fast-time range spectrum data and slow-time Doppler spectrum data and may be configured to determine direction of arrival information for one or more objects the view area based on the digital signal data.

[0093] At 806, the method 800 may include processing the data to determine distance and acceleration data. The MCP 216 may be configured to determine range data based on timing between transmitted signals and reflected signals and to determine acceleration data based on changing timing data.

[0094] At 808, the method 800 may include generating a point cloud based on the determined data, the distance data, and the acceleration data. Each sensor platform 202 may include a preprocessing unit 218 configured to add a timestamp and radar identifier to the data determined by the radar MCU 216 to produce a point cloud 220, including a plurality of data indicative of objects in a field of view of the sensor platform 202. The point cloud P is a multi-dimensional point cloud including J detected points and AT is the number of features (local and global) of each point of the radar point cloud, including a plurality of data indicative of objects in a field of view of the sensor platform 202. The pre-processing unit 218 may be configured to add a timestamp and sensor identifier to the data.

[0095] At 810, the method 800 may include generating an output including Ego-motion estimates with timestamps. The Ego-motion estimates with timestamps 204 from multiple sensor platforms 202 (radar platform, LIDAR platform, other sensor platforms, or any combination thereof) may be concatenated to form an Ego-motion estimate sequence 206, which may be further processed by the temporal fusion and smoothing component 210 to determine the Ego-motion data 118 for the vehicle 102.

[0096] At 812, the method 800 may include providing the output to a temporal fusion and smoothing component 210 to process the output from the sensor platforms to determine EGO- motion data 228 including transitional velocity data and rotational motion data. As discussed above, the temporal fusion and smoothing component 210 may be configured to process selected subsets 501 of the Ego-motion estimate sequence 206 based on a moving Ego-motion estimation window 208 to produce the Ego-motion data 228.

[0097] The DNN Ego-motion estimator 222 may produce the Ego-motion estimates with timestamps 204, which may include feature data extracted from the radar reflection signals. The feature data may include local features and global features, which may be included in the point cloud 220 and which may be used to provide the Ego-motion estimates with timestamps 204. The local patterns may include slow moving objects, which may be extracted from spatial features of the point cloud 220.

[0098] FIG. 9 depicts a flow diagram of a method 900 of determining Ego-motion of a vehicle 102 based on a plurality of Ego-motion estimates with timestamps 204, in accordance with certain embodiments of this disclosure. In an example, the method 900 may be an embodiment of a processexecuted by the temporal fusion and smoothing component 210 to produce the Ego-motion data 228 from the concatenated Ego-motion estimate sequence 206 produced from the plurality of Egomotion estimates produced by multiple radar platforms 202.

[0099] At 902, the method 900 may include selecting a subset (selected subset 501) of outputs of a plurality of EGO-motion estimates, each EGO-motion estimate having a timestamp (Egomotion estimate with timestamp 204) and including point-wise weights and point-wise offsets for a sensor platform 202 of a plurality of sensor platforms 202. In one or more embodiments, the sensor platforms 202 may include radar platforms, LIDAR platforms, other sensor platforms, or any combination thereof.

[0100] At 904, the method 900 may include projecting features based of the subset 501 into a new feature space. The features from the Ego-motion estimates with timestamps 204 may be projected to a new feature space, which may correspond to the structure to which the radar platforms 202 are mounted, such as the vehicle 102.

[0101] At 906, the method 900 may include extracting fused temporal features from the subset 501 and predict estimates (e.g., acceleration estimates 506, Kalman gain estimates 508, and dynamic weights estimates 510). In one or more embodiments, the DNN 212 may include a sequence model 504 that may be configured to extract fused temporal features from the selected subset 501 and the projected features. The sequence model 504 may be configured to determine a plurality of estimates based on the extracted fused temporal features. The estimates may include acceleration estimates 506, Kalman gain estimates 508, and dynamic weight estimates 510.

[0102] At 908, the method 900 may include processing the subset 501 based on the acceleration estimates 506 to produce compensated Ego-motion estimates. In one or more embodiments, the temporal fusion and smoothing component 210 may include an Ego-motion compensation unit 512 configured to receive the acceleration estimates 506 and the selected subset 501 of the Ego-motion estimate sequence 206. The Ego-motion compensation unit 512 may apply the acceleration estimates 506 to produce compensated Ego-motion estimates. The acceleration estimates 506 may be used to compensate the Doppler measurements caused by acceleration of the vehicle 102.

[0103] At 910, the method 900 may include filtering the compensated Ego-motion estimates (subset 501) using a Kalman filter 224 configured according to the Kalman gain estimates 508 toproduce filtered Ego-motion estimates. In one or more embodiments, the temporal fusion and smoothing component 210 may include a Kalman filter 224, which may be a second order Kalman filter. The Kalman filter 224 may be configured to receive the Kalman gain estimates 508, the compensated Ego-motion estimates from the Ego-motion compensation unit 512, and the selected subset 501 of the Ego-motion estimate sequence 206. The Kalman gain estimates 508 may provide a weight to measurements and current-state estimates of the Kalman filter 224, which can be tuned to provide a selected performance. The Kalman filter 224 may use the Kalman gain estimates 508 to generate filtered Ego-motion estimates based on the selected subset 501, which may include Ego-motion estimates 304 from different radars and different times.

[0104] At 912, the method 900 may include applying a weighted moving average to the filtered Ego-motion estimates based on the dynamic weight estimates 510 to produce an output including Ego-motion data 228. The temporal fusion and smoothing component 210 may include a WMA unit 226, which may be configured to receive the filtered Ego-motion estimates from the Kalman filter 224, and which may receive the dynamic weight estimates 510 from the DNN 212. The WMA unit 226 may be configured to use the dynamic weight estimates 510 from the DNN 212 to fuse and smooth the filtered Ego-motion estimates to produce vehicle Ego-motion data 228. In one or more embodiments, the vehicle Ego-motion data 228 may include translation velocity (transitional motion) and yaw rate (rotational motion) of the vehicle 102.

[0105] In one or more embodiments, a system may include multiple radar platforms 202, each of which may include a deep neural network (DNN) Ego-motion estimator 222 configured to determine Ego-motion estimates with timestamps 204 from a plurality of reflections received by the radar platform 202. Each radar platform 202 may produce a plurality of Ego-motion estimates with timestamps 204. The system may concatenate the Ego-motion estimates with timestamps 204 into an Ego-motion estimate sequence 206 including Ego-motion estimates from different radar platforms 202 and different times. The system may include a temporal fusion and smoothing component 210, which may include a DNN 212, and which may be configured to determine Egomotion data 228 for the system based on selected subsets of the Ego-motion estimate sequence 206. The DNN 212 and the DNN Ego-motion estimators 222 of each of the radar platforms 202 may be trained together on radar data from each of the radar platforms 202 to provide a robust and dynamically adjustable Ego-motion determination system.

[0106] In one or more embodiments, using data from a radar sensor network including multiple radar platforms can improve overall accuracy and robustness, achieving performance that can match dedicated sensors. The radar platforms 202 are not limited to specific configurations, and the system may be configured to determine Ego-motion data 228 from unsynchronized radar data from multiple radar platforms 202 having different radar configurations, locations, fields of view, resolutions, and so on.

[0107] In one or more embodiments, in addition to radar platforms 202, the system may include one or more LIDAR platforms 202. Each LIDAR platform 202 may include a DNN Ego-motion estimator 222 configured to determine Ego-motion estimates with timestamps 204 from a plurality of reflections received by the LIDAR platform 202. Each LIDAR platform 202 may produce a plurality of Ego-motion estimates with timestamps 204’. The system may concatenate the Egomotion estimates with timestamps 204’ into an Ego-motion estimate sequence 206 including Egomotion estimates from different sensor platforms 202 and different times. The system may include a temporal fusion and smoothing component 210, which may include a DNN 212, and which may be configured to determine Ego-motion data 228 for the system based on selected subsets of the Ego-motion estimate sequence 206. The DNN 212 and the DNN Ego-motion estimators 222 of each of the sensor platforms 202 may be trained together on radar sensor from each of the sensor platforms 202 to provide a robust and dynamically adjustable Ego-motion determination system.

[0108] In one or more embodiments, the temporal fusion and smoothing component 210 may enable Ego-motion estimation from sensor platforms 202 that are not synchronized and makes synchronization unnecessary. In one or more embodiments, the system may be configured to determine Ego-motion parameters (e.g., acceleration estimates 506, Kalman gain estimates 508, and dynamic weight estimates 510) with temporal information determined from the Ego-motion estimate sequence 206. The system may be configured to function correctly, even under adversarial conditions, to determine Ego-motion data 228, even with sensor data loss, sensor failure, and sensor network out-of-synchronization.

[0109] In one or more embodiments, the system may process Ego-motion estimates with timestamps 204 without frame aggregation or data association. The temporal fusion and smoothing component 210 may temporarily fuse, filter, and smooth the estimated Ego-motion parameters. In one or more embodiments, the DNN Ego-motion estimator 222 may select k data points that have point-wise weights 428 that are higher than a threshold or that are higher than others, discardingdata points that have lower point-wise weights 428, thereby reducing computational costs and impacts of outliers. In this case, the resource requirement and latency may be comparable to single-frame methods.

[0110] In one or more embodiments, one or more of the DNN Ego- estimators 222 or the DNN 212 may be configured to correct for over-estimation and for under-estimation of Doppler measurements of a vehicle 102 due to acceleration, deceleration, or both.

[0111] One or more embodiments of this disclosure may be further understood in light of the illustrative, non-limiting examples presented below.

[0112] Example 1 : A system may include a control circuit including: one or more input / output (I / O) interfaces configured to couple to a plurality of sensor circuits coupled to a vehicle to receive unsynchronized Ego-motion data including data indicative of objects in a view area of one or more of the plurality of sensor circuits; one or more processor circuits coupled to the one or more I / O interfaces, the one or more processor circuits configured to: receive the unsynchronized Egomotion data from one or more of the plurality of sensor circuits; select a subset of the unsynchronized Ego-motion data using a moving window; project features from the subset to a vehicle feature space using a deep neural network (DNN); extract fused temporal features from the subset using the deep neural network; predict, using the DNN, acceleration estimates, Kalman filter estimates, and dynamic weight estimates based on the fused temporal features; compensate the unsynchronized Ego-motion data using the acceleration estimates to produce compensated Ego-motion data; filter the unsynchronized Ego-motion data and the compensated Ego-motion data using a Kalman filter updated with the Kalman gain estimates to produce filtered Ego-motion data; and apply the dynamic weight estimates using a weighted moving average to fuse and smooth the filtered Ego-motion data to determine vehicle Ego-motion data corresponding to transitional motion and rotational motion of the vehicle and relative motion of the objects.

[0113] Example 2: The system of Example 1 , wherein the plurality of sensor circuits comprises one or more of a radar platform or a light-detection and ranging platform.

[0114] Example 3: The system of any of the preceding Examples, where the plurality of sensors includes a plurality of radar platforms, each radar platform including: a radar front end circuit coupled to one or more antenna to receive reflected signals corresponding to one or more objects in a view area of the radar platform; circuitry configured to generate a radar point cloudincluding a plurality of features for each of a plurality of data points corresponding to data determined from the reflected signals, the plurality of features including local features and global features and including Doppler data and Angle of Arrival (AoA) data for each datum; and a second deep neural network configured to determine Ego-motion estimates with timestamps based on the radar point cloud and to provide the determined Ego-motion estimates with timestamps to the control circuit as part of the unsynchronized Ego-motion data.

[0115] Example 4: The system of any of the preceding Examples, where the unsynchronized Ego-motion data includes determined Ego-motion estimates from at least some of the plurality of radar platforms.

[0116] Example 5: The system of the preceding Examples, where, when one or more of the plurality of radar platforms fail, the DNN of the control circuit is configured to determine the vehicle Ego-motion data based on the unsynchronized Ego-motion data from others of the plurality of radar platforms.

[0117] Example 6: The system of any of the Examples 3 through 5, where the second deep neural network processes the radar point cloud prior to fusion.

[0118] Example 7: The system of any of the Examples 3 through 6, where the second deep neural network includes multiple upscaling and downscaling layers configured to determine hierarchical feature extraction to extract the local features and the global features, the Ego-motion estimates with timestamps include the local features and the global features.

[0119] Example 8: The system of any of the Examples 3 through 7, where the second deep neural network of each of the plurality of radar platforms is configured to: determine point- wise weights for the radar point cloud; determine point-wise offsets for the radar point cloud; and determine a weight subset of the point-wise weights that are greater than a threshold weight; and apply a weighted least square regression to selected data points corresponding to the weight subset including the local features, the global features, the point-wise weights, and the point-wise offsets to determine the Ego-motion estimates with timestamps.

[0120] Example 9: The system of any of the Examples 3 through 8, where the DNN and the second deep neural networks of the plurality of radar platforms are trained together using datafrom all of the plurality of radar platforms with the radar point clouds of each of the plurality of radar platforms projected to the vehicle feature space.

[0121] Example 10: The system of any of the preceding Examples, where the Kalman filter comprises a second order Kalman filter.

[0122] Example 11: The system of any of the preceding Examples, where the DNN is configured to concatenate the unsynchronized Ego-motion data from one or more of the plurality of sensor circuits to form a feature vector.

[0123] Example 12: The system of Example 11, where the DNN is configured to select the subset from the feature vector, where the subset has a configural length or time interval.

[0124] Example 13: The system of any of the preceding Examples, where the DNN fuses the subset of the unsynchronized Ego-motion estimates from two or more of the plurality of sensor circuits from different times.

[0125] Example 14: The system of any of the preceding Examples, where the Kalman filter estimates tune the Kalman filter to apply a gain to selected data points of the subset.

[0126] Example 15: The system of any of the preceding Examples, where the dynamic weight estimates filter and smooth the subset to produce the vehicle Ego-motion data.

[0127] Example 16: A method of determining vehicle Ego-motion data, the method including: determining unsynchronized Ego-motion estimates with timestamps at each of a plurality of sensor platforms, the unsynchronized Ego-motion estimates with timestamps including local features and global features; concatenating, at a control circuit, the unsynchronized Ego-motion estimates with timestamps and identifiers from at least some of the plurality of sensor platforms to form a feature vector including unsynchronized Ego-motion estimates from the plurality of sensor platforms at different times; selecting, at the control circuit, a subset of the unsynchronized Ego-motion estimates with timestamps using a moving data window having a selected length or time interval; fusing the subset using a deep neural network (DNN) of the control circuit; predicting, using the DNN, acceleration estimates, Kalman filter gain estimates, and dynamic weight estimates for the subset; applying, at the control circuit, the acceleration estimates to the unsynchronized Egomotion estimates to compensate for under-estimation or over-estimation of Doppler measurement data to produce compensated Ego-motion data; applying, at the control circuit, the Kalman filtergain estimates to the unsynchronized Ego-motion estimates and the compensated Ego-motion data at a second order Kalman filter to produce filtered Ego-motion data; and applying, at the control circuit, the dynamic weight estimates as a weighted moving average to the filtered Ego-motion data to determine vehicle Ego-motion data.

[0128] Example 17: The method of Example 16, where determining the unsynchronized Egomotion estimates with timestamps includes: receiving reflected signals at a radar platform of the plurality of sensor platforms, the radar platform including a radar front end circuit coupled to one or more antennas, the reflected signals corresponding to one or more objects in a view area of the radar platform; generating a radar point cloud including a plurality of features for each of a plurality of data points corresponding to data determined from the reflected signals at processing circuitry of the radar platform, the plurality of features including local features and global features and including Doppler data and Angle of Arrival (AoA) data for each datum; determining Egomotion estimates with timestamps based on the radar point cloud using a second deep neural network; and providing the Ego-motion estimates with timestamps to the control circuit as part of the unsynchronized Ego-motion estimates.

[0129] Example 18: The method of Example 17, where determining the Ego-motion estimates with timestamps is performed by the second deep neural network prior to fusing data points of the radar point cloud.

[0130] Example 19: The method of any of the Examples 16 through 18, where determining the unsynchronized Ego-motion estimates with timestamps includes: receiving reflected signals at a LIDAR platform of the plurality of sensor circuits, the LIDAR platform including a front end circuit coupled to one or more optical sensors, the reflected signals corresponding to one or more objects in a view area of the LIDAR platform; generating a point cloud including a plurality of features for each of a plurality of data points corresponding to data determined from the reflected signals at processing circuitry of the LIDAR platform, the plurality of features including local features and global features; determining Ego-motion estimates with timestamps based on the point cloud using a second deep neural network; and providing the Ego-motion estimates with timestamps to the control circuit as part of the unsynchronized Ego-motion estimates.

[0131] Example 20: The method of any of the Example 17, 18, or 19, where determining the Ego-motion estimates with timestamps using the second deep neural network includes:determining point-wise weights for the radar point cloud; determining point-wise offsets for the radar point cloud; determining a weight subset of the point-wise weights that are greater than a threshold weight; and applying a weighted least square regression to selected data points corresponding to the weight subset including the local features, the global features, the point-wise weights, and the point-wise offsets to determine the Ego-motion estimates with timestamps.

[0132] The preceding detailed description is merely illustrative in nature and is not intended to limit the embodiments of the subject matter or the application and uses of such embodiments. As used herein, the word “exemplary” means “serving as an example, instance, or illustration.” Any implementation described herein as exemplary is not necessarily to be construed as preferred or advantageous over other implementations. Furthermore, there is no intention to be bound by any expressed or implied theory presented in the preceding technical field, background, or detailed description.

[0133] The connecting lines shown in the various figures contained herein are intended to represent exemplary functional relationships and / or physical couplings between the various elements. It should be noted that many alternative or additional functional relationships or physical connections may be present in an embodiment of the subject matter. In addition, certain terminology may also be used herein for the purpose of reference only, and thus are not intended to be limiting, and the terms “first”, “second” and other such numerical terms referring to structures do not imply a sequence or order unless clearly indicated by the context.

[0134] The foregoing description refers to elements or features being “connected” or “coupled” together. As used herein, unless expressly stated otherwise, “connected” means that one element is directly joined to (or directly communicates with) another element, and not necessarily mechanically. Likewise, unless expressly stated otherwise, “coupled” means that one element is directly or indirectly joined to (or directly or indirectly communicates with, electrically or otherwise) another element, and not necessarily mechanically. Thus, although the schematic shown in the figures depict one exemplary arrangement of elements, additional intervening elements, devices, features, or components may be present in an embodiment of the depicted subject matter.

[0135] While at least one exemplary embodiment has been presented in the foregoing detailed description, it should be appreciated that a vast number of variations exist. It should also beappreciated that the exemplary embodiment or embodiments described herein are not intended to limit the scope, applicability, or configuration of the claimed subject matter in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient road map for implementing the described embodiment or embodiments. Various changes can be made in the function and arrangement of elements without departing from the scope defined by the claims.

Claims

CLAIMSWhat is claimed is:

1. A system comprising: a control circuit comprising: one or more input / output (I / O) interfaces configured to couple to a plurality of sensor circuits coupled to a vehicle to receive unsynchronized Ego-motion data including data indicative of objects in a view area of one or more of the plurality of sensor circuits; one or more processor circuits coupled to the one or more I / O interfaces, the one or more processor circuits configured to: receive the unsynchronized Ego-motion data from one or more of the plurality of sensor circuits; select a subset of the unsynchronized Ego-motion data using a moving window; project features from the subset to a vehicle feature space using a deep neural network (DNN); extract fused temporal features from the subset using the deep neural network; predict, using the DNN, acceleration estimates, Kalman filter estimates, and dynamic weight estimates based on the fused temporal features; compensate the unsynchronized Ego-motion data using the acceleration estimates to produce compensated Ego-motion data; filter the unsynchronized Ego-motion data and the compensated Ego-motion data using a Kalman filter updated with the Kalman gain estimates to produce filtered Ego-motion data; and apply the dynamic weight estimates using a weighted moving average to fuse and smooth the filtered Ego-motion data to determine vehicle Ego-motion data corresponding to transitional motion and rotational motion of the vehicle and relative motion of the objects.

2. The system of claim 1, further wherein the plurality of sensor circuits comprises one or more of a radar platform or a light-detection and ranging platform.

3. The system of claim 1, wherein the plurality of sensors includes a plurality of radar platforms, each radar platform comprising: a radar front end circuit coupled to one or more antenna to receive reflected signals corresponding to one or more objects in a view area of the radar platform; circuitry configured to generate a radar point cloud including a plurality of features for each of a plurality of data points corresponding to data determined from the reflected signals, the plurality of features including local features and global features and including Doppler data and Angle of Arrival (AoA) data for each datum; and a second deep neural network configured to determine Ego-motion estimates with timestamps based on the radar point cloud and to provide the determined Ego-motion estimates with timestamps to the control circuit as part of the unsynchronized Ego-motion data.

4. The system of claim 3, wherein the unsynchronized Ego-motion data includes determined Ego-motion estimates from at least some of the plurality of radar platforms.

5. The system of claim 4, wherein, when one or more of the plurality of radar platforms fail, the DNN of the control circuit is configured to determine the vehicle Ego-motion data based on the unsynchronized Ego-motion data from others of the plurality of radar platforms.

6. The system of claim 3, wherein the second deep neural network processes the radar point cloud prior to fusion.

7. The system of claim 3, wherein the second deep neural network includes multiple upscaling and downscaling layers configured to determine hierarchical feature extraction to extract the local features and the global features, the Ego-motion estimates with timestamps include the local features and the global features.

8. The system of claim 3, wherein the second deep neural network of each of the plurality of radar platforms is configured to: determine point- wise weights for the radar point cloud; determine point-wise offsets for the radar point cloud; determine a weight subset of the point-wise weights that are greater than a threshold weight; and apply a weighted least square regression to selected data points corresponding to the weight subset including the local features, the global features, the point-wise weights, and the point-wise offsets to determine the Ego-motion estimates with timestamps.

9. The system of claim 3, wherein the DNN and the second deep neural networks of the plurality of radar platforms are trained together using data from the plurality of radar platforms with the radar point clouds of each of the plurality of radar platforms projected to the vehicle feature space.

10. The system of claim 1 , wherein the Kalman filter comprises a second order Kalman filter.

11. The system of claim 1, wherein the DNN is configured to concatenate the unsynchronized Ego-motion data from one or more of the plurality of sensor circuits to form a feature vector.

12. The system of claim 11 , wherein the DNN is configured to select the subset from the feature vector, wherein the subset has a configural length or time interval.

13. The system of claim 1, wherein the DNN fuses the subset of the unsynchronized Egomotion estimates from two or more of the plurality of sensor circuits from different times.

14. The system of claim 1 , wherein the Kalman filter estimates tune the Kalman filter to apply a gain to selected data points of the subset.

15. The system of claim 1 , wherein the dynamic weight estimates filter and smooth the subset to produce the vehicle Ego-motion data.

16. A method of determining vehicle Ego-motion data, the method comprising: determining unsynchronized Ego-motion estimates with timestamps at each of a plurality of sensor platforms, the unsynchronized Ego-motion estimates with timestamps including local features and global features; concatenating, at a control circuit, the unsynchronized Ego-motion estimates with timestamps and identifiers from at least some of the plurality of sensor platforms to form a feature vector including unsynchronized Ego-motion estimates from the plurality of sensor platforms at different times; selecting, at the control circuit, a subset of the unsynchronized Ego-motion estimates with timestamps using a moving data window having a selected length or time interval; fusing the subset using a deep neural network (DNN) of the control circuit; predicting, using the DNN, acceleration estimates, Kalman filter gain estimates, and dynamic weight estimates for the subset;applying, at the control circuit, the acceleration estimates to the unsynchronized Ego-motion estimates to compensate for under-estimation or over-estimation of Doppler measurement data to produce compensated Ego-motion data; applying, at the control circuit, the Kalman filter gain estimates to the unsynchronized Ego-motion estimates and the compensated Ego-motion data at a second order Kalman filter to produce filtered Ego-motion data; and applying, at the control circuit, the dynamic weight estimates as a weighted moving average to the filtered Ego-motion data to determine vehicle Ego-motion data corresponding to transitional motion and rotational motion of the vehicle and relative motion of the objects.

17. The method of claim 16, wherein determining the unsynchronized Ego-motion estimates with timestamps comprises: receiving reflected signals at a radar platform of the plurality of sensor circuits, the radar platform including a front end circuit coupled to one or more antennas, the reflected signals corresponding to one or more objects in a view area of the radar platform; generating a radar point cloud including a plurality of features for each of a plurality of data points corresponding to data determined from the reflected signals at processing circuitry of the radar platform, the plurality of features including local features and global features and including Doppler data and Angle of Arrival (AoA) data for each datum; determining Ego-motion estimates with timestamps based on the radar point cloud using a second deep neural network; and providing the Ego-motion estimates with timestamps to the control circuit as part of the unsynchronized Ego-motion estimates.

18. The method of claim 17, wherein determining the Ego-motion estimates with timestamps is performed by the second deep neural network prior to fusing data points of the radar point cloud.

19. The method of claim 16, wherein determining the unsynchronized Ego-motion estimates with timestamps comprises: receiving reflected signals at a LIDAR platform of the plurality of sensor circuits, the LIDAR platform including a front end circuit coupled to one or more optical sensors, the reflected signals corresponding to one or more objects in a view area of the LIDAR platform; generating a point cloud including a plurality of features for each of a plurality of data points corresponding to data determined from the reflected signals at processing circuitry of the LIDAR platform, the plurality of features including local features and global features; determining Ego-motion estimates with timestamps based on the point cloud using a second deep neural network; and providing the Ego-motion estimates with timestamps to the control circuit as part of the unsynchronized Ego-motion estimates.

20. The method of claim 17, wherein determining the Ego-motion estimates with timestamps using the second deep neural network comprises: determining point-wise weights for the radar point cloud; determining point-wise offsets for the radar point cloud; determining a weight subset of the point-wise weights that are greater than a threshold weight; and applying a weighted least square regression to selected data points corresponding to the weight subset including the local features, the global features, the point-wise weights, and the point-wise offsets to determine the Ego-motion estimates with timestamps.

Images