Radar Deep Learning

Abstract

Techniques for analyzing radar signals using deep learning are disclosed. In one aspect, an onboard computer of a host vehicle receives a plurality of radar frames from a radar sensor of the vehicle; executes a neural network on a subset of the plurality of radar frames; and detects one or more objects in the subset of the plurality of radar frames based on executing the neural network on the subset of the plurality of radar frames. In addition, techniques for transforming polar coordinates to Cartesian coordinates in a neural network are disclosed. In one aspect, the neural network receives a plurality of radar frames in a polar coordinate space, a polar-to-Cartesian transformation layer of the neural network transforms the plurality of radar frames to a Cartesian coordinate space, and the neural network outputs a plurality of radar frames in a Cartesian coordinate space.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims the benefit of U.S. Provisional Application No. 62 / 774,018, filed on November 30, 2018, entitled “RADAR DEEP LEARNING,” and U.S. Non-Provisional Application No. 16 / 689,870, filed on November 27, 2019, entitled “RADAR DEEP LEARNING,” both of which are assigned to their assignees and are expressly incorporated herein by reference in their entireties. Technical Field

[0003] Aspects of the present disclosure generally relate to radar deep learning. Background Art

[0004] Radar and camera sensors are often employed on vehicles to enable enhanced vehicle safety for systems such as adaptive cruise control (ACC), forward collision warning (FCW), collision mitigation or avoidance via autonomous braking), pre-collision features (such as airbag arming or pre-activation), and lane departure warning (LDW). Systems employing both radar and camera sensors can provide advanced active safety capabilities and are increasingly available in production vehicles.

[0005] One or more camera sensors mounted on the vehicle capture images of the scene in front of the vehicle (and possibly behind and to the sides of the vehicle). A digital signal processor (DSP) within the vehicle then attempts to identify objects within the captured images. Such objects may be other vehicles, pedestrians, road signs, objects within the path of travel, etc. Radar systems may also be used to detect objects along the path of travel. Radar systems utilize radio waves to determine the distance, altitude, direction, and / or speed of an object along the road. A transmitter sends a pulse of radio waves that bounces off an object in the path of the object. The pulse reflected from the object returns a smaller portion of the radio wave energy to the receiver, which is typically located at the same location as the transmitter.

[0006] Radar systems provide reasonably accurate measurements of object distance and velocity in all weather conditions. However, radar systems typically have insufficient resolution to identify features of detected objects. However, camera sensors typically do provide sufficient resolution to identify features of objects. Cues to object shape and appearance extracted from captured images can provide features sufficient to classify different objects. Given the complementary properties of the two sensors, data from the two sensors can be combined (referred to as "fusion") in a single system for improved performance.

[0007] Radar vision fusion methods differ primarily in the fusion level. Low-level fusion combines several sources of raw data to produce new raw data that is expected to be more informative and synthetic than the input. In mid-level fusion, various features (such as edges, corners, lines, texture parameters, etc.) are combined into feature maps that are then used by other processing stages. In high-level fusion, each input source produces a decision, and these decisions are fused. Summary of the invention

[0008] The following presents a simplified summary of the invention related to one or more aspects disclosed herein. Thus, the following summary should not be considered as an extensive overview related to all contemplated aspects, nor should the following summary be considered to identify key or important elements related to all contemplated aspects or to delineate the scope associated with any particular aspect. Accordingly, the following summary has the following sole purpose: to present certain concepts related to one or more aspects of the mechanism disclosed herein in a simplified form prior to the embodiments presented below.

[0009] In one aspect, a method of employing deep learning to analyze radar signals performed by an onboard computer of a host vehicle includes: receiving a plurality of radar frames from a radar sensor of the host vehicle; executing a neural network on at least a subset of the plurality of radar frames; and detecting one or more objects in the subset of the plurality of radar frames based on executing the neural network on the subset of the plurality of radar frames.

[0010] In one aspect, a method of transforming polar coordinates to Cartesian coordinates in a neural network executed by an onboard computer of a host vehicle includes receiving a plurality of radar frames or one or more potential representations of the plurality of radar frames in a polar coordinate space; transforming the plurality of radar frames or one or more potential representations of the plurality of radar frames from the polar coordinate space to a Cartesian coordinate space; and outputting the plurality of radar frames or one or more potential representations of the plurality of radar frames in the Cartesian coordinate space.

[0011] In one aspect, an onboard computer of a host vehicle includes at least one processor configured to: receive a plurality of radar frames from a radar sensor of the host vehicle; execute a neural network on at least a subset of the plurality of radar frames; and detect one or more objects in the subset of the plurality of radar frames based on executing the neural network on the subset of the plurality of radar frames.

[0012] In one aspect, an onboard computer of a host vehicle includes at least one processor configured to: receive a plurality of radar frames or one or more potential representations of the plurality of radar frames in a polar coordinate space; transform the plurality of radar frames or one or more potential representations of the plurality of radar frames from the polar coordinate space to a Cartesian coordinate space; and output the plurality of radar frames or one or more potential representations of the plurality of radar frames in the Cartesian coordinate space.

[0013] In one aspect, an onboard computer of a host vehicle includes: means for receiving a plurality of radar frames from a radar sensor of the host vehicle; means for executing a neural network on at least a subset of the plurality of radar frames; and means for detecting one or more objects in the subset of the plurality of radar frames based on executing the neural network on the subset of the plurality of radar frames.

[0014] In one aspect, an onboard computer of a host vehicle includes: means for receiving a plurality of radar frames or one or more potential representations of the plurality of radar frames in a polar coordinate space; means for transforming the plurality of radar frames or one or more potential representations of the plurality of radar frames from the polar coordinate space to a Cartesian coordinate space; and means for outputting the plurality of radar frames or one or more potential representations of the plurality of radar frames in the Cartesian coordinate space.

[0015] In one aspect, a non-transitory computer-readable medium storing computer-executable instructions includes computer-executable instructions including: at least one instruction to instruct an onboard computer of a vehicle to receive a plurality of radar frames from a radar sensor of a host vehicle; at least one instruction to instruct the onboard computer to execute a neural network on at least a subset of the plurality of radar frames; and at least one instruction to instruct the onboard computer to detect one or more objects in a subset of the plurality of radar frames based on executing the neural network on the subset of the plurality of radar frames.

[0016] In one aspect, a non-transitory computer-readable medium storing computer-executable instructions includes computer-executable instructions including: at least one instruction to instruct an onboard computer of a vehicle to receive a plurality of radar frames or one or more potential representations of the plurality of radar frames in a polar coordinate space; at least one instruction to instruct the onboard computer to transform the plurality of radar frames or one or more potential representations of the plurality of radar frames from the polar coordinate space to a Cartesian coordinate space; and at least one instruction to instruct the onboard computer to output the plurality of radar frames or one or more potential representations of the plurality of radar frames in the Cartesian coordinate space.

[0017] Other objects and advantages associated with the aspects disclosed herein will be apparent to those skilled in the art based on the accompanying drawings and detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are presented to aid in describing various aspects of the disclosure and are provided solely for illustration of the aspects and not limitation thereof.

[0019] Figure 1 A top view of a vehicle with an integrated radar-camera sensor behind the windshield according to various aspects.

[0020] Figure 2 An onboard computer architecture according to various aspects is shown.

[0021] Figure 3 is a schematic diagram of the sensed observation radar grid.

[0022] Figure 4 An exemplary difference between a camera image and a radar image of the same scene is shown.

[0023] Figure 5 is a block diagram illustrating how a radar-camera sensor module and / or one or more processors process data from a radar sensor in accordance with various aspects.

[0024] Figure 6 An example of computer vision where occlusion has occurred is shown.

[0025] Figure 7 A series of key frames and p-frames in a radar frame sequence are shown.

[0026] Figure 8 An architecture is shown that uses a separate feature extraction layer for each 2D segment from a set of selected segments before fusing them together and subsequently fusing them with the rest of the network.

[0027] Fig. 9 Latent spatial ego-motion compensation techniques in accordance with various aspects of the present disclosure are shown.

[0028] Fig.10 A video frame, a radar frame in polar coordinate space, and an interpolated radar frame in Cartesian coordinate space are shown.

[0029] Fig.11 It is shown that the sampling grid in the polar coordinate space is transformed into the sampling grid in the Cartesian coordinate space.

[0030] Fig.12 An exemplary neural network architecture for vehicle detection in a highway automobile scenario is shown in accordance with various aspects.

[0031] Fig.13 The sampling grid is shown.

[0032] Fig.14 A specific neural network architecture is shown.

[0033] Fig.15 An exemplary method of analyzing radar signals using deep learning according to aspects of the present disclosure is shown.

[0034] Fig.16 An exemplary method for transforming polar coordinates to Cartesian coordinates in a neural network is shown. DETAILED DESCRIPTION

[0035] Various aspects of the present disclosure are provided in the following description and in the accompanying drawings for various examples provided for illustrative purposes. Alternative aspects may be designed without departing from the scope of the present disclosure. In addition, well-known aspects of the present disclosure may not be described in detail or may be omitted to avoid obfuscation of more relevant details.

[0036] Those skilled in the art will appreciate that any of a variety of different technologies and techniques may be used to represent the information and signals described below. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the following description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof, depending on the specific application, partly on the desired design, partly on the corresponding technology, etc.

[0037] Furthermore, many aspects are described in terms of sequences of actions performed, for example, by elements of a computing device. It will be appreciated that the various actions described herein may be performed by specific circuits (e.g., application specific integrated circuits (ASICs)), by program instructions executed by one or more processors, or by a combination of both. Additionally, for each aspect described herein, the corresponding form of any such aspect may be implemented, for example, as “logic configured to perform the described actions.”

[0038] Reference now Figure 1 , a vehicle 100 (referred to as the “ego vehicle” or “host vehicle”) is shown including a radar-camera sensor module 120 located in an interior compartment of the vehicle 100 behind a windshield 112. The radar-camera sensor module 120 includes a radar assembly configured to transmit radar signals through the windshield 112 in a horizontal coverage area 150 (shown by dashed lines) and receive reflected radar signals reflected from any object within the coverage area 150. The radar-camera sensor module 120 further includes a camera assembly for capturing images based on light waves seen and captured through the windshield 112 in a horizontal coverage area 160 (shown by dashed lines).

[0039] although Figure 1 An example is shown in which the radar assembly and the camera assembly are collocated components in a shared housing, but it should be understood that they may be separately housed in different locations within the vehicle 100. For example, the camera may be as shown. Figure 1 , and the radar assembly may be located in the guardrail or front bumper of the vehicle 100. In addition, although Figure 1 The radar-camera sensor module 120 is shown located behind the windshield 112, but it may alternatively be located in a roof sensor array or elsewhere. Figure 1Only a single radar-camera sensor module 120 is shown, but it should be understood that the vehicle 100 may have multiple radar-camera sensor modules 120 pointing in different directions (to the side, the front, the rear, etc.). The various radar-camera sensor modules 120 may be located below the "skin" of the vehicle (e.g., behind the windshield 112, doors, bumpers, guardrails, etc.) or in a roof-mounted sensor array.

[0040] The radar-camera sensor module 120 may detect one or more objects (or no objects) relative to the vehicle 100. Figure 1 In the example of FIG. 1 , there are two objects within the horizontal coverage areas 150 and 160 that the radar-camera sensor module 120 can detect: vehicles 130 and 140. The radar-camera sensor module 120 can estimate parameters of the detected object(s), such as position, distance, direction, speed, size, classification (e.g., vehicle, pedestrian, road sign, etc.), etc. The radar-camera sensor module 120 can be employed on the vehicle 100 for automotive safety applications such as adaptive cruise control (ACC), forward collision warning (FCW), collision mitigation or avoidance via autonomous braking, lane departure warning (LDW), etc.

[0041] Collocating the camera with the radar allows these components to share electronics and signal processing, and in particular enables early radar-camera data fusion. For example, the radar and camera can be integrated onto a single board. Joint radar-camera alignment techniques can be used to align both the radar and the camera. However, collocation of the radar and camera is not required to practice the techniques described herein.

[0042] Figure 2 An onboard computer (OBC) 200 of a vehicle 100 is shown in accordance with various aspects of the present disclosure. In one aspect, the OBC 200 may be part of an ADAS or ADS. The OBC 200 includes a non-transitory computer-readable storage medium (i.e., memory 204) and one or more processors 206 that communicate with the memory 204 via a data bus 208. The memory 204 includes one or more storage modules that store computer-readable instructions that can be executed by the processor(s) 206 to perform the functions of the OBC 200 described herein. For example, the processor(s) 206 in conjunction with the memory 204 may implement the various neural network architectures described herein.

[0043] One or more radar-camera sensor modules 120 are coupled to the OBC 200 (at Figure 2In some aspects, the radar-camera sensor module 120 includes at least one camera 212, at least one radar 214, and optionally a light detection and ranging (LiDAR) sensor 216. The OBC 200 also includes one or more system interfaces 220 that connect the processor(s) 206 to the radar-camera sensor module 120 and optionally to other vehicle subsystems (not shown) via a data bus 208.

[0044] At least in some cases, the OBC 200 also includes a wireless wide area network (WWAN) transceiver 230 configured to communicate via one or more wireless communication networks (not shown) (such as NR networks, LTE networks, GSM networks, etc.). The WWAN transceiver 230 can be connected to one or more antennas (not shown) to communicate with other network nodes (such as other vehicle UEs, pedestrian UEs, infrastructure access points, roadside units (RSUs), base stations (e.g., eNBs, gNBs), etc.) via at least one designated RAT (e.g., NR, LTE, GSM, etc.) through a wireless communication medium of interest (e.g., a certain set of time / frequency resources in a specific frequency spectrum). The WWAN transceiver 230 can be configured in different ways according to the designated RAT for sending signals (e.g., messages, indications, information, etc.) and encoding them, and conversely for receiving signals (e.g., messages, indications, information, pilots, etc.) and decoding them.

[0045] At least in some cases, the OBC 200 also includes a wireless local area network (WLAN) transceiver 240. The WLAN transceiver 240 can be connected to one or more antennas (not shown) to communicate with other network nodes (such as other vehicle UEs, pedestrian UEs, infrastructure access points, RSUs, etc.) over a wireless communication medium of interest via at least one designated RAT (e.g., cellular vehicle-to-everything (C-V2X), IEEE 802.11p (also known as wireless access in vehicular environment (WAVE)), dedicated short range communication (DSRC), etc.). The WLAN transceiver 240 can be configured in different ways according to the designated RAT for sending signals (e.g., messages, indications, information, etc.) and encoding them, and conversely for receiving signals (e.g., messages, indications, information, pilots, etc.) and decoding them.

[0046] As used herein, a "transceiver" may include transmitter circuitry, receiver circuitry, or a combination thereof, but need not provide both transmit and receive functionality in all designs. For example, a low-functionality receiver circuit (e.g., a receiver chip or similar circuit that provides only low-level sniffing) may be employed in some designs to reduce cost when full communication need not be provided.

[0047] At least in some cases, OBC 200 also includes a global positioning system (GPS) receiver 250. GPS receiver 250 can be connected to one or more antennas (not shown) for receiving satellite signals. GPS receiver 250 can include any suitable hardware and / or software for receiving and processing GPS signals. GPS receiver 250 requests information and operations from other systems as appropriate, and uses measurements obtained by any suitable GPS algorithm to perform calculations required to determine the location of vehicle 100.

[0048] In one aspect, the OBC 200 may utilize the WWAN transceiver 230 and / or the WLAN transceiver 240 to download one or more maps 202, which may then be stored in the memory 204 and used for vehicle navigation. The map(s) 202 may be one or more high definition (HD) maps that may provide accuracy within 7 cm to 10 cm absolute distance, a highly detailed inventory of all stationary physical assets associated with the road (such as road lanes, road edges, shoulders, dividers, traffic signals, signs, painted markings, poles), and other data useful for safe navigation of roads and intersections by the vehicle 100. The map(s) 202 may also provide electronic horizon prediction awareness that enables the vehicle 100 to know what is ahead.

[0049] In one aspect, the camera 212 may capture the visible area of ​​the camera 212 at a periodic rate (e.g., Figure 1 Similarly, radar 214 may capture image frames (also referred to herein as camera frames) of a scene within the visible area of ​​radar 214 (e.g., shown as horizontal coverage area 160 in FIG. 1 ). Figure 1 150). The periodicity rates at which camera 212 and radar 214 capture their respective frames may be the same or different. Each camera frame and radar frame may be time-stamped. Thus, in the case of different periodicity rates, the timestamps may be used to select captured camera frames and radar frames simultaneously or nearly simultaneously for further processing (e.g., fusion).

[0050] Figure 3 2. A radar grid 300 of sensed observations is shown. A transmitter (eg, a transmit antenna array) of radar 214 transmits signals from an object (eg, Figure 1 A portion of the electromagnetic RF waves reflected from the object(s) are returned to a receiver (e.g., a receiving antenna array) of radar 214, which is typically located at the same site as the transmitter of radar 214.

[0051] In one aspect, radar 214 may be an imaging radar that uses beamforming to scan horizontally and vertically. Beamforming is a technique for aiming the effective direction of a radar beam by varying the delays between different transmit antennas so that the signal constructively increases in a specified direction. Thus, radar 214 may scan a sensing region (e.g., horizontal coverage region 150) horizontally and vertically by using a transmitter that includes an electronically steered antenna array.

[0052] The returned response measured by the radar 214 is characterized as an observation grid 340 having a plurality of observation cells 342. Each cell 342 represents the measured returned response value at a specific distance (r) and angle / azimuth (θ). Each cell 342 is alternately referred to as a range-angle bin. Features 344 are extracted from the cells 342 to determine whether the feature 344 is an object (e.g., a vehicle 130 / 140). Each feature 344 within the corresponding cell 342 can be identified as having up to four parameters: distance, Doppler, azimuth, and elevation. This is referred to as a radar frame. As an example, the feature 344 within the cell 342 can be a signal-to-noise ratio (SNR) calculated by a constant false alarm rare (CFAR) algorithm. However, it should be understood that other methods can be used to target and identify the feature 344 within the cell 342.

[0053] Compared to images (e.g., images from camera 212), radar signals (e.g., radar signals from radar 214) have several unique characteristics. One characteristic is specular reflection, where only certain surfaces on a target with a favorable orientation reflect the radar signal, which typically results in a small number of reflections. A second characteristic is non-uniformity, where objects that appear at large distances in {range, azimuth} signal space are consistently smaller and have a different shape than those that appear at closer distances. This is not the case with camera images.

[0054] Figure 4 An exemplary difference between a camera image 410 and a radar image 420 of the same scene is shown. Camera image 410 may have been captured by camera 212, while radar image 420 may have been captured by radar 214. In camera image 410, three objects (here, vehicles) have been detected, each surrounded by bounding boxes 412, 414, and 416. In radar image 420, the same three objects (vehicles) have been detected and are surrounded by bounding boxes 422, 424, and 426. Figure 4As can be seen in FIG. 4 , the shapes of objects / bounding boxes 422, 424, and 426 in radar image 420 appear different sizes and shapes based on whether they are located below centerline 428 of radar image 420 (i.e., bounding box 422) or above the centerline (i.e., bounding boxes 424 and 426). Additionally, since fewer reflected radio waves are received at radar 214 due to the greater distance between radar 214 and other objects, the bounding box associated with the closer object (i.e., bounding box 422) is larger than the bounding boxes associated with the farther object (i.e., bounding boxes 424 and 426).

[0055] Figure 5 5 is a block diagram 500 illustrating how radar-camera sensor module 120 and / or processor(s) 206 process data from radar 214 in accordance with various aspects of the present disclosure. The data is received as analog data by analog front end (AFE) 502 and converted to digital data by digital front end (DFE) 504. Digital front end 504 outputs I / Q samples 506 of the digital data. I / Q samples 506 are raw data that are processed to generate tensors to be used during back end processing.

[0056] The computational front end 508 performs tensor generation, windowing, subsampling, fast Fourier transform (FFT), motion compensation, etc. to generate a four-dimensional (4D) tensor 510, which represents, for example, features 344 within a cell 342 of an observation grid 340 detected by the radar 214. In one aspect, the radar 214 can be an imaging radar that scans horizontally and vertically. Thus, the tensor 510 represents the distance (the distance from the vehicle 100 to the detected feature 344), the azimuth (the horizontal distance between the feature 344 and a reference RF ray emitted by the radar 214 (such as the initial RF ray of the radar sweep)), the Doppler (indicating the speed of the detected feature 344), and the altitude (the vertical direction from the radar 214 to the detected feature 344) of each detected feature 344. These tensors are then passed to the computational back end 512, which performs object detection, object classification, localization, and attribute estimation based on the tensor 510 and the image received from the camera 212.

[0057] The results 514 of the processing performed by the computational backend 512 are various attributes of the detected object(s), such as a bounding box tightly enclosing the object(s) in Cartesian xy coordinates, the type of the object(s) (e.g., car, truck, pedestrian, motorcycle, road sign, etc.), relative speed (e.g., relative to the vehicle 100), orientation, softemax function scores (probability of a given output of a tensor in binary classification) for all attributes, predicted positions of the object(s) after a certain amount of time (e.g., 50ms, 100ms, etc.), and the like.

[0058] The results 514 may be fed back to the computational front end 508 and the digital front end 504 as configurations 516 and 518, respectively. The configuration 516 may include a range-azimuth-Doppler (RAD) tensor of the detected object(s), a range-azimuth (RA) tensor (in the case of Doppler dimension collapse), a range-azimuth, range-Doppler, and azimuth-Doppler tensors, etc. The computational front end 508 may use this information to better identify these attributes of the other features 344. The configuration 518 may include waveform parameters that the digital front end 504 may use to better convert the analog data from the analog front end 502 into the I / Q samples 506.

[0059] Referring to the computational front end 508 in more detail, the computational front end 508 receives as input the I / Q samples 506, from which it computes and outputs a radar frame. The computational front end 508 may output a 4D composite value tensor representing range, azimuth, Doppler, and altitude. Optionally, the computational front end 508 may perform sub-selection, grouping, compression, "argmax" (a point in a function domain where the function value is maximized), or "any function" of one or more of these dimensions. Alternatively, the computational front end 508 may output a 3D composite value tensor representing range, azimuth, and Doppler. As yet another alternative, the computational front end 508 may output a set of two-dimensional (2D) composite value tensors representing one or more of range and azimuth, range and Doppler, Doppler and azimuth, range and altitude, Doppler and altitude, or azimuth and altitude. The salient information is sometimes contained in the frequency content of the signal, in which case further signal processing may be performed to extract such frequency information.

[0060] In one aspect, the computational front end 508 can perform normalization on the radar signal. To do this, it computes the mean and standard deviation of the radar signal at each distance (using the entire data set, i.e., all training frames). It then fits a function (e.g., a linear function) to the data and uses it to normalize the radar signal along the distance dimension. This provides a small gain in accuracy (approximately 1%).

[0061] Referring to the computational backend 512 in more detail, the computational backend 512 may implement various "deep learning" techniques (machine learning methods based on learned data representations, rather than task-specific algorithms) to determine results 514. Deep neural networks may be used to detect, classify, locate, and understand objects detected from radar signals. A deep neural network (DNN) is an artificial neural network (ANN) having multiple "layers" between the input layer and the output layer. Specifically, a DNN performs multiple mathematical operations (called "layers") to calculate the probability of a specific output based on a given input. The result of each of these operations (or layers) is a "potential representation" or "potential feature map", which is no longer "potential" except for the final or final result, but is the prediction / response / output of the model. For example, a DNN trained to recognize the type of object that may be encountered by an autonomous or semi-autonomous vehicle (e.g., vehicle 100) will analyze a given radar frame and calculate the probability that each object detected in the frame is a vehicle, a pedestrian, a road sign, etc. During training of the DNN, the user can inspect the results and select which probabilities the DNN should display (e.g., outputs with probabilities above a certain threshold, etc.), and the proposed labels are returned.

[0062] On the one hand, a recursive neural network (RNN) can be used as a deep learning technique by the computing backend 512 to determine the result 514. RNN is a kind of ANN in which the connection between nodes forms a directed graph along a time series, which allows the RNN to exhibit temporal dynamic behavior. RNN elements can be used in radar object detection neural networks to improve detection accuracy. Convolutional long short-term memory (LSTM) (a kind of RNN that can learn long-term correlation), convolutional gated recurrent unit (GRU) (similar to LSTM with forget gate, but because it lacks output gate and has fewer parameters than LSTM) or stacked RNN can be applied to the potential feature map of the object detection neural network to build the memory of the latest observation. In fact, convolutional GRU is more parameter efficient. However, convolutional LSTM achieves slightly higher accuracy (in terms of mean average precision (mAP), mean absolute error (MAE), accuracy and recall, etc.). In fact, stacked convolutional GRU does not achieve the same accuracy as single-layer convolutional LSTM. This is likely due to the need for a longer history for achieving high accuracy. With the same accuracy as a fully trained radar object detection neural network as the goal, the number of parameters can be cut by about four times. Since adding memory elements does not add any significant burden, the inference time is more or less the same as the base network (a network without (multiple) LSTM / RNN / GRU modules that can be added on top of the "base network"). Compared with conventional vehicle radars that can only estimate the radial component of the relative velocity, the use of RNNs can estimate the complete relative velocity vector of the object. More specifically, RNNs have the ability to view the position of an object through multiple time steps (relative to only at a given point in time). Based on a known position over a period of time, calculating the speed is just calculating how fast the position is moving. Therefore, in conventional radars and when using only radar signals, only relative radial velocity (i.e., the part of the relative velocity close to or away from the radar) can be measured. The speed in other directions cannot be detected. However, the speed in other directions can also be estimated using RNNs with "history". Therefore, the "complete" velocity vector is a complete 3D vector, not just the radial part.

[0063] It should be noted that convolution is a mathematical operation on two functions to produce a third function that represents how the shape of one function is modified by the other. The term "convolution" refers to the resulting function and to the process of computing that function. Convolution is similar to cross correlation; for discrete, real-valued functions, they differ in that the order of one of the functions is reversed.

[0064] In one aspect, spatially variable convolutions (i.e., different convolution weights for different locations) can be used by the computational backend 512 as a deep learning technique to determine the results 514. In a first solution, local spatial convolution kernels (convolution operations act on two signals, radar frames or potential representations, and kernels, and combine them into one) can be applied to the early layers (i.e., the first stage) of the radar-based object detection neural network. Compared to camera-based object detection neural networks, radar frames do not have spatially constant properties, which means that in camera frames, no matter where the object is, it looks more or less the same (regardless of the viewing angle). However, in radar frames, the range-azimuth trajectory changes significantly because the ego vehicle (e.g., vehicle 100) moves along either the range or azimuth. Thus, local spatial convolutions can be used in the early stages of the object detection neural network to apply different convolution weights (i.e., learned model parameters) to different locations on the radar frame or potential representation. After a few layers, these features are more evenly distributed in space, and weighted shared convolutions (also called group convolutions) can be used in subsequent layers of the network. Since the depth of the feature map can be relatively small in the early stages of an object detection neural network, this should not add very high overhead in terms of the number of weights.

[0065] In the second solution using spatially variable convolutions, micro neural networks (smaller auxiliary neural networks whose purpose is not to solve the original problem (detecting objects), but to generate these spatially variable kernels and weights) can be used to generate spatially variable kernels and weights. Similar to the previous solution, the number of weights required to perform spatially variable convolutions can be further reduced. The idea is to use a shallow network (potentially only a few layers) to generate all the weights required for different kernels. Under the condition that the Kolmogorov complexity of this problem (the length of the shortest computer program (using a predetermined programming language) that produces an object as output) does not seem too high, this object detection neural network can be seen as a way to compress the weights that the local spatial convolution kernel itself can generate. This network generates a large number of weights that should be highly correlated and can potentially be used to generate weights for multiple layers. Techniques similar to dynamic filter networks (networks in which filters are dynamically generated conditioned on the input) can also be used to dynamically generate these weights by using {distance, azimuth} pairs as inputs to the object detection neural network for situations where storage or bandwidth may be limited and these weights are generated "on the fly".

[0066] In a third solution using spatially variable convolutions, explicit transformations can be used to generate weights for spatially variable kernels applied to a radar-based object detection neural network. An alternative to generating weights for different kernels of local spatial convolutions using an object detection neural network is to use explicit transformations on the weights. Assume that the oracle filter required for one channel of one of the feature maps is known, and the transformations for all other positions of the same channel (i.e., the same color) and the same feature map are known (assuming that the space is constant in the Cartesian domain). One problem is that the size of the filter is typically small. To address this problem, the size of the filter can be increased to a larger size (e.g., 7×7), then the transformation can be applied (in a continuous domain), and the resulting kernel can be clipped back to, for example, 3×3 or 5×5. It should be understood that the second solution described above is more efficient than the first solution, and the third solution is more efficient than the second solution.

[0067] The uncertainty of the detection of the object detection neural network can also be estimated based on the output of the network. In the case of autonomous driving, most of the time, the object is not in the bird's eye view (captured by the radar image) Figure 1 This is quite different from camera-vision based networks where occlusions may also occur in conventional settings. For example, Figure 6 As shown in image 600 in , three different bounding boxes corresponding to three different objects overlap each other. On the one hand, non-maximum suppression (combining multiple bounding boxes around the target object to reduce to one bounding box) can be used in the case of object detection neural networks to suppress overlapping bounding boxes, thereby reducing the number of remaining bounding boxes and keeping those bounding boxes with the highest confidence. Since there should be no overlapping vehicles in the case of radar-based object detection for vehicles, the parameters of non-maximum suppression can be changed to account for this difference (e.g., the overlap threshold and the number of potential bounding boxes can be changed).

[0068] Multiple overlapping bounding box predictions can be used to estimate the network's uncertainty about the predictions. In most cases, unlike camera images, for radar frames, only one of the overlapping boxes can correspond to a physical target. Since the object detection neural network always outputs a set number of bounding boxes, there needs to be a final step in the algorithm to keep only the highest confidence detections when multiple detections correspond to the same object. However, patterns (e.g., density, diffusion, etc.) in which detections (corresponding to the same object) overlap each other can be used to derive the uncertainty of the system. For example, if the detections (corresponding to the same object) have large changes in position, it may indicate that the object detection neural network is uncertain about the location of the object to be detected. This information can be used to estimate the network's uncertainty about the detection location.

[0069] As another technique disclosed herein for determining results 514, the computational backend 512 can use a recursive radar-based object detection neural network to significantly speed up the inference time (i.e., the time it takes for the model (neural network) to perform the calculations required to get the final prediction based on the input). Typically, the neural network techniques described above will be performed on each radar frame. An effective way to speed up the average inference time is to run the network only on a subset of radar frames and predict object detections in omitted steps (i.e., frames in this context). Similar to a Moving Picture Experts Group (MPEG) encoder, a subset of radar frames can be identified as key frames (which can be evenly or unevenly distributed in the timeline). These key frames (e.g., Figure 7 The key frames 702 in the image are fed to the RNN radar-based object detection neural network. Then, the training is expected to predict the frames between the key frames 702 (called “p frames” and Figure 7 A network that outputs bounding boxes for frames (shown as p frame 704 in the figure). This network learns how to generate (or better synthesize) high-level feature maps for future frames so that the single-shot detection (SSD) head can perform effective object detection on these future frames that the network will not be able to see. This technique should provide significant improvements in inference time.

[0070] SSD is a method for detecting objects in images using a single deep neural network. SSD discretizes the output space of bounding boxes into a set of default boxes with different aspect ratios and scales them by feature map locations. At prediction time, the object detection neural network generates scores for the presence of each object category in each default box and produces adjustments to the box to better match the object shape. In addition, the network combines predictions from multiple feature maps with different resolutions to naturally handle objects of various sizes. The SSD model eliminates proposal generation and subsequent pixel or feature resampling stages and encapsulates all computations in a single network. This makes SSD easy to train and allows SSD to be directly integrated into systems that require detection components.

[0071] Expanding on this point, hybrid tracking techniques can be used to speed up inference. In the case of autonomous driving, given the fact that bounding boxes typically do not change drastically from one frame to the next, the radar-based object detection neural network can be run on only a subset of frames (e.g., key frames 702). For p frames (e.g., p frames 704), a simple tracking approach (e.g., bilinear extrapolation / interpolation - linear interpolation of an interpolation function of two variables extending over a 2D grid) or a more complex tracking approach (e.g., Kalman filtering with track segments (small subsegments of a path or "track" of an object between its origin and destination)) can be used to update the position of the bounding box. In practice, this can result in a small percentage reduction in mAP, but with a corresponding speed.

[0072] However, if a new object suddenly appears in a frame, it may be missed by not running the full object detection neural network on every frame. One way to solve this problem is to have a shallow anomaly detection network that can tell when the boundaries of a keyframe should be reset (or act as a keyframe detector). Such a network can be trained using the differences in ground truth (GT) (information provided by direct observation, not by inference) between frames when running the full object detection neural network. This means that the anomaly detection network can be trained to detect when new objects appear in the scene by using GT boxes (i.e., annotations). The annotations are tracked by having a start and end time. Therefore, using the GT annotations, it is easy to tell when a frame is "anomalous" - the frame where the object first appears.

[0073] On the other hand, the computational backend 512 may use a neural network architecture that consumes multiple 2D slices of a 4D {range, azimuth, doppler, altitude} tensor (referred to herein as a "SliceNet"). A "slice" is the result of a dimensionality reduction method, such as summing the power (of radar returns) along one or more dimensions. Previously, radar perception relied solely on range-azimuth slices for detecting objects. As a result, doppler information was effectively ignored. In contrast, Figure 8A SliceNet architecture 800 is shown that uses a separate feature extraction layer for each 2D segment (e.g., range-azimuth 802, range-Doppler 804, and azimuth-Doppler 806) from a set of selected segments before fusing the set of selected segments together as a 3D tensor 810 and then fusing it with the rest of the object detection neural network (i.e., the remaining layers of the object detection network appear after this tensor). This means that more convolutional layers follow immediately after the processed segments have been merged into the 3D tensor. Therefore, each segment is processed by a dedicated convolutional neural network branch. The results are then fused into a single 3D tensor, and another convolutional neural network takes this 3D tensor as input and ultimately outputs an object detection. The combined architecture is called "SliceNet." This produces improved performance for object classification and detection at the computational backend 512.

[0074] Latent space ego-motion compensation techniques are also disclosed herein. When input frames with consecutive timestamps are fed to an object detection neural network, ego-motion (the movement of the radar system itself (e.g., radar 214 when vehicle 100 is moving)) makes the input from the radar system more difficult to interpret by the computational backend 512. This can be particularly problematic when trying to predict the motion of detected objects (e.g., other vehicles, pedestrians, etc.). For example, as described above with reference to Figure 4 As discussed, radar imaging is not translationally equivariant, so compensation for ego-motion is important. For example, the radar signal cannot translate in the image plane. The current state of the art LiDAR-based approaches rely on ego-motion compensation, as described, for example, in "Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net" by Luo et al., which is incorporated herein by reference in its entirety.

[0075] In order to solve these problems, the present disclosure provides Fig. 9 The latent space ego motion compensation technique shown in . Fig. 9, each input frame 902 and 904 (received at t1 and t2, where the horizontal lines represent the changes in the position of the radar sensor (e.g., radar 214)) is fed to feature extractors 906 and 908, respectively (which can be the same feature extractor, although two feature extractors are shown to more clearly illustrate the process flow). Feature extraction also includes a polar to Cartesian coordinate transformation. The feature map 910 / 912 generated from one frame 902 / 904 is a representation of a given spatial region. The feature maps 910 and 912 are in "ego coordinates", which means that the ego point (i.e., the location of the radar sensor) is always at a fixed location on the radar frame or latent representation. Therefore, the movement of stationary objects on the map is opposite to the ego motion.

[0076] Next, the ego motion between frames 902 and 904 is obtained. The ego motion between frames 902 and 904 is the change in the position of the radar sensor. This can be obtained in various ways (such as GPS or other sensors), or the neural network can estimate the motion, including rotation (i.e., change in orientation of the vehicle 100).

[0077] Next, the ego coordinates of each key frame 902 and 904 are converted to world coordinates. The conversion process begins by initializing an empty feature tensor that is spatially larger than the original empty feature tensor (feature maps 910 and 912 corresponding to frames 902 and 904). This new tensor is "world coordinates", meaning that stationary objects remain in the same position on the feature map over time, and the ego position can change (depending on ego motion). Based on the ego motion according to the previous step of the process, the feature maps 910 and 912 of each frame 902 and 904 are transformed into new feature maps 914 and 916 in world coordinates, respectively. This includes filling the feature maps 914 and / or 916 with zeros in areas that have not been seen but can potentially be seen in other frames.

[0078] exist Fig. 9 In the example of , the second frame 904 is selected as the world frame, and therefore, there is no transformation between the feature maps 912 and 916. In contrast, the first frame 902 is transformed. Fig. 9 In the example above, this transformation is just a translation on the x-axis.

[0079] Next, the resulting set of feature maps (i.e., world coordinate feature maps 914 and 916) are combined into a combined feature map 918. They can be combined in various ways. As a first option, an indexing process can be used to combine them. For example, if the area is not seen in any of the frames, it is left as a "0" value. If the area is only seen on one frame, those values ​​are used. And if the area is seen on multiple frames, for example, the average of those values ​​is used. As a second option, a neural network can be used. The network will take multiple feature maps 914 and 916 in world coordinates as input and produce a combined feature map 918. The combined feature map 918 is then processed by the SSD 920 that outputs the object detection 922.

[0080] Also disclosed herein are techniques for cross-sensor annotation of radar signals (e.g., output of radar 214) using LiDAR sensor 216 and camera 212. LiDAR is a detection technology in which LiDAR sensor 216 transmits a pulsed laser and the sensor measures the reflected laser pulses. Differences in laser pulse return times and wavelengths can be used to determine the distance to an object in the path of the laser pulse.

[0081] The radar signal may be automatically or manually annotated with data from the LiDAR sensor 216. For automatic LiDAR annotation, the LiDAR signal may be used to detect and locate objects of interest (e.g., vehicles, pedestrians, road signs, etc.) (e.g., determine the general location of the object of interest). The object of interest within the generated LiDAR frame may then be identified with a 3D bounding box. The bounding box may be mapped from the LiDAR frame to the radar frame using an extrinsic calibration between the LiDAR sensor 216 and the radar 214.

[0082] Manual annotations can be bootstrapped with the automatic annotations described above. For example, radar frames can be manually annotated by a person from scratches, or the automatic annotations can be corrected by visualizing data from the three complementary sensors, camera 212, LiDAR 216, and radar 214.

[0083] On the one hand, if Fig.10As shown in method 1000 in "Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net", early LiDAR radar fusion can be performed to improve the quality of automatic annotation. The method disclosed in "Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net" can be used to convert LiDAR point cloud data (1004) into a bird's-eye view range-azimuth tensor (1006). This tensor can be concatenated (1008) with the raw radar data (1002), which is used as input to the object detection neural network (1010) to perform object detection (1012). Radar and LiDAR complement each other because radar is able to detect objects at long distances, while LiDAR provides details at short distances. In addition, when the LiDAR is mounted on the top of the vehicle 100, the LiDAR can sometimes "see" other vehicles from above.

[0084] On the other hand, if Fig.11 As shown in method 1100 in "Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net", intermediate LiDAR radar fusion can be performed to improve the quality of automatic annotation. Moreover, the method disclosed in "Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net" can be used to convert the LiDAR point cloud data (1108) into a bird's-eye view Cartesian tensor (1110). The neural network (1104; 1112) processes the raw radar data (1102) and the LiDAR tensor (1110) in two separate branches to extract features. The radar features are transformed to Cartesian coordinate space (1106) using the method described below. The radar and LiDAR features are then concatenated (1114) for further processing by the neural network (1116) for object detection (1118).

[0085] Also disclosed herein is a neural network with polar to Cartesian transformation for radar signal processing. The goal of imaging radar for autonomous driving is a radar-based system (road world model) that produces a high-resolution semantic representation of the output world. The road world model can provide the location of other vehicles, pedestrians and static objects, their dimensions, speed, etc. Compared to traditional radars, imaging radars are achieved through a combination of improved hardware (e.g., more antennas) and improved signal processing. On the signal processing side, deep learning techniques are used to interpret radar signals (as in the various techniques described above).

[0086] The "native format" of radar signals is in polar coordinate space. In polar coordinate space, one axis represents range and the other axis represents azimuth, such as Fig.12 As shown in , a radar frame is a 2D "image" of the power of the radar signal in each range-azimuth grouping. Fig.12 , the camera frame 1210 is on the left, the original radar frame 1220 in polar coordinate space is in the center, and the interpolated radar frame 1230 in Cartesian coordinate space is on the right. Ground truth annotation boxes 1222 and 1232 representing the location of the object (here, the vehicle) are overlaid on the radar frame 1220 and the interpolated radar frame 1230, respectively.

[0087] This polar coordinate space is caused by the way radar sensors work. For example, the azimuth direction is measured as the phase difference between adjacent antennas and retrieved using an FFT algorithm. Radar can in principle sense 180 degrees, but the resolution will be lower towards the leftmost and rightmost edges of the resulting radar frame. Because the preferred output space for automotive applications is Cartesian (xy), the polar coordinate space poses challenges for deep learning signal processing.

[0088] Fig.13 A sampling grid 1310 in polar coordinate space on the left is shown, which is transformed into a sampling grid 1320 in Cartesian coordinate space on the right. As can be seen, the sampling grid 1320 becomes sparser in Cartesian coordinate space the further away from the radar source (i.e., the bottom center of the sampling grid 1320). Fig.13 It is also shown that the resolution becomes lower towards the left and right edges of the sampling grid 1320.

[0089] The present disclosure proposes a neural network that takes its input map in polar coordinate space, but makes its output in Cartesian coordinate space. Fig.14An example neural network architecture 1400 for object detection in a highway car scene is shown. The transformation is performed inside the neural network by a fixed-function polar-to-Cartesian transformation layer 1410. That is, the neural network architecture 1400 does not have to learn the transformation. The fixed-function polar-to-Cartesian transformation layer 1410 uses a nearest neighbor mapping (an algorithm for estimating the value at a point in space when only the true value of a discrete point is known by taking the value at the nearest known sample), bilinear interpolation, or bicubic interpolation (an extension of cubic interpolation for interpolating data points on a two-dimensional regular grid) to map features from polar coordinate space (represented as polar coordinate space frame 1402) to Cartesian coordinate space (represented as Cartesian coordinate space frame 1404). After the transformation, the neural network architecture 1400 continues with several more convolutional layers (represented by feature maps with resolution "64×64"). Finally, the "SSD head" 1406 performs object detection. The entire network is trained end-to-end (rather than individually trained piece by piece). The neural network architecture 1400 is referred to herein as a "polar-to-Cartesian" network. It should be noted that blocks / feature maps labeled as "256x64", "128x32", "64x32", etc. are feature maps with the labeled resolution.

[0090] In practice, a full Cartesian map is not required. For highway automotive applications, for example, preferably only 250 meters ahead are visible, and only 25 meters to the left and right are visible. Therefore, clipping is performed as part of the polar to Cartesian transformation, thereby reducing computation time and memory usage.

[0091] With the above Fig.14 There are multiple advantages to the end-to-end trainable neural network architecture 1400 with polar coordinate input space and Cartesian coordinate output space described in. The first advantage is higher accuracy and more stable results. For example, the measure of accuracy mAP can be increased by 20% (relatively) by switching from a Cartesian coordinate to Cartesian coordinate network to a polar coordinate to Cartesian coordinate network (e.g., neural network architecture 1400). In a qualitative sense, the results of the polar coordinate to Cartesian coordinate network are better than both the polar coordinate to polar coordinate network and the Cartesian coordinate to Cartesian coordinate network. For example, there is less flicker between frames and fewer missed detections.

[0092] The second advantage is the reduced computation time compared to a Cartesian-to-Cartesian network, since the input can be kept in (memory efficient) polar space for most processing. In experiments, the Polar-to-Cartesian network was 2.3 times faster than the Cartesian-to-Cartesian network.

[0093] Another advantage is that no post-processing is required to obtain the object in Cartesian coordinate space.

[0094] Another advantage is that because the output of the polar to Cartesian network (e.g., neural network architecture 1400) is in Cartesian coordinates, the output has the advantage of being translationally equivariant compared to being in polar coordinates (where the object is deformed depending on the location at which the object is detected).

[0095] Yet another advantage is that only a single SSD head is needed because all vehicles are of similar size. In polar coordinate space, neighboring objects are detected as being very wide, requiring object detection at multiple feature levels.

[0096] Fig.15 An exemplary method 1500 for analyzing radar signals using deep learning according to aspects of the present disclosure is shown. In one aspect, the method 1500 can be performed by an onboard computer of a host vehicle, such as the OBC 200 of the vehicle 100.

[0097] At 1510, the OBC receives a plurality of radar frames (e.g., radar frame 420) from a radar sensor (e.g., radar 214) of the host vehicle. In one aspect, operation 1510 may be performed by system interface(s) 210, data bus 208, memory 204, and / or processor(s) 206 (any or all of which may be considered means for performing such operations).

[0098] At 1520, the OBC performs a neural network on at least a subset of the plurality of radar frames (eg, key frames 702). In one aspect, operation 1520 may be performed by processor(s) 206 (which may be viewed as a means for performing such operation).

[0099] At 1530, the OBC detects one or more objects (e.g., vehicles) in the subset of the plurality of radar frames based on executing a neural network on the subset of the plurality of radar frames. In one aspect, operation 1530 may be performed by processor(s) 206 (which may be viewed as a means for performing such operation).

[0100] Fig.16 An exemplary method 1600 for transforming polar coordinates to Cartesian coordinates in a neural network is shown. In one aspect, the method 1600 may be performed by an onboard computer of a host vehicle, such as the OBC 200 of the vehicle 100.

[0101] At 1610, the OBC receives a plurality of radar frames (e.g., radar frame 420) or one or more potential representations of a plurality of radar frames (radar frames that have been processed by the neural network) in polar coordinate space. In one aspect, operation 1610 may be performed by system interface(s) 210, data bus 208, memory 204, and / or processor(s) 206 (any or all of which may be considered means for performing such operations).

[0102] At 1620, the OBC transforms the plurality of radar frames or one or more potential representations of the plurality of radar frames from the polar coordinate space to the Cartesian coordinate space. In one aspect, operation 1620 may be performed by processor(s) 206 (which may be viewed as a means for performing such an operation).

[0103] At 1630, the OBC outputs a plurality of radar frames or one or more potential representations of the plurality of radar frames in a Cartesian coordinate space. In one aspect, operation 1630 may be performed by system interface(s) 210, data bus 208, memory 204, and / or processor(s) 206 (any or all of which may be considered as means for performing such operations).

[0104] It should be noted that although specific frequencies, integrated circuits (ICs), hardware, and other features are described in various aspects herein, alternative aspects are also variable. That is, alternative aspects may utilize additional or alternative frequencies (e.g., in addition to the 60 GHz and / or 28 GHz frequency bands), antenna elements (e.g., arrays of antenna elements of different sizes / shapes), scanning cycles (including both static scanning cycles and dynamic scanning cycles), electronic devices (e.g., mobile phones, tablet computers, personal computers (PCs), etc.), and / or other features. One of ordinary skill in the art will appreciate such variations.

[0105] It should be understood that any reference to an element using names such as "first", "second", etc. herein does not usually limit the quantity or order of those elements. More specifically, these names can be used as a convenient method to distinguish two or more elements or element instances in this article. Therefore, the reference to the first and second elements does not mean that only two elements can be adopted here, or the first element must be before the second element in some way. In addition, unless otherwise specified, a group of elements may include one or more elements. In addition, the term "at least one of A, B or C" or "one or more of A, B or C" or "at least one of the group consisting of A, B and C" used in the specification or claims indicates "A or B or C or any combination of these elements". For example, this term can include A, or B, or C, or A and B, or A and C, or A and B and C, or 2A, or 2B, or 2C, etc.

[0106] In view of the above description and explanation, it will be appreciated by those skilled in the art that the various illustrative logic blocks, modules, circuits and algorithmic steps described in conjunction with the aspects disclosed herein can be implemented into electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, frames, modules, circuits and steps have been generally described above in terms of their functions. Whether this functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the entire system. The technician can implement the described functionality in different ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the scope of the present disclosure.

[0107] Thus, it should be understood that, for example, as taught herein, a device or any component of a device may be configured (or operable or adapted) to provide functionality. This may be achieved, for example, by manufacturing (e.g., making) a device or component so that it will provide functionality; programming a device or component so that it will provide functionality; or using some other suitable implementation technique. As an example, an integrated circuit may be fabricated to provide the necessary functionality. As another example, an integrated circuit may be fabricated to support the necessary functionality and then configured (e.g., via programming) to provide the necessary functionality. As yet another example, a processor circuit may execute code to provide the necessary functionality.

[0108] In addition, the steps of the methods, sequences and / or algorithms described in conjunction with the aspects disclosed herein may be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, a storage medium (e.g., a cache memory) may be integrated with the processor.

[0109]

[0013] It should also be appreciated, therefore, that certain aspects of the present disclosure may include, for example, a computer-readable medium embodying a method for personalizing facial authentication using radar images.

[0110] Although the foregoing disclosure shows various illustrative aspects, it should be noted that various changes and modifications may be made to the examples shown without departing from the scope defined by the appended claims. The present disclosure is not intended to be limited solely to the specifically shown examples. For example, unless otherwise stated, the functions, steps, and / or actions of the method claims according to the aspects of the present disclosure described herein do not need to be performed in any particular order. In addition, although certain aspects may be described or claimed in the singular, plural forms are contemplated unless limitation to the singular is explicitly stated.

Claims

1. A method of using deep learning to analyze radar signals performed by an onboard computer of a host vehicle, comprising: receiving a plurality of radar frames from a radar sensor of the host vehicle; executing a neural network on at least a subset of the plurality of radar frames, wherein light detection and ranging (LiDAR) data is concatenated with raw radar data into a single input tensor prior to executing the neural network on the subset of the plurality of radar frames; as well as One or more objects are detected in the subset of the plurality of radar frames based on executing the neural network on the subset of the plurality of radar frames.

2. The method of claim 1, wherein the neural network comprises a recurrent neural network.

3. The method of claim 2, wherein detecting the one or more objects based on executing the recurrent neural network is based on applying a single convolutional long short-term memory (LSTM) module to a feature map generated during executing the recurrent neural network.

4. The method of claim 1 , wherein executing the neural network comprises applying local spatial convolutions to apply different convolution weights to different locations in the subset of the plurality of radar frames.

5. The method of claim 1, wherein executing the neural network comprises generating a shallow neural network with all weights for different spatially variable kernels.

6. The method of claim 1, wherein executing the neural network comprises using an explicit transformation on each weight of the neural network.

7. The method of claim 1, wherein executing the neural network comprises suppressing overlapping bounding boxes using non-maximum suppression.

8. The method according to claim 7, further comprising: An uncertainty in detecting the one or more objects is estimated based on the overlapping bounding boxes.

9. The method of claim 1, wherein the subset of the plurality of radar frames comprises key frames of the plurality of radar frames, and wherein radar frames between the key frames of the plurality of radar frames comprise predicted frames (p frames).

10. The method of claim 9, wherein the anomaly detection neural network identifies when to reset the boundaries of a keyframe based on an initial detection of an object in a p-frame of the plurality of radar frames.

11. The method of claim 1 , wherein executing the neural network comprises: using a separate feature extraction layer of the neural network for each of a set of segments including a range-azimuth segment, a range-Doppler segment, and an azimuth-Doppler segment; as well as The processed set of segments is fused into a three-dimensional segment including parameters of range, azimuth and Doppler.

12. The method of claim 1, wherein executing the neural network comprises: performing feature extraction on each radar frame in the subset of the plurality of radar frames to generate a feature map for each radar frame in the subset of the plurality of radar frames, wherein the feature map is an ego coordinate; obtaining an ego-motion between each radar frame in the subset of the plurality of radar frames; transforming the ego coordinates of each radar frame in the subset of the plurality of radar frames into world coordinates; as well as The feature maps of the subset of the plurality of radar frames are combined.

13. The method of claim 1, wherein the neural network is trained using annotations based on the subset of the plurality of radar frames having light detection and ranging (LiDAR) data and / or camera data.

14. The method according to claim 1, further comprising: The plurality of radar frames are normalized prior to executing the neural network on the subset of the plurality of radar frames.

15. The method of claim 14, wherein the standardization comprises: calculating a mean and a standard deviation for each of the plurality of radar frames at each distance of the plurality of radar frames; fitting a function to the results of said calculation; as well as The function is used to normalize the plurality of radar frames along a range dimension.

16. The method of claim 1, wherein radar data is processed by a first branch of the neural network, LiDAR data is processed by a second branch of the neural network, features of the two branches are transformed to the same coordinate space and concatenated, and the one or more objects are detected based on the features of the two branches.

17. A vehicle-mounted computer of a host vehicle, comprising: at least one processor, the at least one processor being configured to: receiving a plurality of radar frames from a radar sensor of the host vehicle; executing a neural network on at least a subset of the plurality of radar frames, wherein light detection and ranging (LiDAR) data is concatenated with raw radar data into a single input tensor prior to executing the neural network on the subset of the plurality of radar frames; as well as One or more objects are detected in the subset of the plurality of radar frames based on executing the neural network on the subset of the plurality of radar frames.

18. The onboard computer of claim 17, wherein the subset of the plurality of radar frames comprises key frames of the plurality of radar frames, and wherein radar frames between the key frames of the plurality of radar frames comprise predicted frames (p frames).

19. The onboard computer of claim 18, wherein the anomaly detection neural network identifies when to reset a boundary of a keyframe based on an initial detection of an object in a p-frame of the plurality of radar frames.

20. The onboard computer of claim 17, wherein the at least one processor is configured to execute the neural network comprises the at least one processor being configured to: using a separate feature extraction layer of the neural network for each of a set of segments including a range-azimuth segment, a range-Doppler segment, and an azimuth-Doppler segment; and The processed set of segments is fused into a three-dimensional segment including parameters of range, azimuth and Doppler.

21. The onboard computer of claim 17, wherein the at least one processor is configured to execute the neural network comprises the at least one processor being configured to: performing feature extraction on each radar frame in the subset of the plurality of radar frames to generate a feature map for each radar frame in the subset of the plurality of radar frames, wherein the feature map is an ego coordinate; obtaining an ego-motion between each radar frame in the subset of the plurality of radar frames; transforming the ego coordinates of each radar frame in the subset of the plurality of radar frames into world coordinates; as well as The feature maps of the subset of the plurality of radar frames are combined.

Images