Radar systems using machine learning models for stationary object detection

Abstract

This document describes the techniques and systems associated with a radar system using a machine learning model for stationary object detection. The radar system includes a processor capable of receiving radar data in the form of time-series frames associated with electromagnetic (EM) energy. The processor uses the radar data to generate a range-time map of the EM energy, which is then input to the machine learning model. The machine learning model is capable of receiving features corresponding to stationary objects extracted from the range-time map of multiple range intervals at each time-series frame as input. In this manner, the described radar system and techniques are able to accurately detect stationary objects of various sizes and extract key features corresponding to the stationary objects.

Description

Background Technology

[0001] Many vehicles use radar systems to detect stationary objects. Some radar systems use a point cloud representation of radar data to detect stationary objects. Point cloud data is often combined with data from other sensors (e.g., cameras or lidar systems) to detect stationary objects more accurately. Such systems can detect relatively large stationary objects (e.g., parked vehicles) while driving along a road, but struggle to detect smaller stationary objects with sufficient accuracy and speed, relying on this for autonomous or semi-autonomous control. Even when detected, these systems may fail to identify the necessary characteristics of approaching stationary objects (e.g., distance, angle, geometric features, intensity-based features, distinction from guardrails), potentially leading to unstable or unintended vehicle behavior. Summary of the Invention

[0002] This document describes the techniques and systems associated with a radar system using a machine learning model for stationary object detection. The radar system includes a processor capable of receiving radar data in the form of time-series frames associated with electromagnetic (EM) energy. The processor uses the radar data to generate a range-time map of the EM energy, which is then input to the machine learning model. The machine learning model is capable of receiving features corresponding to stationary objects extracted from the range-time map of multiple range intervals at each time-series frame as input. In this manner, the described radar system and techniques are able to accurately detect stationary objects of various sizes and extract key features corresponding to the stationary objects.

[0003] This document also describes the methods performed by the technologies and components summarized above, as well as other configurations of the radar system described herein, and the apparatus for performing these methods.

[0004] This invention provides a simplified concept related to radar systems using machine learning models for stationary object detection, which is further described in the detailed description and accompanying drawings. This invention is not intended to identify essential features of the claimed subject matter, nor is it intended to define the scope of the claimed subject matter. Attached Figure Description

[0005] The following figures illustrate in detail one or more aspects of a radar system using a machine learning model for stationary object detection. Throughout the figures, the same numbers are generally used to refer to similar features and components:

[0006] Figure 1 An example environment is shown, in which a radar system is capable of using a machine learning model for stationary object detection, according to the technology of this disclosure.

[0007] Figure 2An example configuration of a vehicle with a radar system using a machine learning model for stationary object detection is shown;

[0008] Figure 3 An example conceptual diagram of a radar system using a machine learning model for stationary object detection is shown;

[0009] Figure 4A and Figure 4B Example distance-time maps for stationary object detection are shown, generated from a distance-azimuth map and an interpolated distance-azimuth map, respectively.

[0010] Figure 5 An example concept diagram is shown for training a machine learning model for static object detection; and

[0011] Figure 6 A flowchart illustrating an example method of a radar system using a machine learning model for stationary object detection, based on the technology of this disclosure. Detailed Implementation

[0012] Overview

[0013] Radar systems can be configured as a critical sensing technology, relied upon by vehicle-based systems to acquire information about the surrounding environment. For example, vehicle-based systems can use radar systems to detect stationary objects in or near a road and take necessary actions (e.g., reduce speed, change lanes) to avoid collisions. Radar systems typically use point cloud representations of radar data (e.g., detecting horizontal data) to detect such objects. On moving vehicles, such systems are generally able to detect relatively large stationary objects, such as parked vehicles; however, when the vehicle is not stationary and is traveling at high speed, the system often fails to detect smaller stationary objects. If the primary vehicle is traveling at a non-uniform speed, these radar systems may also struggle to distinguish between stationary and moving objects.

[0014] Radar systems typically process radar data in the form of a series of frames collected at equal time intervals. To preprocess, label, and extract features from the radar data, these systems use the speed of the primary vehicle to track and correlate stationary objects. Due to variations in vehicle speed, the vehicle's displacement differs in each frame. These variations in vehicle speed can make it difficult for these radar systems to accurately detect and label stationary objects.

[0015] To improve the accuracy and speed of radar detection of small stationary objects, this document describes techniques and systems for using machine learning models based on low-level radar data for stationary object detection in radar systems. Low-level radar data (e.g., range-Doppler maps, range-azimuth maps, Doppler-azimuth maps) provides more information than point cloud representations. By using low-level radar data, the described machine learning models are able to accurately detect stationary objects of various sizes and identify them more quickly.

[0016] The described technology and system also enable interpolation of low-level radar data so that each frame is normalized based on vehicle speed. In this way, the potential detection of stationary objects is represented as a 45-degree straight line using the interpolated distance-time map. The interpolated distance-time map simplifies stationary object detection and improves its accuracy and confidence.

[0017] This is just one example of the techniques and systems used in a radar system employing a machine learning model for stationary object detection. Other examples and implementations are described in this document.

[0018] Operating environment

[0019] Figure 1 An example environment 100 is shown, in which a radar system uses a machine learning model for stationary object detection, according to the technology of this disclosure. In the depicted environment 100, a radar system 104 is mounted to or integrated into a vehicle 102 traveling on a road 106. Within a field of view 108, the radar system 104 is capable of detecting one or more stationary objects 110 near the vehicle 102.

[0020] Although shown as a truck, vehicle 102 can represent other types of motorized vehicles (e.g., cars, motorcycles, buses, tractors, semi-trailers), non-motorized vehicles (e.g., bicycles), rail vehicles (e.g., trains), water vehicles (e.g., boats), aircraft (e.g., airplanes), or spacecraft (e.g., satellites). Typically, manufacturers can mount radar system 104 onto any mobile platform, including mobile machinery or robotic equipment.

[0021] In the depicted implementation, radar system 104 is mounted on the front of vehicle 102 and illuminates stationary object 110. Radar system 104 is capable of detecting stationary object 110 from any external surface of vehicle 102. For example, vehicle manufacturers may integrate, mount, or attach radar system 104 to the front, bumper, side mirrors, headlights, taillights, or any other internal or external location where stationary object 110 needs to be detected. In some cases, vehicle 102 includes multiple radar systems 104, such as a first radar system 104 and a second radar system 104 providing a larger field of view 108. Generally, vehicle manufacturers may design the location of one or more radar systems 104 to provide a specific field of view 108 that includes a region of interest. Example fields of view 108 include 360-degree fields of view, one or more 180-degree fields of view, one or more 90-degree fields of view, etc., which may overlap to form a field of view 108 of a specific size.

[0022] The stationary object 110 comprises one or more materials that reflect radar signals. Depending on the application, the stationary object 110 may represent a target of interest. For example, the stationary object 110 may be a parked vehicle, a roadside sign, a road obstacle, or debris on road 106.

[0023] Radar system 104 emits EM radiation by transmitting electromagnetic (EM) signals or waveforms via antenna elements. In environment 100, radar system 104 is capable of detecting and tracking stationary object 110 by transmitting and receiving one or more radar signals. For example, radar system 104 is capable of transmitting EM signals between 100 and 400 GHz, between 4 and 100 GHz, or between approximately 70 and 80 GHz.

[0024] Radar system 104 may include a transmitter 112, which includes at least one antenna to transmit EM signals. Radar system 104 may also include a receiver 114, which includes at least one antenna to receive a reflected version of the EM signal. Transmitter 112 includes one or more components for transmitting EM signals. Receiver 114 includes one or more components for detecting the reflected EM signal. Manufacturers may integrate transmitter 112 and receiver 114 together on the same integrated circuit (e.g., configured as a transceiver) or integrate them separately on different integrated circuits.

[0025] The radar system 104 also includes one or more processors 116 (e.g., energy processing units) and a computer-readable storage medium (CRM) 118. The processor 116 may be a microprocessor or a system-on-a-chip. The processor 116 is capable of executing computer-executable instructions stored in the CRM 118. For example, the processor 116 is capable of processing EM energy received by the receiver 114 and using the stationary object detection module 120 to determine the position of a stationary object 110 relative to the radar system 104. The stationary object detection module 120 is also capable of detecting various characteristics of the stationary object 110 (e.g., range, target angle, velocity).

[0026] Processor 116 is also capable of generating radar data for at least one vehicle system. For example, processor 116 can control the autonomous or semi-autonomous driving system of vehicle 102 based on processed EM energy from receiver 114. For example, the autonomous driving system can control the operation of vehicle 102 to maneuver around stationary object 110, or decelerate or stop to avoid a collision with stationary object 110. As another example, the semi-autonomous driving system can warn the operator of vehicle 102 that stationary object 110 is in road 106.

[0027] The stationary object detection module 120 receives radar data, such as raw or time-series frames associated with EM energy received by receiver 114, and determines whether a stationary object 110 is in road 106 and various features associated with the stationary object 110. The stationary object detection module 120 can use machine learning model 122 to assist in the described operation and functionality. The radar system 104 can implement the stationary object detection module 120 and machine learning module 122 as computer-executable instructions, hardware, software, or a combination thereof executed by processor 116 in CRM 118.

[0028] Machine learning module 122 is capable of performing still object detection for still object 110 using the input distance-time map and extracted features. Machine learning model 122 is capable of using a neural network (e.g., a Long Short-Term Memory (LSTM) model) to detect still object 110. Machine learning model 122 is trained to receive the distance-time map and extracted features to perform still object detection. The output of machine learning model 122 may include an identifier for still object 110.

[0029] Machine learning model 122 can be or includes one or more machine learning models of various types. In some implementations, machine learning model 122 is capable of performing classification, clustering, tracing, and / or other tasks. For classification, supervised learning techniques can be used to train machine learning model 122. For example, stationary object detection module 120 can train machine learning model 122 using training data (e.g., real data) including distance-time maps and extracted features corresponding to stationary objects, where example detected objects are labeled as stationary (or non-stationary). Labels can be applied manually by engineers or provided by other techniques (e.g., based on data from other sensor systems). The training dataset can include distance-time maps similar to those input to machine learning model 122 during operation of vehicle 102.

[0030] Machine learning model 122 can be trained offline, for example, in a training computing system, and then provided for storage and implementation on one or more computing devices. For example, the training computing system may include a model trainer. The training computing system may be included in the computing device implementing machine learning model 122 or separate from the computing device. (Reference) Figure 5 The training of machine learning model 122 is described in more detail.

[0031] In some implementations, the machine learning model 122 may be one or more artificial neural networks, or may include one or more artificial neural networks. A neural network may include a set of connected nodes organized into one or more layers. A neural network comprising multiple layers may be referred to as a deep network. A deep network may include an input layer, an output layer, and one or more hidden layers located between the input and output layers. The nodes of a neural network may be connected or not fully connected.

[0032] In other implementations, the machine learning model 122 may be one or more recurrent neural networks or may include one or more recurrent neural networks. In some instances, at least some nodes of the recurrent neural network are capable of forming loops. Recurrent neural networks (e.g., LSTM networks with multiple layers) are particularly useful for processing inherently continuous input data (e.g., a series of frames in radar data). Specifically, recurrent neural networks are able to pass or retain information from a previous portion (e.g., the initial frame) of the input data sequence to a subsequent portion (e.g., later frames) of the input data sequence by using recursive or directed recurrent node connections.

[0033] Compared to radar data using the Compressed Data Cube (CDC) format, by using low-level radar data, radar system 104 and stationary object detection module 120 are able to extract more information about the EM energy distribution in the range, Doppler, azimuth, and elevation dimensions. In addition to the range, angle, and Doppler characteristics associated with stationary object 110, low-level radar data allows stationary object detection module 120 to capture intensity-based or geometric features of stationary object 110.

[0034] For example, Figure 1 A vehicle 102 is shown traveling on road 106. A radar system 104 detects a stationary object 110. The radar system 104 is also capable of tracking the stationary object 110 and extracting features associated with it. As described above, the vehicle 102 may also include at least one vehicle system, such as a driver assistance system, autonomous driving system, or semi-autonomous driving system, that relies on data from the radar system 104. The radar system 104 may include an interface that interfaces with the vehicle system that relies on the data. For example, a processor 116 outputs a signal based on EM energy received by a receiver 114 via this interface.

[0035] Typically, automotive systems use radar data provided by radar system 104 to perform functions. For example, driver assistance systems can provide blind spot monitoring and generate warnings indicating a potential collision with stationary object 110. Radar data can also indicate when changing lanes is safe or unsafe. Autonomous driving systems can move vehicle 102 to a specific location on road 106 while avoiding collisions with stationary object 110. Radar data provided by radar system 104 can also provide information about the distance to and position of stationary object 110, enabling the autonomous driving system to perform emergency braking, lane changes, or adjust the speed of vehicle 102.

[0036] Figure 2 An example configuration of a vehicle 102 with a radar system 104 is shown, which is capable of using a machine learning model 122 for stationary object detection. (See reference...) Figure 1 As described, vehicle 102 may include radar system 104, processor 116, CRM 118, stationary object detection module 120, and machine learning module 122. Vehicle 102 may also include one or more communication devices 202 and one or more vehicle-based systems 210.

[0037] The communication device 202 may include a sensor interface and a vehicle-based system interface. For example, when a separate component of the stationary object detection module 120 is integrated within the vehicle 102, the sensor interface and the vehicle-based system interface can transmit data on the communication bus of the vehicle 102.

[0038] The stationary object detection module 120 may include an input processing module 204 with a feature extraction module 206. The stationary object detection module 120 may also include a post-processing module 208 with a machine learning module 122. The input processing module 204 is capable of receiving radar data as input from the receiver 114. Generally, the radar data is received in the form of low-level time-series data. In contrast, some radar systems process radar data from the receiver 114 in the form of cloud-like points. The stationary object detection module 120 is able to process low-level time-series data to provide better detection resolution and extract features associated with the stationary object 110.

[0039] The input processing module 204 processes radar data to generate interpolated range-azimuth maps, including interpolated range-azimuth maps and / or interpolated range-elevation maps. By setting the distance based on the speed of vehicle 102, the interpolated range-azimuth map represents the potential detection of stationary objects as a straight line with a 45-degree angle. The interpolated range-azimuth format improves the accuracy of the stationary object detection module 120 by simplifying the labeling of stationary objects 110 by the machine learning model 122.

[0040] The stationary object detection module 120 or the feature extraction module 206 can process the interpolated range-azimuth map to generate a range-time map of the radar data. The feature extraction module 206 can input the range-time map into the machine learning model 122 to make predictions about stationary objects. The feature extraction module 206 can also perform additional processing to extract features associated with the stationary object 110.

[0041] The post-processing module 208 can perform additional processing on the prediction of stationary objects to remove noise. Then, the post-processor module 208 can provide the stationary object detection to the vehicle-based system 210.

[0042] Vehicle 102 also includes vehicle-based systems 210, such as driver assistance system 212 and autonomous driving system 214, which rely on data from stationary object detection module 120 to control the operation of vehicle 102 (e.g., braking, lane changing). Generally, vehicle-based systems 210 are capable of using data provided by stationary object detection module 120 to control the operation of vehicle 102 and perform specific functions. For example, driver assistance system 212 can warn the driver of stationary object 110 and perform evasive maneuvers to avoid a collision with stationary object 110. As another example, autonomous driving system 214 can navigate vehicle 102 to a specific destination to avoid a collision with stationary object 110.

[0043] Figure 3 An example conceptual diagram 300 of a radar system 104 is shown, which uses a machine learning model 122 for stationary object detection. (See reference...) Figure 2 As described, the vehicle 102 may include an input processing module 204, a feature extraction module 206, and a post-processing module 208 for the stationary object detection module 120. Conceptual diagram 300 illustrates example inputs, outputs, and operations of the stationary object detection module 120; however, the stationary object detection module 120 is not necessarily limited to the order or combination of inputs, outputs, and operations shown herein. Furthermore, any one or more operations may be repeated, combined, or recombined to provide additional functionality.

[0044] Radar system 104 provides time-series frames of EM energy as radar data 302 to input processing module 204. Radar data 302 is low-level radar data, which can include more information than point cloud data used by some radar systems for stationary object detection. Because stationary object detection module 120 uses radar data 302, it does not require additional input data from other sensors (e.g., cameras or lidar systems) to detect stationary objects. Radar data 302 includes information associated with stationary object 110 in multiple dimensions, including range space, Doppler space, elevation space, and azimuth space. Radar data 302 may include a beam vector that includes all range and Doppler intervals. In some implementations, stationary object detection module 120 may use only the amplitude information of radar data 302 without using phase information. In this way, stationary object detection module 120 can assume that a non-zero yaw rate is not applicable to detecting stationary object 110.

[0045] At operation 304, input processing module 204 performs initial processing on radar data 302. Input processing module 204 generates a stationary Doppler beam vector from radar data 302. The beam vector may include all range and Doppler intervals, and includes intensity data associated with nearby objects (e.g., both stationary and moving objects) that are fused together. Input processing module 204 performs additional processing on the beam vector to separate the intensity data associated with various detected objects.

[0046] At operation 306, input processing module 204 performs super-resolution on the stationary Doppler beam vector across the azimuth plane. In other implementations, the super-resolution operation can be performed across different planes (e.g., the elevation plane). The super-resolution operation may include Fourier transform and iterative adaptive methods (IAA). Super-resolution is applied to generate azimuth and / or elevation data in each range interval by folding the Doppler dimensions. For stationary objects, stationary object detection module 120 focuses on the range-azimuth spectrum, and input processing module 204 generates range angle maps for each time frame of the radar data, including range-azimuth and / or range-elevation maps.

[0047] Considering that the stationary object detection module 120 processes a series of range-azimuth maps collected by the radar system 104 at different times (e.g., consecutive time frames), the object detection algorithm typically works best when the main vehicle (e.g., vehicle 102) and each stationary object (e.g., stationary object 110) move a constant distance (e.g., distance) relative to each other between data captures. Although the time intervals between data captures are usually equal, the distance intervals between the main vehicle and the stationary objects are not always equal because the vehicle's speed may vary between data captures.

[0048] At operation 308, input processing module 204 performs distance sampling and interpolation on the range azimuth map. Input processing module 204 is capable of creating a set of appearances for each distance interval traversed by the movement of vehicle 102 by interpolating actual data captures or time frames. For a given distance interval, input processing module 204 is capable of generating an interpolated time frame by determining the time at which the vehicle arrives at that distance interval (e.g., "distance interval time"). Input processing module 204 then selects a data capture with a time frame equivalent to the distance interval time (e.g., "equivalent data capture"). The distance position of vehicle 102 at the equivalent data capture is determined. Input processing module 204 then shifts the equivalent data capture in the distance dimension so that the positions of vehicle 102 and stationary object 110 match those positions in the interpolated time frames to be generated. The intensity in the two shifted appearances can be combined using a weighted average based on the relative differences between the equivalent data capture and the interpolated time frames. Alternatively, two equivalent filtered intensity values ​​from the data capture (e.g., minimum intensity value, average intensity value, or maximum intensity value) can be used to create an interpolated time frame. This latter approach can suppress transient signals. The input processing module 204 can combine the interpolated time frames to generate an interpolated range-azimuth map 310 of the radar data.

[0049] Before performing the interpolation operation, the input processing module 204 can also perform distance downsampling. The range azimuth map can be downsampled over distance by a given factor. For example, the input processing module 204 can perform distance downsampling by taking filtered intensity values ​​(e.g., maximum intensity, average intensity, or minimum intensity) at each azimuth point (or elevation point in different implementations) over N consecutive range intervals, where N is a positive integer. In this way, the input processing module 204 effectively compresses the range azimuth data in the range dimension by a factor of N. The range interval size and other distance-dependent variables are multiplied by the factor N. The increased range interval size causes the input processing module 204 to produce interpolated time frames reduced by the factor N. Combined with the distance compression by the factor N, the interpolated time frames reduced by the factor N can result in a reduction in the runtime of subsequent processing by approximately N squared (N... 2 In this way, even when using radar data 302 or time-series frames of received EM energy as input, the operation of the stationary object detection module 120 can be performed in real time or near real time. Similarly, the machine learning module 122 can process radar data faster and / or has a smaller size.

[0050] At operation 312, input processing module 204 performs image signal processing on the interpolated range-azimuth map 310 to generate a range-time map 314. Input processing module 204 stacks the vectors generated from the interpolated range-azimuth map 310 to generate the range-time map 314. Since the stationary object 110 moves one distance interval for each interpolated time frame, the interpolation makes the latent detection of the stationary object 110 appear as a straight line with a 45-degree angle in the range-time map 314. Depicting the stationary object 110 as a 45-degree straight line simplifies and improves the labeling accuracy of the machine learning model 122. Interpolation also ensures that the stationary object 110 exists in every distance interval and time frame in the range-time map 314, which also simplifies the labeling of the stationary object 110 by the machine learning model 122. The interpolation operation performed by input processing module 204 also results in the detection of the stationary object being independent of the speed of the vehicle 102. Example range-time maps generated from the range-azimuth map and the interpolated range-azimuth map are referenced respectively. Figure 4A and Figure 4B The description is as follows.

[0051] The input processing module 204 processes the distance-time map to identify stationary objects 110 from the dimensions of azimuth, elevation, and rate of change of distance, and collapses the data cube along these dimensions. This data cube results in an energy distribution vector for each potential stationary object 110 along the distance dimension.

[0052] At operation 316, feature extraction module 206 uses distance-time graph 314 to extract features of stationary object 110 and label stationary object 110. (See reference...) Figure 5 In more detail, similar operations can be performed to train the machine learning model 122. Since it may be difficult to distinguish a stationary object from the radar reflections of other received objects (e.g., guardrails, moving vehicles), the feature extraction module 206 performs several steps to label and identify features associated with the stationary object 110. The distance-time map 314 is processed using a sliding window to generate a window-based distance-time map. For example, the window may include the desired Doppler interval at each distance interval. The window-based distance-time map is then fed into a pre-trained encoder-based feature extractor.

[0053] Then, the machine learning model 122 is able to generate a predicted detection 318 for the stationary object 110. The machine learning model 122 is able to use an acyclic graph model to assist in the detection of stationary objects. The acyclic graph model identifies and preserves the trace of each stationary object 110. Therefore, for a given value in a time series signal, the time frame closer to it will have a more significant impact than the time frame further away.

[0054] Machine learning model 122 may include a convolutional neural network. Machine learning model 122 may also use a pre-trained deep learning model within the convolutional neural network to extract features and feed the extracted features into a deep sequence model (e.g., a long short-term memory (LSTM) network with multiple layers), which is trained to detect the presence of stationary objects and generate detection probabilities 318. Detection probabilities 318 provide a distance-temporal probability map, which includes predictions across each distance interval and each time frame.

[0055] At operation 320, post-processing module 208 performs additional processing to smooth the detection probability 318 by applying a smoothing function and generating detection 322 (e.g., removing noise). Detection 322 may include the detection probability at each distance interval of each time frame of stationary object 110. The smoothing function that generates the final output probability of detection 322 at a specific distance is given by equation (1):

[0056]

[0057] Where p(t-1) represents the predicted probability of machine learning model 122 at each distance for a set of time frames, including the current time frame; p(t) represents the output of post-processing operation 320, which is the smoothed pseudo-probability of the object detected at the input distance; and α and β represent parameters controlling the degree of smoothing. For example, post-processing module 208 can set α to equal 1 and β to equal This time frame represents past intensity values ​​of the input feature set used to form the machine learning model 122. The post-processing module 208 is capable of applying post-processing to each frame to provide a smoothed probability value for the detection 322. This probability value represents the confidence level of the detection 322.

[0058] Figure 4A and Figure 4B Example distance-time plots 400 and 410 of the detection 402 of stationary object 110 are shown, generated from a distance-azimuth plot and an interpolated distance-azimuth plot, respectively. Plots 400 and 410 provide distance 404 as the y-axis to indicate the distance between vehicle 102 and stationary object 110.

[0059] Distance-time plot 400 provides time frame 406 as the x-axis to indicate appearance or data capture. In contrast, interpolated distance-time plot 410 provides interpolated time frame 412 as the x-axis to indicate the interpolated appearance. Based on a reference... Figure 3 The interpolation operation described herein results in interpolated frames 412 having no constant interval between consecutive appearances. In other words, the interpolated time frames 412 are not equidistant in time.

[0060] The distance-time graph 400 depicts the detection 402 of the stationary object 110 as a curve. As it approaches the stationary object 110, the vehicle 102 may experience speed changes, and the distance 404 to the stationary object 110 will change non-linearly over time frame 406. Furthermore, the slope of the detection 402 in the distance-time graph 400 will depend on the speed of the vehicle 102, and the change in slope will depend on the speed change of the vehicle 102. In contrast to the distance-time graph 400, the interpolation performed by the input processing module 204 results in the detection 402 of the stationary object 110 appearing as a straight line at 45 degrees in the distance-time graph 410, because the stationary object 110 moves one distance interval for each interpolated time frame 412.

[0061] Figure 5 A sample concept diagram 500 is shown for training a machine learning model 122 for still object detection. Specifically, concept diagram 500 illustrates the feature extraction and labeling process of the still object detection module 120 for training the machine learning model 122. Concept diagram 500 shows sample inputs, outputs, and operations of the still object detection module 120, but the still object detection module 120 is not necessarily limited to the order or combination of inputs, outputs, and operations shown herein. Furthermore, any one or more operations can be repeated, combined, or recombined to provide additional functionality.

[0062] At operation 502, feature extraction module 206 extracts feature vectors from distance-time graph 314. The feature vectors represent the intensity of various distances across time frames in the distance-time graph. Feature extraction module 206 uses the feature vectors to represent the content in distance-time graph 314 and the distance at which the features appear. To determine the content in distance-time graph 314, feature extraction module 206 takes intensity values ​​along a 45-degree line for specific past time frames and enhances them using distance intervals to accurately determine where the target appears. By including distance in the feature vectors, machine learning model 122 is able to detect distant stationary objects 110 even when the target intensity is weak.

[0063] Feature set 504 is taken from the 45-degree lines in the distance-time map 314. Since the potential detection of stationary objects in the distance-time map 314 moves one distance interval for each time frame and is represented by a 45-degree line, feature extraction module 206 can label each stationary object 110 and non-stationary object as "1" or "0," respectively. Each target in a set of distance-time maps 314 is obtained through a feature extraction and labeling process to prepare a training dataset (e.g., real data) for machine learning model 122.

[0064] Machine learning model 122 can be trained offline or online. In offline training (e.g., batch learning), machine learning model 122 is trained using a static training dataset. In online training, machine learning model 122 is continuously trained (or retrained) as new training data becomes available (e.g., when the machine learning model is used to perform static object detection).

[0065] Centralized training of multiple machine learning models 122 (e.g., based on a centrally stored dataset) can be performed. In other implementations, the trainer can use decentralized training techniques, including distributed training or federated learning, to train, update, or personalize the machine learning models 122.

[0066] Once training is complete, the machine learning model 122 can be deployed during the inference phase. During the inference phase, the machine learning model 122 receives multiple range intervals as input at each time frame of the range-time map 314, including all available range intervals. During the inference phase, radar data 302 is passed through the stationary object detection module 120 to generate predictions by the machine learning model 122. In the inference phase, each input feature is multiplied by trained weights to produce a probability at each range interval and each time frame.

[0067] Example Method

[0068] Figure 6 A flowchart of an example method 600 for radar system 104 using a machine learning model 122 for stationary object detection is shown. Method 600 is shown as a set of operations (or actions) performed, but is not limited to the order or combination of operations shown herein. Furthermore, any one or more operations can be repeated, combined, or recombined to provide additional functionality. References can be found in the sections discussed below. Figure 1 Environment 100 and Figures 1 to 5 The entities detailed herein are referenced for illustrative purposes only. This technique is not limited to being performed by one or more entities.

[0069] At position 602, a model is generated to process radar data and detect stationary objects. For example, a machine learning model can be generated to process range-time maps and label stationary objects. References can be used. Figures 1 to 5 The techniques and systems described are used to generate machine learning models.

[0070] At 604, the model can be trained to process radar data and detect stationary objects. For example, a machine learning model can be trained to process radar data, including range-time maps, and detect and label stationary objects 110. Furthermore, the machine learning model can be trained to extract features associated with stationary objects 110. Boxes 602 and 604 are optional operations of method 600, which can be performed by different systems or components than those in boxes 606 to 610, at different times and / or in different locations.

[0071] At 606, one or more processors of the radar system receive radar data, which includes time-series frames associated with EM energy. For example, processor 116 of radar system 104 receives radar data, which includes time-series frames associated with EM energy. EM energy can be received by receiver 114 of radar system 104. EM energy is a reference... Figure 3 The described radar data 302. Processor 116 is also capable of processing EM energy by generating one or more Doppler beam vectors of EM energy using time-series frames. Processor 116 is capable of generating a range-azimuth map of EM energy for each time-series frame using one or more Doppler beam vectors and super-resolution operations. (Reference...) Figure 3 In a more detailed description of the interpolation operation, processor 116 can then generate an interpolated range-azimuth map of the EM energy. Before performing the interpolation operation, processor 116 can also effectively compress the range-azimuth map by downsampling the range at each azimuth point across multiple consecutive range intervals, taking the maximum intensity.

[0072] At 608, one or more processors use radar data to generate a range-time map of EM energy. For example, processor 116 processes radar data 302 to generate an interpolated range-azimuth map 310. The interpolated range-azimuth map 310 can be generated after input processing module 204 performs preprocessing, super-resolution across one or more planes (e.g., the azimuth plane), range sampling, and / or interpolation on the radar data 302. Input processing module 204 can then perform additional processing to generate a range-time map 314 using the interpolated range-azimuth map 310.

[0073] At 610, one or more processors use a machine learning model to detect stationary objects. The machine learning model is configured to take into account features corresponding to the stationary object extracted from a range-time map of multiple range intervals at each time frame of the radar data. For example, processor 116 can input features corresponding to the stationary object 110 extracted from range-time map 314 into machine learning model 122 to detect the stationary object 110. Machine learning model 122 can take into account the extracted features from multiple range intervals at each time frame of the radar data. The extracted features may include the stationary object 110's range, target angle, velocity, geometric features, and intensity-based features.

[0074] Method 600 can return to box 604 to optionally retrain or update the machine learning model 122 using additional real data. For example, the implementation of machine learning model 122 can be updated based on additional or new real data obtained and processed in another computing system and / or location.

[0075] Example

[0076] Examples are provided in the following sections.

[0077] Example 1. A method comprising: receiving radar data, the radar data including time-series frames associated with electromagnetic (EM) energy; generating a distance-time map of the EM energy using the radar data; and detecting stationary objects in a road using a machine learning model, the machine learning model being configured to take into input features corresponding to the stationary objects extracted from the distance-time map of multiple distance intervals at each time-series frame.

[0078] Example 2. The method of Example 1, further comprising: receiving radar data by processing the EM energy at least in the following manner when the EM energy is received at the antenna: generating a Doppler beam vector of the EM energy using time-series frames; generating a range-angle map of the EM energy for each time-series frame using the Doppler beam vector and super-resolution operations, the angle of the range-angle map including elevation or azimuth; and generating an interpolated range-angle map of the EM energy using the range-angle map and interpolation operations.

[0079] Example 3. The method of Example 2, where: the angle is the azimuth; the distance angle map is the distance azimuth map; the interpolated distance angle map is the interpolated distance azimuth map; and the distance time map represents the potential detection of stationary objects in stationary objects as a 45-degree straight line.

[0080] Example 4. The method of Example 3, wherein the super-resolution operation may include at least one of Fourier transform or iterative adaptive method to generate azimuth data at each distance interval.

[0081] Example 5. The method of Example 3 or 4, wherein the interpolation operation includes: determining a distance interval time for each of a plurality of distance intervals within a range azimuth map, the distance interval time being the time series frame in which the vehicle arrives at the distance interval, the radar system being attached to a part of the vehicle; selecting data captures from the radar data that have time series frames equivalent to the distance interval time; determining the distance position of the vehicle at the data capture equivalent to the distance interval time; and shifting the data capture equivalent to the distance interval time in the distance dimension so that the positions of the vehicle and stationary objects match those positions in the interpolated time series frames.

[0082] Example 6. The method of Example 5, wherein shifting the data capture equivalent to the distance interval time in the distance dimension includes: using a weighted average of the data capture equivalent to the distance interval time, the weighted average being based on the relative difference between the data capture equivalent to the distance interval time and the interpolated time frame; or selecting a filtered intensity value of the data capture equivalent to the distance interval time.

[0083] Example 7. Any of the methods in Examples 3 to 6, wherein processing the EM energy received by the antenna of the radar system further comprises: downsampling the range azimuth map at the range by taking the filtered intensity at each azimuth point over a number of consecutive range intervals in order to effectively compress the range azimuth map.

[0084] Example 8. The method of Example 7, wherein processing the EM energy received by the antenna of the radar system further includes: multiplying the range interval size by the amount to effectively reduce the number of interpolated time frames.

[0085] Example 9. The method of any of the preceding examples, wherein the time series frame includes information associated with a stationary object in multiple dimensions, including at least three of the following dimensions: distance dimension, Doppler dimension, elevation angle dimension, and azimuth angle dimension.

[0086] Example 10. The method of any of the preceding examples, wherein the extracted features include at least two of the following: distance to a stationary object, target angle, velocity, geometric features, or intensity-based features.

[0087] Example 11. The method of any of the preceding examples, wherein the machine learning model includes a Long Short-Term Memory (LSTM) network with multiple layers.

[0088] Example 12. A system including a radar system comprising one or more processors configured to detect stationary objects by performing any of the methods in the preceding examples.

[0089] Example 13. The system of Example 12, wherein the radar system is configured to be integrated into or installed in a vehicle.

[0090] Example 14. The system of Example 12 or 13, where the machine learning model includes a Long Short-Term Memory (LSTM) network with multiple layers.

[0091] Example 15. A computer-readable storage medium including computer-executable instructions that, when executed, cause a processor of a radar system to perform the method of any one of Examples 1 to 11.

[0092] Conclusion

[0093] While various embodiments of the present disclosure have been described in the foregoing description and illustrated in the accompanying drawings, it should be understood that the present disclosure is not limited thereto, but can be practiced in various ways within the scope of the following claims. It will be apparent from the foregoing description that various modifications can be made without departing from the scope of the present disclosure as defined by the appended claims.

Claims

1. A system for a vehicle, the system comprising: The radar system in the vehicle includes one or more processors configured to detect stationary objects in the following manner: Receive radar data, the radar data comprising multiple time-series frames associated with electromagnetic EM energy reflected by one or more objects in the road and received by the antenna of the radar system, each time-series frame in the radar data comprising multiple range intervals; The Doppler beam vector of the EM energy is generated using the time-series frames of the radar data; The Doppler beam vector and super-resolution operation are used to generate a range azimuth map of the EM energy for each of the multiple time series frames. The interpolated range-azimuth map of the EM energy is generated using the aforementioned range-azimuth map in the following manner: For each of the multiple distance intervals in the distance azimuth map, a distance interval time is determined, wherein the distance interval time is the time series frame in which the vehicle arrives at the distance interval; Select equivalent data acquisition from the radar data that has a time series frame equivalent to the range interval time for the range interval time; Determine the distance and position of the vehicle at the equivalent data capture point; and In the distance dimension, the distance values ​​captured by the equivalent data are shifted so that the positions of the vehicles and stationary objects among the one or more objects match those positions in the interpolated time series frames; The distance-time map of the EM energy is generated using the interpolated distance-azimuth map, which represents the latent detection of the stationary object among the one or more objects as a 45-degree straight line; and The machine learning model is used to detect stationary objects among the one or more objects. The machine learning model is configured to receive features corresponding to the one or more objects extracted from a distance-time map of multiple distance intervals at each time series frame in the time series frame as input. The extracted features include at least two of the following: distance, target angle, velocity, geometric features, or intensity-based features of the one or more objects.

2. The system as described in claim 1, characterized in that, The super-resolution operation can include at least one of Fourier transform or iterative adaptive methods to generate azimuth data at each distance interval.

3. The system as described in claim 1, characterized in that, The one or more processors are configured to shift the equivalent data capture in the distance dimension in the following manner: A weighted average of the distance values ​​captured using the equivalent data, the weighted average being based on the relative difference between the time series frames captured using the equivalent data and the interpolated time series frames; or Select the filtered distance value from the distance value captured by the equivalent data.

4. The system as described in claim 1, characterized in that, The one or more processors are further configured to process the EM energy upon receipt of the EM energy by the antenna in the following manner: The range azimuth map is downsampled at a distance by taking filtered intensity at each azimuth point in a first number of consecutive distance intervals, so as to effectively compress the range azimuth map.

5. The system as described in claim 4, characterized in that, The one or more processors are further configured to process the EM energy upon receipt of the EM energy by the antenna in the following manner: The distance interval size of the distance azimuth map is amplified by the first number of factors so as to effectively reduce the amount of interpolated time frames by the first number of factors.

6. The system as described in claim 1, characterized in that, The time series frame includes information associated with the stationary object in multiple dimensions, including at least three of the following dimensions: distance, Doppler, elevation, and azimuth.

7. The system as described in claim 1, characterized in that, The extracted features include at least two of the following: distance to the stationary object, target angle, velocity, geometric features, or intensity-based features.

8. The system as described in claim 1, characterized in that: The machine learning model includes a Long Short-Term Memory (LSTM) network with multiple layers.

9. A method for a means of transport, the method comprising: Receive radar data, the radar data comprising multiple time-series frames associated with electromagnetic EM energy reflected by one or more objects in the road and received by the antenna of the radar system in the vehicle, each time-series frame in the radar data comprising multiple range intervals; The Doppler beam vector of the EM energy is generated using the time-series frames of the radar data; The Doppler beam vector and super-resolution operation are used to generate a range azimuth map of the EM energy for each of the multiple time series frames. The interpolated range-azimuth map of the EM energy is generated using the aforementioned range-azimuth map in the following manner: For each of the multiple distance intervals in the distance azimuth map, a distance interval time is determined, wherein the distance interval time is the time series frame in which the vehicle arrives at the distance interval; Select equivalent data acquisition from the radar data that has a time series frame equivalent to the range interval time for the range interval time; Determine the distance and position of the vehicle at the equivalent data capture point; and In the distance dimension, the distance values ​​captured by the equivalent data are shifted so that the positions of the vehicles and stationary objects among the one or more objects match those positions in the interpolated time series frames; The distance-time map of the EM energy is generated using the interpolated distance-azimuth map, which represents the potential detection of the stationary object among the one or more objects as a 45-degree straight line; as well as A machine learning model is used to detect stationary objects in one or more objects in a road. The machine learning model is configured to receive features corresponding to the one or more objects extracted from a distance-time map of multiple distance intervals at each time series frame as input. The extracted features include at least two of the following: distance, target angle, velocity, geometric features, or intensity-based features of the one or more objects.

10. The method as described in claim 9, characterized in that, Shifting the equivalent data capture in the distance dimension includes: A weighted average of the distance values ​​captured using the equivalent data, the weighted average being based on the relative difference between the time series frames captured using the equivalent data and the interpolated time series frames; or Select the filtered distance value from the distance value captured by the equivalent data.

11. The method as described in claim 9, characterized in that, Processing the EM energy received by the antenna of the radar system further includes: The range azimuth map is downsampled in distance by taking filtered intensity at each azimuth point in a first number of consecutive distance intervals, so as to effectively compress the range azimuth map.

12. The method as described in claim 11, characterized in that, Processing the EM energy received by the antenna of the radar system further includes: The distance interval size of the distance azimuth map is amplified by the first number of factors so as to effectively reduce the amount of interpolated time frames by the first number of factors.

13. The method as described in claim 9, characterized in that, The time series frame includes information associated with the stationary object in multiple dimensions, including at least three of the following dimensions: distance, Doppler, elevation angle, and azimuth angle.

14. A computer-readable storage medium comprising computer-executable instructions, which, when executed, cause a processor of a radar system in a vehicle to: Receive radar data, the radar data comprising multiple time-series frames associated with electromagnetic EM energy reflected by one or more objects in the road and received by the antenna of the radar system, each time-series frame in the radar data comprising multiple range intervals; The Doppler beam vector of the EM energy is generated using the time-series frames of the radar data; The Doppler beam vector and super-resolution operation are used to generate a range azimuth map of the EM energy for each of the multiple time series frames. The interpolated range-azimuth map of the EM energy is generated using the aforementioned range-azimuth map in the following manner: For each of the multiple distance intervals in the distance azimuth map, a distance interval time is determined, wherein the distance interval time is the time series frame in which the vehicle arrives at the distance interval; Select equivalent data acquisition from the radar data that has a time series frame equivalent to the range interval time for the range interval time; Determine the distance and position of the vehicle at the equivalent data capture point; and In the distance dimension, the distance values ​​captured by the equivalent data are shifted so that the positions of the vehicles and stationary objects among the one or more objects match those positions in the interpolated time series frames; The distance-time map of the EM energy is generated using the interpolated distance-azimuth map, which represents the potential detection of the stationary object among the one or more objects as a 45-degree straight line; as well as A machine learning model is used to detect stationary objects among the one or more objects. The machine learning model is configured to receive features corresponding to the one or more objects extracted from a distance-time map of multiple distance intervals at each time series frame in the time series frame as input. The extracted features include at least two of the following: distance, target angle, velocity, geometric features, or intensity-based features of the one or more objects.

Images