System and method for radar object recognition using inter-frame temporal relationships - Patents.com

Abstract

The present disclosure provides a system and method for detecting and tracking objects. The method includes permuting an order of frames in a sequence of radar image frames to generate a plurality of permuted sequences having frames that differ in dominant positions in a corresponding permuted sequence of radar image frames. Each permuted sequence of radar image frames is processed with a first neural network to generate temporally enhanced features for each of the frames in the sequence of radar image frames. Further, a feature map is reconstructed from the temporally enhanced features of each of the frames in the sequence of radar image frames to generate a sequence of feature maps. The method further includes processing a list of feature vectors from each feature map with a second neural network to generate a temporally enhanced heat map.

Description

[Technical field]

[0001] The present disclosure relates generally to radar systems, and more particularly to systems and methods for detecting and tracking moving objects with radar systems. [Background technology]

[0002] Autonomous driving utilizes sensing technologies for robust dynamic object recognition and continuously uses various perception sensors for reliable and safe vehicle decision making. Among the various perception sensors, cameras and Lidar are the two mainstream sensors used to recognize surrounding objects. Cameras provide meaning-rich visual features in traffic situations, while Lidar provides high-resolution point clouds that can capture reflections on objects.

[0003] Compared to cameras and lidars, radars are advantageous for automotive applications. Radars transmit millimeter-wavelength electromagnetic waves to estimate the distance, speed, and angle of objects. At such wavelengths, it can penetrate or diffract around particles in rain, fog, snow, and dust, providing long-range perception even under such adverse weather conditions. In contrast, the lasers sent by lidars at much shorter wavelengths can bounce off particles, which significantly reduces the operating range. Compared to cameras, radars are also resilient to lighting conditions, e.g., nighttime, solar glare, etc. Moreover, radars provide a cost-effective and reliable option to complement other sensors.

[0004] However, conventional automotive radars are not well suited for high-resolution perception during autonomous driving. Rather, automotive radars generate range-Doppler (i.e., distance-radius-velocity) spectra, which are used for safety features such as emergency braking and cruise control. High angular resolution in the azimuth and elevation domains is essential for radar-aided automotive perception.

[0005] Sub-degree azimuth resolution is becoming available in recent automotive radar datasets, but elevation resolution still lags behind. At sub-degree azimuth resolution, semantic features (e.g., corners and shape) can be observed for objects at short distances, but object profiles can still be blurred due to cross-range resolution. Furthermore, the ability to localize and identify objects of interest in the radar frame is still far from the quality and / or resolution suitable for a full level of autonomous driving. Summary of the Invention

[0006] An objective of some embodiments is to address the object recognition problem in automotive radar used for autonomous driving. Although radar is cost-effective and robust under all-weather conditions for autonomous driving perception, radar measurements suffer from low angular resolution and accuracy in recognizing surrounding objects. To improve the capabilities of automotive radar, some embodiments utilize temporal information from a sequence of radar image frames to improve the quality and / or resolution of object recognition with automotive radar. Thus, an objective of some embodiments is to exploit the consistency of object presence and features to model relationships between objects in a sequence of radar image frames. The sequence of radar image frames shows radar measurements of a scene at different successive instances of time. The scene may include one or more objects, such as vehicles.

[0007] Some embodiments are based on the realization that it is possible to extract various features from a sequence of radar image frames and place these features in context with each other to exploit the temporal correlation between the sequences of radar image frames. However, while such an approach may be practical for other modalities such as camera images and audio signals, the quality of current automotive radars presents insufficient semantic features, resulting in poor results for radar object recognition suitable for autonomous driving.

[0008] Some embodiments are based on the recognition that the nature and number of features in each radar image frame is one of the reasons for the poor quality of the temporal correlation between radar image frames of a sequence of radar image frames. For example, in contrast to camera image frames, the features of radar image frames are semantically poor quality, which makes the temporal correlation more difficult to obtain. Furthermore, the diversity and rapid changes of traffic conditions can cause relevant features in the radar image frames to change. Although such changes can be corrected based on the meaning of the relevant features, as described above, the features of the radar image frames are semantically poor quality. These properties of radar image frames, taken together, reduce the ability to train neural networks suitable for object recognition in autonomous driving using automotive radar.

[0009] Some embodiments are based on the recognition that these problems can be addressed by feature extraction with temporal correlation at the object level. Multi-directional feature extraction allows these features to be extracted together from radar image frames to present various temporal representations at the feature level. In contrast, implementing temporal correlation at the object level (as opposed to the feature level) allows for interrelated high-level features of detected objects that are not available at the feature level. Thus, temporal correlation is performed at least twice for both the feature level and the object level to improve the quality of the temporal correlation.

[0010] It is therefore an objective of some embodiments to provide a system utilizing machine learning that can incorporate temporal correlation of consecutive radar image frames at feature level and object level in a single framework. Additionally or alternatively, it is an objective of some embodiments to provide a principle of temporal correlation at the feature level that is distinct from that at the object level in order to improve the quality of the joint temporal correlation.

[0011] Some embodiments are based on the recognition that the feature-level temporal correlation can be performed by extracting features from different permutations of frames in a sequence of radar image frames based on the order of each frame after permutation. For example, the sequence of radar image frames includes a current frame and a previous frame. The order of the frames in the sequence of radar image frames is permuted to generate multiple permuted sequences of radar image frames having different frames in dominant positions in the corresponding permuted sequences of radar image frames. For example, a first permuted sequence is generated in which the current frame is in the dominant position and the previous frame is in a subordinate position. Similarly, a second permuted sequence is generated in which the previous frame is in the dominant position and the current frame is in a subordinate position. The dominant position refers to the first or last position in the sequence of radar image frames, and the subordinate position refers to a position other than the dominant position. Here, the dominant position is considered to be the first position.

[0012] Further, each of the sorted sequences of radar image frames is processed with a first neural network. The first neural network is trained to extract features of a frame at a dominant position that are temporally correlated with features of one or more frames at a subordinate position to generate a temporally enhanced feature for each of the frames in the sequence of radar image frames. For example, the first sorted sequence is processed with the first neural network to extract features of a current frame that are temporally correlated with features of a previous frame to generate a temporally enhanced feature of the current frame. Similarly, the second sorted sequence is processed with the first neural network to extract features of a previous frame that are temporally correlated with features of the current frame to generate a temporally enhanced feature of the previous frame. Thus, the sorting order in the sequence of radar image frames allows for the creation of a sequence of features that are temporally correlated but that highlight different frames in the sequence of radar images. In this way, the temporal correlation of features at the feature level can be performed in a manner that allows for subsequent correlation at the object level.

[0013] To perform object-level temporal correlation, some embodiments generate feature maps based on the temporally enhanced features of the current frame and the temporally enhanced features of the previous frame, respectively. The feature maps are then processed with a second neural network to correlate the feature maps with each other. For example, some embodiments use an attention mechanism to place the feature map generated based on the temporally enhanced features of the current frame in view of the feature map generated based on the previous frame generated from a sequence of radar image frames in a dominant position, and / or vice versa.

[0014] Accordingly, one embodiment discloses a radar system for detecting and tracking one or more objects, the radar system including a processor and a memory having stored thereon instructions that, when executed by the processor, cause the radar system to collect a sequence of radar image frames indicative of radar measurements of a scene at different successive instances in time, the scene including the one or more objects, the sequence of radar image frames including a current frame and at least one previous frame, the instructions further causing a reordering of frames in the sequence of radar image frames to generate a plurality of reordered sequences of radar image frames having different frames at dominant positions in corresponding reordered sequences of radar image frames, and extracting a feature of the frame at the dominant position that is temporally correlated with features of one or more frames at sub-positions to generate a temporally enhanced feature for each of the frames in the sequence of radar image frames. the instructions further cause the instructions to process each of the permuted sequence of radar image frames with a first neural network trained to process the first permuted sequence of radar image frames using a first neural network trained to process the first permuted sequence of radar image frames using a second neural network trained to process the first permuted sequence of radar image frames using a first neural network trained to process the first permuted sequence of radar image frames using a first neural network trained to process the first permuted sequence of radar image frames using a second neural network trained to process the first permuted sequence of radar image frames using a first neural network trained to process the first permuted sequence of radar image frames using a first neural network trained to process the first permuted sequence of radar image frames using a first neural network trained to process the first permuted sequence of radar image frames using a second neural network trained to process the first permuted sequence of radar image frames using a

[0015] Accordingly, another embodiment discloses a method for detecting and tracking one or more objects, the method comprising collecting a sequence of radar image frames illustrating radar measurements of a scene at different successive instances in time, the scene including the one or more objects, the sequence of radar image frames including a current frame and at least one previous frame, the method further comprising: reordering frames in the sequence of radar image frames to generate a plurality of reordered sequences of radar image frames having different frames in dominant positions in corresponding reordered sequences of radar image frames; and reordering frames in the sequence of radar image frames using a first neural network trained to extract features of a frame in the dominant position that are temporally correlated with features of one or more frames in a subposition to generate temporally enhanced features for each of the frames in the sequence of radar image frames. The method includes processing each permuted sequence; reconstructing a feature map from the temporally enhanced features of each of the frames in the sequence of radar image frames to generate a sequence of feature maps, the feature map being indicative of one or more properties of the one or more objects in the scene; and processing a list of feature vectors from each feature map of the sequence of feature maps using a second neural network trained to aggregate the one or more properties of objects in one feature map with one or more properties of objects in another feature map to generate a temporally enhanced heat map, each feature vector supplemented with a position encoding, the method further including processing the temporally enhanced heat map based on one or more regression heads to generate at least one property of the one or more objects.

[0016] Accordingly, yet another embodiment discloses a non-transitory computer-readable storage medium having embodied thereon a program executable by a processor for performing a method for detecting and tracking one or more objects, the method comprising: collecting a sequence of radar image frames illustrating radar measurements of a scene at different successive instances in time, the scene including the one or more objects, the sequence of radar image frames including a current frame and at least one previous frame, the method further comprising: reordering an order of frames in the sequence of radar image frames to generate a plurality of reordered sequences of radar image frames having different frames at dominant positions in corresponding reordered sequences of radar image frames; and reordering a sequence of the radar image frames using a first neural network trained to extract features of a frame at the dominant position that are temporally correlated with features of one or more frames at sub-positions to generate a temporally enhanced feature for each of the frames in the sequence of radar image frames. The method includes processing each permuted sequence; reconstructing a feature map from the temporally enhanced features of each of the frames in the sequence of radar image frames to generate a sequence of feature maps, the feature map being indicative of one or more properties of the one or more objects in the scene; and processing a list of feature vectors from each feature map of the sequence of feature maps using a second neural network trained to aggregate the one or more properties of objects in one feature map with one or more properties of objects in another feature map to generate a temporally enhanced heat map, each feature vector supplemented with a position encoding, the method further including processing the temporally enhanced heat map based on one or more regression heads to generate at least one property of the one or more objects.

[0017]

[0023] Embodiments of the present disclosure will now be further described with reference to the accompanying drawings, in which:

[0024] The drawings shown are not necessarily to scale, with emphasis instead being placed on illustrating the principles of embodiments of the present disclosure. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a block diagram illustrating a radar system for detecting and tracking one or more objects, according to one embodiment of the present disclosure. [Figure 2A] FIG. 1 is a schematic diagram illustrating a framework incorporating feature-level and object-level temporal correlation for object recognition according to one embodiment of the present disclosure. [Figure 2B] FIG. 13 is a schematic diagram illustrating an example of adding an updated feature vector back to a feature map according to an embodiment of the present disclosure. [Diagram 3] FIG. 1 illustrates a schematic for reconstructing a radar image using a weighted combination of temporally enhanced heatmaps, according to one embodiment of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram illustrating a deep convolutional neural network with skip connections, according to one embodiment of the present disclosure. [Diagram 5] FIG. 2 illustrates an algorithm for tracking multiple objects according to one embodiment of the present disclosure. [Figure 6] FIG. 2 is a block diagram illustrating a method for detecting and tracking one or more objects according to one embodiment of the present disclosure. [Figure 7] FIG. 1 is a schematic diagram illustrating a computing device that can be used to implement the systems and methods of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, devices and methods are shown only in block diagram form in order to avoid obscuring the present disclosure.

[0020] As used herein and in the claims, the words "for example," "for example," and "such as," as well as "comprising," "having," "including," and other verb forms thereof, when used in conjunction with a list of one or more components or other items, should be construed as open-ended. This means that the list should not be considered to exclude additional components or items. The term "based on" means based at least in part on. Furthermore, it should be understood that the terms and terminology used herein are for purposes of description and should not be considered as limiting. Any headings used herein are for convenience only and should not be construed as having any legal or limiting effect.

[0021] 1 illustrates a block diagram of a radar system 100 for detecting and tracking one or more objects according to one embodiment of the disclosure. The radar system 100 includes a network interface controller (NIC) 101 adapted to connect the radar system 100 to a network 105 (also referred to as a communication channel) via a bus 103. The radar system 100 collects, wirelessly or wired, a sequence of radar image frames 107 over the network 105. The sequence of radar image frames 107 shows radar measurements of a scene at different successive instances in time. The scene may include one or more objects, such as vehicles.

[0022] Further, in some implementations, a human machine interface (HMI) 109 in the radar system 100 connects the radar system 100 to a keyboard 111 and a pointing device 113. The pointing device 113 may include a mouse, a trackball, a touchpad, a joystick, a pointing stick, a stylus, or a touch screen, among others. Furthermore, the radar system 100 includes an application interface 115 for connecting the radar system 100 to an application device 117 for performing various operations. Additionally, the radar system 100 may be linked via the bus 103 to a display interface 119 adapted to connect the radar system 100 to a display device 121, such as a computer monitor, a television, a projector, or a mobile device, among others.

[0023] The radar system 100 further includes a processor 123 and a memory 125 that stores instructions executable by the processor 123. The processor 123 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 125 may include random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory system. The processor 123 is connected to one or more input / output devices via a bus 103. The stored instructions implement a method for tracking objects present in a scene.

[0024] An objective of some embodiments is to address the object recognition problem in automotive radar used for autonomous driving. Although radar is cost-effective and robust under all weather conditions for autonomous driving perception, radar measurements suffer from low angular resolution and accuracy in recognizing surrounding objects. To improve the capabilities of automotive radar, some embodiments utilize temporal information from a sequence of radar image frames 107 to improve the quality and / or resolution of object recognition. Thus, an objective of some embodiments is to exploit the consistency of object presence and features to model relationships between objects in the sequence of radar image frames 107.

[0025] Some embodiments are based on the realization that it is possible to extract various features from a sequence of radar image frames 107 and place these features in context with each other to exploit temporal correlations between the sequence of radar image frames 107. However, while such an approach may be practical for other modalities such as camera images and audio signals, it also obtains poor results for radar object recognition suitable for autonomous driving.

[0026] Some embodiments are based on the recognition that the nature and number of features in each radar image frame is one of the reasons for the poor quality of the temporal correlation of the radar image frames of the sequence 107 of radar image frames. For example, in contrast to camera image frames, the features of radar image frames are semantically poor quality, which makes the temporal correlation more difficult to obtain. Furthermore, the variety and rapid changes of traffic conditions can cause relevant features in the radar image frames to change. Although such changes can be corrected based on the meaning of the relevant features, as described above, the features of the radar image frames are semantically poor quality. These properties of radar image frames, taken together, reduce the ability to train neural networks suitable for object recognition in autonomous driving using automotive radar.

[0027] Some embodiments are based on the recognition that these problems can be addressed by feature extraction with temporal correlation at the object level. Multi-directional feature extraction allows these features to be extracted together from radar image frames to present various temporal representations at the feature level. In contrast, implementing temporal correlation at the object level (as opposed to the feature level) allows for interrelated high-level features of detected objects that are not available at the feature level. Thus, temporal correlation is performed at least twice, both for the feature level and the object level, to improve the quality of the temporal correlation.

[0028] To that end, some embodiments provide a framework that incorporates temporal correlation at the feature and object levels.

[0029] 2A illustrates a schematic diagram of a framework 200 incorporating feature-level and object-level temporal correlation for object recognition, according to one embodiment of the present disclosure. According to one embodiment, a sequence of radar image frames 107 includes a current frame 107a and a previous frame 107b. Additionally or alternatively, the sequence of radar image frames 107 may include multiple previous frames. For ease of explanation, a single previous frame is considered.

[0030] The processor 123 rearranges the order of frames in the sequence 107 of radar image frames to generate multiple rearranged sequences of radar image frames having different frames in dominant positions in corresponding rearranged sequences of radar image frames. For example, the processor 123 generates a first rearranged sequence 201 in which the current frame 107a is in the dominant position and the previous frame 107b is in the subordinate position. Similarly, the processor 123 generates a second rearranged sequence 203 in which the previous frame 107b is in the dominant position and the current frame 107a is in the subordinate position. A dominant position refers to a first or last position in a sequence of radar image frames, and a subordinate position refers to a position other than the dominant position. Here, the dominant position is considered to be the first position.

[0031] Further, the processor 123 processes each of the reordered sequences of radar image frames using a first neural network 205. The first neural network 205 is trained to extract features of a frame at a dominant position that are temporally correlated with features of one or more frames at a subordinate position to generate a temporally enhanced feature for each of the frames in the sequence of radar image frames 107. For example, the processor 123 processes the first reordered sequence 201 using the first neural network 205 to extract features of the current frame 107a that are temporally correlated with features of the previous frame 107b to generate a temporally enhanced feature of the current frame 107a. Similarly, the processor 123 processes the second reordered sequence 203 using the first neural network 205 to extract features of the previous frame 107b that are temporally correlated with features of the current frame 107a to generate a temporally enhanced feature of the previous frame 107b. The weights of the first neural network 205 are shared to process the first permuted sequence 201 and the second permuted sequence 203 .

[0032] Further, based on the temporally enhanced features of the current frame 107a and the temporally enhanced features of the previous frame 107b, the processor 123 reconstructs the feature maps 207 and 209 to represent the temporally enhanced features for the dominant frames of the input sequences 201 and 203, respectively. The feature maps 207 and 209 include feature vectors, each of which represents a potential object from a predominant radar image frame. In addition, the feature vectors may indicate one or more characteristics of the object, such as the object's position, orientation, object's angular curvature, object's length and width, etc.

[0033] Further, the processor 123 selects a list of feature vectors from the feature maps 207 and 209, respectively, based on the selection criteria. For example, in one embodiment, the processor 123 selects feature vectors with top-k activations from the feature maps 207 and 209, respectively. For example, the feature vectors 207a, 207b, 207c, and 207d are feature vectors selected from the feature map 207. Similarly, the feature vectors 209a, 209b, 209c, and 209d are feature vectors selected from the feature map 209. The processor 123 selects these feature vectors and submits them to the second neural network 215 for performing temporal correlation at the object level.

[0034] However, some embodiments are based on the recognition that the selected features 207a-207d and 209a-207d may not include absolute location information to the feature maps 207 and 209. However, location information is important in the temporal relationship of objects because objects at a spatial distance in two consecutive radar image frames are likely to be associated and may share similar object characteristics. To that end, before submitting the selected feature vectors to the second neural network 215, each selected feature vector is supplemented with a location encoding by feature concatenation. For example, the selected feature vectors 207a, 207b, 207c, and 207d are supplemented with location encodings 211a, 211b, 211c, and 211d, respectively. Similarly, the selected feature vectors 209a, 209b, 209c, and 209d are supplemented with location encodings 213a, 213b, 213c, and 213d, respectively. Further, such selected feature vectors supplemented with their respective positional encodings are processed in a second neural network 215 .

[0035] The second neural network 215 is trained to aggregate one or more characteristics of objects in the feature map 207 with one or more characteristics of objects in another feature map 209 to generate a temporally enhanced heat map. In particular, the second neural network 215 is trained to aggregate selected feature vectors 207a, 207b, 207c, and 207d of the feature map 207 with selected feature vectors 209a, 209b, 209c, and 209d of the feature map 209 to generate updated feature vectors 217a, 217b, 217c, and 217d for the feature map 207 and updated feature vectors 219a, 219b, 219c, and 219d for the feature map 209.

[0036] Further, the processor 123 generates a temporally enhanced heat map 221 based on the updated feature vectors 217a, 217b, 217c, and 217d. The processor 123 generates a temporally enhanced heat map 223 based on the updated feature vectors 219a, 219b, 219c, and 219d. Specifically, in one embodiment, the updated feature vectors 217a-217d and 219a-219d are returned to the feature maps of 207 and 209, respectively, according to their spatial locations, and the temporally enhanced heat maps 221 and 223 are generated by the learnable module / head. An example of returning the updated feature vectors 217a-217d to the feature map of 207 is shown in FIG. 2B. The selected feature vectors 207a, 207b, 207c, and 207d from the feature map 207 are processed by a second neural network 215 to generate updated feature vectors 217a, 217b, 217c, and 217d. The updated feature vectors 217a, 217b, 217c, and 217d are returned to the feature map 207 according to their respective spatial locations 227a, 227b, 227c, and 227d. Similarly, the updated feature vectors 219a, 219b, 219c, and 219d are returned to the feature map 209 according to their respective spatial locations.

[0037] Further, in one embodiment, the processor 123 submits the temporally enhanced heatmaps 221 and 223 to different regression heads (with trainable parameters) to generate heatmaps, positions, orientations, lengths and widths of objects. The processor 123 compares the temporally enhanced heatmaps 221 and 223 with corresponding ground truths to calculate a loss function. The loss function is used to train the first neural network 205 and the second neural network 215 to learn the temporally enhanced features. In one embodiment, the ground truth may be constructed from labeled bounding boxes in the two frames 107a and 107b using a 2D Gaussian spread function. In an alternative embodiment, the processor 123 reconstructs the radar image 225 based on the temporally enhanced heatmap 221 and the temporally enhanced heatmap 223. Additionally or alternatively, an embodiment reconstructs the radar image 225 using a weighted combination of the temporally enhanced heatmap 221 and the temporally enhanced heatmap 223. In this manner, the framework 200 can also take into account temporal correlations on the level of radar image reconstruction. As a result, the temporal considerations are extended to three levels: the feature level, the object level, and the radar image reconstruction level.

[0038] FIG. 3 shows a schematic diagram for reconstructing a radar image 225 using a weighted combination of the temporally enhanced heatmap 221 and the temporally enhanced heatmap 223, according to one embodiment of the present disclosure. For example, in the weighted combination, a weight w1 is associated with the temporally enhanced heatmap 221, and a weight w2 is associated with the temporally enhanced heatmap 223. In different embodiments, the weights w1 and w2 in the weighted combination are the same or different. In some embodiments, the weights w1 and w2 in the weighted combination are predetermined or adaptively determined online during radar image reconstruction. In different embodiments, the weighted combination is analytically determined or learned using a neural network trained to accept input data indicative of the temporally enhanced heatmaps 221 and 223 to output the radar image 225. The neural network corresponds, for example, to a ResNet, an EfficientNet, or a convolutional neural network.

[0039] According to one embodiment, the first neural network 205 in the framework 200 is a convolutional neural network (CNN) with or without skip connections. For example, in one embodiment, the CNN is a deep convolutional neural network with skip connections for multi-scale feature extraction and ensembling. Such a CNN is advantageous because it can process multiple radar image frames together based on their order. For example, the deep convolutional neural network can accept a sequence 107 of radar image frames stacked into a tensor to form a multi-channel radar image. The deep convolutional neural network is trained with a loss function that minimizes the error of the output of the deep convolutional neural network using ground truth information of the frames in the dominant positions. The deep convolutional neural network with skip connections is described below in connection with FIG. 4.

[0040]

number

[0041]

number

[0042]

number

[0043]

number

[0044]

number

[0045]

number

[0046] Further, in some embodiments, Zc and Z p Regression is applied to obtain object properties such as center coordinates, orientation, length and width.

[0047]

number

[0048]

number

[0049]

number

[0050]

number

[0051]

number

[0052]

number

[0053] For simplicity, the balancing coefficients for each term are omitted. The training objective is a loss function used to train the parameters in the first neural network 205 and the second neural network 215. The parameters include weights in the first neural network 205 and the second neural network 215 and regression heads (e.g., heatmap heads, length / width regression heads, orientation heads, etc.). For each training step, the loss L is calculated and run backwards simultaneously for both the current frame 107a and the previous frame 107b. In the current frame 107a, the object in the current frame 107a receives information from the previous frame 107b for object recognition. On the other hand, from the perspective of the previous frame, the object utilizes the time information from the nearest future frame. Therefore, the training objective (12) can be envisioned as a bidirectional backward-forward training toward two consecutive frames.

[0054] Different embodiments use different machine learning paradigms to train the first neural network 205 and / or the second neural network 215. For example, one embodiment provides end-to-end multi-branch training with each branch designed for a separate permuted sequence of radar image frames, where end-to-end refers to training the first neural network 205 and / or the second neural network 215 from input to output, and multi-branch refers to two branches for the current frame and the previous frame.

[0055] Some embodiments are based on the recognition that the framework 200 can be extended for multiple object tracking. For multiple object tracking, a regression head is added to the center feature vector to predict the 2D translation offset between the centers of objects that hold the same tracking ID in the current and previous frames. Furthermore, Euclidean distance is used to associate objects from consecutive frames. The algorithm for multiple object tracking is described below in FIG. 5.

[0056] 5 illustrates an algorithm 500 for multiple object tracking according to one embodiment of the present disclosure. Specifically, the algorithm 500 pairs N detected objects with M previously tracked objects based on their Euclidean distance. For example, W in line 2 ij calculate an offset-compensated distance between the detected object i and the pre-tracked object ID j. Based on whether the offset-compensated distance is greater than or less than the distance threshold k, the object i may be merged with the pre-tracked object ID j.

[0057] FIG. 6 illustrates a block diagram of a method 600 for detecting and tracking one or more objects according to an embodiment of the present disclosure. In block 601, the method 600 includes collecting a sequence 107 of radar image frames showing radar measurements of a scene including one or more objects at different successive instances of time. The sequence 107 of radar image frames includes a current frame 107a and at least one previous frame 107b. In block 603, the method 600 includes rearranging the order of frames in the sequence 107 of radar image frames to generate a plurality of rearranged sequences of radar image frames having different frames in a dominant position in a corresponding rearranged sequence of radar image frames. For example, a first rearranged sequence 201 is generated with the current frame 107a in a dominant position and the previous frame 107b in a subordinate position. Similarly, a second rearranged sequence 203 is generated with the previous frame 107b in a dominant position and the current frame 107a in a subordinate position.

[0058] At block 605, method 600 includes processing each of the sorted sequence of radar image frames with a first neural network 205 trained to extract features of a frame at a dominant position that are temporally correlated with features of one or more frames at a subposition to generate a temporally enhanced feature for each of the frames in the sequence of radar image frames 107. At block 607, method 600 includes reconstructing a feature map indicative of one or more characteristics of one or more objects in the scene from the temporally enhanced features of each of the frames in the sequence of radar image frames to generate a sequence of feature maps, e.g., feature maps 207 and 209 as described in FIG.

[0059] At block 609, the method 600 includes processing the list of feature vectors from each feature map of the sequence of feature maps with a second neural network 215 trained to aggregate one or more characteristics of an object in one feature map (e.g., feature map 207) with one or more characteristics of an object in another feature map (e.g., feature map 209) to generate a temporally enhanced heatmap. At block 611, the method 600 includes processing the temporally enhanced heatmap based on one or more regression heads to generate at least one characteristic of the one or more objects.

[0060] FIG. 7 is a schematic diagram illustrating a computing device 700 for implementing the method and system of the present disclosure. The computing device 700 includes a power supply 701, a processor 703, a memory 705, and a storage device 707, all of which are connected to a bus 709. Further, a high-speed interface 711, a low-speed interface 713, a high-speed expansion port 715, and a low-speed connection port 717 can be connected to the bus 709. In addition, a low-speed expansion port 719 is connected to the bus 709. Further, an input interface 721 can be connected to an external receiver 723 and an output interface 725 via the bus 709. The receiver 727 can be connected to an external transmitter 729 and a transmitter 731 via the bus 709. Also, an external memory 733, an external sensor 735, a machine 737, and an environment 739 can be connected to the bus 709. Further, one or more external input / output devices 741 can be connected to the bus 709. A network interface controller (NIC) 743 may be adapted to connect to a network 745 via bus 709, and data or other data may be rendered on, among other things, a third party display device, a third party imaging device, and / or a third party printing device outside of the computing device 700.

[0061] The memory 705 can store instructions executable by the computing device 700, as well as any data that can be utilized by the methods and systems of the present disclosure. The memory 705 can include random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory system. The memory 705 can be a volatile and / or non-volatile memory unit. The memory 705 can also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0062] The storage device 707 may be adapted to store supplemental data and / or software modules used by the computing device 700. The storage device 707 may include a hard drive, an optical drive, a thumb drive, an array of drives, or any combination thereof. Additionally, the storage device 707 may include an array of devices including computer readable media such as floppy disk devices, hard disk devices, optical disk devices, or tape devices, flash memory or other similar solid state memory devices, or storage area networks or other configuration devices. The instructions may be stored on an information carrier. The instructions, when executed by one or more processing devices (e.g., processor 703), perform one or more methods as described above.

[0063] The computing device 700 can be coupled via bus 709 to an optional display or user interface (HMI) 747 adapted to connect the computing device 700 to a display device 749 and a keyboard 751, which may include, among others, a computer monitor, a camera, a television, a projector, or a mobile device. In some implementations, the computing device 700 can include a printer interface for connecting to a printing device. The printing device may include, among others, a liquid inkjet printer, a solid ink printer, a large scale commercial printer, a thermal printer, a UV printer, or a dye-sublimation printer.

[0064] The high-speed interface 711 manages bandwidth-intensive operations for the computing device 700, and the low-speed interface 713 manages lower-bandwidth intensive operations. Such an allocation of functions is merely an example. In some implementations, the high-speed interface 711 may be coupled to memory 705, a user interface (HMI) 747, a keyboard 751 and a display 749 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 715 that may accept various expansion cards via a bus 709. In one implementation, the low-speed interface 713 is coupled to a storage device 707 and a low-speed expansion port 717 via a bus 709. The low-speed expansion port 717, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices 741. The computing device 700 may be connected to a server 753 and a rack server 755. The computing device 700 may be implemented in several different forms. For example, the computing device 700 may be implemented as part of a rack server 755 .

[0065] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. It is contemplated that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

[0066] In the following description, specific details are given to provide a thorough understanding of the embodiments. However, those skilled in the art will understand that the embodiments may be practiced without these specific details. For example, systems, processes and other elements in the disclosed subject matter may be shown as components in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other cases, well-known processes, structures and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments. Additionally, like reference numbers and names in the various drawings refer to like elements.

[0067] Also, particular embodiments may be described as a process that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. A process may be terminated when its operations are completed, but may include additional steps not described or included in the diagram. Moreover, not all operations in any process specifically described may be performed in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, or the like. When a process corresponds to a function, the end of the function may correspond to a return of the function to the calling function or to the main function.

[0068] Furthermore, embodiments of the disclosed subject matter may be implemented, at least in part, either manually or automatically. The manual or automated implementations may be performed or at least assisted by the use of machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored on a machine-readable medium. A processor may perform the necessary tasks.

[0069] The various methods or processes outlined herein may be coded as software executable on one or more processors employing any one of a variety of operating systems or platforms. In addition, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and compiled as executable machine language code or intermediate code that runs on a framework or virtual machine. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0070] The embodiments of the present disclosure may be embodied as methods, which are provided as examples. The operations performed as part of the method may be ordered in any suitable manner. Thus, while shown as sequential operations in the exemplary embodiments, embodiments may be constructed in which operations are performed in a different order than that shown, including performing some operations simultaneously.

[0071] Although the present disclosure has been described with reference to certain preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the present disclosure. It is therefore the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the present disclosure.

Claims

1. 1. A radar system for detecting and tracking one or more objects, the radar system including a processor and a memory having stored thereon instructions that, when executed by the processor, cause the radar system to: The instructions further include: collecting a sequence of radar image frames indicative of radar measurements of a scene at different successive instances in time, the scene including the one or more objects, the sequence of radar image frames including a current frame and at least one previous frame. permuting the order of frames in a sequence of radar image frames to generate a plurality of permuted sequences of radar image frames having frames that differ in dominant positions in corresponding permuted sequences of radar image frames; processing each of the permuted sequences of radar image frames with a first neural network trained to extract features of frames at the dominant location that are temporally correlated with features of one or more frames at sub-locations to generate temporally enhanced features for each of the frames in the sequence of radar image frames; reconstructing a feature map indicative of one or more properties of the one or more objects in the scene from the temporally enhanced features of each of the frames in the sequence of radar image frames to generate a sequence of feature maps; and processing a list of feature vectors from each feature map of the sequence of feature maps with a second neural network trained to aggregate the one or more properties of objects in one feature map with one or more properties of objects in another feature map to generate a temporally enhanced heat map, each feature vector supplemented with a positional encoding, the instructions further comprising: and processing the temporally enhanced heat map based on one or more regression heads to generate at least one characteristic of the one or more objects.

2. 2. The radar system of claim 1, wherein the processor is further configured to reconstruct a radar image of the scene using a weighted combination of the temporally enhanced heat maps corresponding to the permuted sequence of the radar image frames having frames that differ in the dominant position.

3. 3. The radar system of claim 2, wherein the weighted combination is analytically determined or learned using a neural network trained to accept input data indicative of the temporally enhanced heatmap and to output the radar image.

4. 2. The radar system of claim 1, wherein the first neural network is a deep convolutional neural network with skip connections for multi-scale feature extraction and ensembling, the deep convolutional neural network configured to accept a plurality of frames stacked into tensors to form a multi-channel radar image and trained with a loss function, the loss function configured to minimize an error in an output of the deep convolutional neural network using ground truth information of frames at the dominant positions.

5. 10. The radar system of claim 1, wherein the second neural network employs an attention mechanism to place corresponding object characteristics from different feature maps in context with each other.

6. The processor further comprises: Selecting feature vectors with the top k activations on the feature map; supplementing the selected feature vectors with corresponding positional encodings; 2. The radar system of claim 1, configured to process the selected feature vectors supplemented with the corresponding location encoding using the second neural network. 【Request 7】 【Number 1】 2. The radar system of claim 1, configured to determine a heat map of the scene based on a sigmoid function having the temporally enhanced feature map as an input.

8. 2. The radar system of claim 1, wherein one or more of the first neural network and the second neural network are trained based on end-to-end multi-branch training, where each branch is designed for a separate permuted sequence of radar image frames.

9. 1. A method for detecting and tracking one or more objects, comprising: collecting a sequence of radar image frames indicative of radar measurements of a scene at different successive instances in time, the scene including the one or more objects, the sequence of radar image frames including a current frame and at least one previous frame, the method further comprising: reordering frames in a sequence of radar image frames to generate a plurality of reordered sequences of radar image frames having frames that differ in dominant positions in corresponding reordered sequences of radar image frames; processing each permuted sequence of radar image frames with a first neural network trained to extract features of a frame at the dominant location that are temporally correlated with features of one or more frames at a sublocation to generate a temporally enhanced feature for each of the frames in the sequence of radar image frames; reconstructing a feature map indicative of one or more properties of the one or more objects in the scene from the temporally enhanced features of each of the frames in the sequence of radar image frames to generate a sequence of feature maps; and processing a list of feature vectors from each feature map of the sequence of feature maps using a second neural network trained to aggregate the one or more properties of objects in one feature map with one or more properties of objects in another feature map to generate a temporally enhanced heat map, each feature vector supplemented with a positional encoding, the method further comprising: processing the temporally enhanced heatmap based on one or more regression heads to generate at least one characteristic of the one or more objects.

10. 10. The method of claim 9, further comprising reconstructing a radar image of the scene using a weighted combination of the temporally enhanced heat maps corresponding to the permuted sequence of the radar image frames having frames that differ in the dominant position.

11. 10. The method of claim 9, wherein the weighted combination is analytically determined or learned using a neural network trained to accept input data indicative of the temporally enhanced heatmap and output the radar image.

12. 10. The method of claim 9, wherein the first neural network is a deep convolutional neural network with skip connections for multi-scale feature extraction and ensembling, the deep convolutional neural network configured to accept a plurality of frames stacked into tensors to form a multi-channel radar image and trained with a loss function, the loss function configured to minimize an error in an output of the deep convolutional neural network using ground truth information of frames at the dominant positions.

13. 10. The method of claim 9, wherein the second neural network uses an attention mechanism to place features of corresponding objects from different feature maps in context with each other.

14. The method further comprises: selecting feature vectors with the top k activations on the feature map; supplementing the selected feature vectors with corresponding positional encodings; and processing the selected feature vectors supplemented with the corresponding positional encoding using the second neural network.

15. 10. The method of claim 9, wherein one or more of the first neural network and the second neural network are trained based on end-to-end multi-branch training, where each branch is designed for a separate permuted sequence of radar image frames.

16. 1. A non-transitory computer readable storage medium having embodied thereon a program executable by a processor for performing a method for detecting and tracking one or more objects, the method comprising: collecting a sequence of radar image frames indicative of radar measurements of a scene at different successive instances in time, the scene including the one or more objects, the sequence of radar image frames including a current frame and at least one previous frame, the method further comprising: reordering frames in a sequence of radar image frames to generate a plurality of reordered sequences of radar image frames having frames that differ in dominant positions in corresponding reordered sequences of radar image frames; processing each permuted sequence of radar image frames with a first neural network trained to extract features of a frame at the dominant location that are temporally correlated with features of one or more frames at a sublocation to generate a temporally enhanced feature for each of the frames in the sequence of radar image frames; reconstructing a feature map indicative of one or more properties of the one or more objects in the scene from the temporally enhanced features of each of the frames in the sequence of radar image frames to generate a sequence of feature maps; and processing a list of feature vectors from each feature map of the sequence of feature maps using a second neural network trained to aggregate the one or more properties of objects in one feature map with one or more properties of objects in another feature map to generate a temporally enhanced heat map, each feature vector supplemented with a positional encoding, the method further comprising:

22. A non-transitory computer-readable storage medium comprising: processing the temporally enhanced heatmap based on one or more regression heads to generate at least one characteristic of the one or more objects.

17. 20. The non-transitory computer-readable storage medium of claim 16, wherein the method further comprises reconstructing a radar image of the scene using a weighted combination of the temporally enhanced heat maps corresponding to the permuted sequence of the radar image frames having frames that differ in the dominant position.

18. 17. The non-transitory computer-readable storage medium of claim 16, wherein the first neural network is a deep convolutional neural network with skip connections for multi-scale feature extraction and ensembling, the deep convolutional neural network configured to accept a plurality of frames stacked into tensors to form a multi-channel radar image and trained with a loss function, the loss function minimizing an error in an output of the deep convolutional neural network using ground truth information of frames at the dominant positions.

19. 20. The computer-readable storage medium of claim 16, wherein the second neural network employs an attention mechanism to place characteristics of corresponding objects from different feature maps in context with each other.

20. The method further comprises: selecting feature vectors with the top k activations on the feature map; supplementing the selected feature vectors with corresponding positional encodings; and processing the selected feature vectors supplemented with the corresponding positional encoding with the second neural network.

Images