System and method for camera-to-radar knowledge distillation

Abstract

To provide a workflow for training a radar data processing model with reduced requirements for annotated training data.SOLUTION: A training workflow utilizes an imaging sensor, such as a camera, that is synchronized with a radar sensor to capture a synchronized image and radar point cloud data. The training workflow utilizes, using a pre-trained image model, self-supervised knowledge distillation processing to pre-train a radar data processing model on an unannotated synchronized image and radar point cloud data. The radar data processing model is then fine-tuned using a limited set of annotated radar point cloud data, thereby significantly reducing a burden of human annotation.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] Technical Field Apparatus and methods disclosed herein relate to machine learning, and more particularly, to training radar data processing models. [Background technology]

[0002] background Unless otherwise expressly stated herein, the material described in this section is not admitted to be prior art by its inclusion in this section.

[0003] The extensive development of mmWave radars has led to their widespread incorporation into commercial vehicles to gather precise distance, direction and speed information from objects in the surrounding environment, making them a key component in autonomous driving systems. These capabilities make radars a perfect candidate for object detection, one of the most critical tasks in autonomous driving solutions, especially in extreme conditions such as fog, rain and other visibility-denying conditions. Radars can enable more accurate object detection and image segmentation by leveraging distance, direction and speed measurements of objects. Similarly, there is a significant potential for radar systems to be employed in a variety of other scenarios, such as human detection indoors, smart vacuum cleaners, or object detection for autonomous driving outdoors. Summary of the Invention [Problem to be solved by the invention]

[0004] However, due to the limitation of radar point sparsity, existing radar-based approaches require significant human effort to annotate multiple radar point clouds to build large data sets, making the use of existing radar-based approaches prohibitively costly. What is needed is an approach that reduces the burden of human annotation of radar point clouds so that radar-based approaches can be deployed at lower cost. [Means for solving the problem]

[0005] overview Described herein is a method for training a first neural network to perform a radar data processing task. The method includes receiving, by a processor, a plurality of training data pairs, each of which includes a respective image and a respective radar point cloud captured synchronously with respect to a same scene. The method further includes training, by the processor, in a first phase based on the plurality of training data pairs, the first neural network to extract features from the radar point cloud using a second neural network pre-trained to extract features from the image. The method further includes receiving, by the processor, a plurality of annotated radar point clouds having labels corresponding to the radar data processing task. The method further includes further training, by the processor, in a second phase based on the plurality of annotated radar point clouds, the first neural network to perform the radar data processing task.

[0006] A non-transitory computer readable medium is described herein. The non-transitory computer readable medium has program instructions stored thereon, which, when executed by a processor, cause the processor to receive a plurality of training data pairs, each of which includes a respective image and a respective radar point cloud captured synchronously with each other of a same scene. When executed by the processor, the program instructions further cause the processor to train a first neural network to extract features from the radar point cloud in a first phase based on the plurality of training data pairs using a second neural network pre-trained to extract features from the image. When executed by the processor, the program instructions further cause the processor to receive a plurality of annotated radar point clouds having labels corresponding to a radar data processing task. When executed by the processor, the program instructions further cause the processor to further train the first neural network to perform a radar data processing task in a second phase based on the plurality of annotated radar point clouds.

[0007] The above aspects and other features of the method are explained in the following description, with reference to the accompanying drawings. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 summarizes a workflow for training a radar data processing model with reduced requirements for annotated training data. [Diagram 2] FIG. 1 illustrates an example embodiment of a computing device that can be used to train a radar data processing model. [Diagram 3] 1 is a flow chart of a method for training a radar data processing model configured to perform a radar data processing task. [Figure 4] 1 is a flow chart of a method for pre-training a radar data processing model in a pre-training phase that leverages superpixels and superpoints. [Figure 5A] FIG. 2 illustrates an example of segmenting an image into superpixels. [Figure 5B] FIG. 2 illustrates an example of segmenting an image into superpixels. [Figure 5C] FIG. 2 illustrates an example of segmenting an image into superpixels. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Detailed Description For the purpose of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings and described in the following specification. It is understood that no limitation to the scope of the present disclosure is intended thereby. Moreover, it is understood that any alternatives and modifications to the illustrated embodiments, and further applications of the principles of the present disclosure that would normally occur to a person skilled in the art to which the present disclosure pertains, are included in the present disclosure.

[0010] overview FIG. 1 summarizes a workflow 10 for training a radar data processing model 30 with reduced requirements for annotated training data. In the illustrated embodiment, the radar data processing model 30 is a U-Net-based model having an encoder 40 followed by a decoder 50. In particular, the encoder 40 is configured to extract features from the radar point cloud, and the decoder 50 is configured to perform downstream tasks on the radar point cloud based on the extracted features. The training workflow 10 leverages an image sensor, such as a camera, that is synchronized with the radar sensor to capture synchronized image and radar point cloud data. The training workflow 10 utilizes a self-supervised knowledge distillation process to pre-train the radar data processing model 30 with unannotated synchronized image and radar point cloud data, followed by fine-tuning with a limited set of annotated radar point cloud data, thereby significantly reducing the burden of human annotation.

[0011] The training workflow 10 consists of two phases: a pre-training phase and a fine-tuning phase. In the pre-training phase, the pre-trained image model 20 is utilized to pre-train the encoder 40 of the radar data processing model 30. In particular, aligned image and radar point cloud pairs are input to the pre-trained image model 20 and the encoder 40 of the radar data processing model 30, respectively. The encoder 40 of the radar data processing model 30 is pre-trained by performing matching between 2D pixel features extracted from the aligned images and 3D point features extracted from the radar point cloud. In the fine-tuning phase, the decoder 50 of the radar data processing model 30 is trained using a small set of annotated radar point cloud data with labels corresponding to a downstream task to be learned, such as object detection or semantic segmentation.

[0012] Thus, the training workflow 10 advantageously transfers knowledge from existing camera image-based models and significantly reduces the human effort required to label radar point cloud data while achieving comparable performance. In particular, the radar data processing model 30 is pre-trained on large amounts of unannotated data, and later requires only a small set of annotated data for fine-tuning.

[0013] The training workflow 10 may be useful for a wide range of applications including autonomous driving, surveillance and public safety applications, intrusion detection, and vulnerable pedestrian detection. A system capturing co-located camera and radar data may be used for training after which the radar may be operated independently to detect objects of interest and their distance from the radar sensor using the proposed method. In autonomous vehicle applications, such radar independent operation may assist in obstacle avoidance in poor visibility scenarios. In surveillance and public safety applications, such radar independent operation may notify authorities or guards or trigger an alarm if an intrusion is detected.

[0014] Exemplary Hardware Implementation 2 illustrates an exemplary embodiment of a computing device 100 that can be used to train a radar data processing model 30 to perform radar data processing tasks such as object detection or semantic segmentation. Similarly, the computing device 100 can be used to operate a pre-trained radar data processing model 30 to perform radar data processing tasks. The computing device 100 includes a processor 110, a memory 120, a display screen 130, a user interface 140, and at least one network communication module 150. It will be understood that the illustrated embodiment of the computing device 100 is merely one exemplary embodiment and represents any of a variety of different approaches or configurations of a server, desktop computer, laptop computer, mobile phone, tablet computer, or any other computing device that operates in the manner described herein. In at least some embodiments, the computing device 100 is in communication with a database 102, which may be hosted by another device or may be stored in the memory 120 of the computing device 100 itself.

[0015] The processor 110 is configured to execute instructions to operate the computing device 100 to enable the features, functions, characteristics, and / or the like as described herein. To this end, the processor 110 is operatively connected to the memory 120, the display screen 130, and the network communication module 150. The processor 110 generally includes one or more processors, which may operate in parallel with one another or may otherwise operate in cooperation with one another. Those skilled in the art will recognize that a "processor" includes any hardware system, mechanism, or component that processes data, signals, or other information. Thus, the processor 110 may include a system with a central processing unit, a graphics processing unit, multiple processing units, dedicated circuits to achieve functions, programmable logic, or other processing systems.

[0016] Memory 120 is configured to store data and program instructions that, when executed by processor 110, enable computing device 100 to perform the various operations described herein. Memory 120 may be any type of device capable of storing information accessible by processor 110, such as a memory card, ROM, RAM, a hard drive, a disk, flash memory, or any of a variety of other computer-readable media that function as a data storage device as will be appreciated by those skilled in the art.

[0017] Display screen 130 may include any of a variety of different known types of displays, such as an LCD or OLED screen configured to display a graphical user interface. User interface 140 may include a variety of different interfaces for operating computing device 100, such as buttons, switches, a keyboard or other keypad, a speaker, and a microphone. Alternatively or additionally, display screen 130 may include a touch screen configured to receive touch input from a user.

[0018] Network communication module 150 may include one or more transceivers, modems, processors, memory, oscillators, antennas, or other hardware conventionally included in a communication module to enable communication with a variety of other devices. In particular, network communication module 150 typically includes an Ethernet adapter or a Wi-Fi module configured to enable communication with a wired or wireless network, and / or includes a router (not shown) configured to enable communication with a variety of other devices. Additionally, network communication module 150 may include a Bluetooth module (not shown), as well as one or more cellular modems configured to communicate with a wireless telephone network.

[0019] In at least some embodiments, memory 120 stores program instructions for radar data processing model 30 that are configured to perform a radar data processing task once training has been performed. In at least some embodiments, database 102 stores a plurality of training data pairs, each training data pair including a respective image and a respective radar point cloud that are synchronously captured of a scene. Additionally, in at least some embodiments, database 102 stores a plurality of annotated radar point clouds having labels that correspond to the radar data processing task to be learned.

[0020] Method for training a radar data processing model In the following, different operations and processes are described for operating computing device 100 to develop and train radar data processing model 30 to perform radar data processing tasks. In these descriptions, a description of a method, processor, and / or system performing a task or function refers to a controller or processor (e.g., processor 110 of computing device 100) executing programmed instructions stored in a non-transitory computer-readable storage medium (e.g., memory 120 of computing device 100) operably connected to the controller or processor to manipulate data or operate one or more components within computing device 100 or of database 102 to perform the task or function. Additionally, the steps of the method may be performed in any feasible temporal order, regardless of the order shown in the figures or the order in which the steps are described.

[0021] FIG. 3 shows a flow chart for a method 200 for training a radar data processing model configured to perform a radar data processing task. A key insight and advantage of the method 200 is that object-level representations can be transferred between models using contrastive learning algorithms. In particular, with self-supervised knowledge distillation, the encoder 40 of the radar data processing model 30 learns to extract latent representations of radar point cloud data by leveraging existing image models and using aligned radar and image data. Additionally, the method 200 advantageously leverages superpixels and superpoints. In particular, instead of performing distillation point-by-point or pixel-by-pixel, the method 200 groups points and pixels that belong to the same object and performs contrastive learning to distill knowledge from camera to radar. The method 200 uses contrastive loss to perform matching between pairs of corresponding 3D radar point features and 2D image pixel features.

[0022] The method 200 begins by receiving a plurality of training data pairs, each of which includes a synchronously captured image and a radar point cloud of a scene (block 210). In particular, the processor 110 receives the plurality of training data pairs, <P t ,I t and / or the database 102 receives a plurality of training data pairs <P t ,I t Each training data pair is captured by a radar sensor.

number

number

[0023] The method 200 continues by training the first neural network to extract features from the radar point cloud using the second neural network that was pre-trained to extract features from images in a pre-training phase based on a plurality of training data pairs (block 220). In particular, in the pre-training phase, the processor 110 trains the first neural network to extract features from the radar point cloud using the second neural network that was pre-trained to extract features from images in a pre-training phase based on a plurality of training data pairs. <P t ,I t Based on the pre-trained,h,class,, the feature extraction from images is done by ω We use a pre-trained image model 20 denoted by (·) to extract features from the radar point cloud, θ The radar data processing model 30 is trained using the feature output f θ Let (P) be the corresponding image model output h from the pre-trained image model 20. ω (I) and (II). In at least one embodiment, the parameters (e.g., kernel weights, model coefficients, etc.) of the pre-trained image model 20 are frozen while training the radar data processing model 30 in the pre-training phase.

[0024] In some embodiments, the pre-trained image model 20

number

[0025] In some embodiments, the radar data processing model 30

number

[0026] During the pre-training phase, for each training data pair <Pt ,I t For each radar point cloud P t is provided as an input to the encoder 40 of the radar data processing model 30, and each image I t as an input to the pre-trained image model 20. Using the pre-trained image model 20, the processor 110 t Based on the D-dimensional image feature map h ω Similarly, using the encoder 40 of the radar data processing model 30, the processor 110 determines the radar point cloud P t Based on the D-dimensional radar feature map f θ The processor 110 also determines the radar signature map f θ (P) and image feature map h ω (I) to determine a training loss based on the comparison. In at least one embodiment, the training loss is a control loss. Later, the processor 110 refines the radar data processing model 30 (e.g., updates model coefficients, weights, parameters, etc.) based on the determined training loss.

[0027] In at least some embodiments, the processor 110 is configured to: (i) generate a plurality of training data pairs; <P t ,I t > Radar point cloud P t points, and (ii) multiple training data pairs <P t ,I t >Image I t In particular, the first step in knowledge distillation is to align the radar point cloud with the camera image. Based on the known relative pose of the radar and camera sensors, processor 110 determines the mapping between the pixels of each radar point p i onto the camera frame to obtain image I t More specifically, the processor 110 performs a mapping

number

[0028] It should be appreciated that simply distilling knowledge in a pixel-to-radar point manner may not be able to transfer useful knowledge to the radar data processing model 30, thus hindering the goal of pre-training the radar data processing model 30 using an existing pre-trained image model 20 without any human annotation. The reason is that the radar point cloud from a radar sensor has a different resolution compared to images from a camera sensor. Furthermore, radar is capable of recognizing object-level information instead of pixel-level information. Therefore, rather than distilling overly coarse scene-level knowledge from the pre-trained image model 20 to the radar data processing model 30, it is advantageous for the knowledge distillation process to distill object-level knowledge instead. To this end, in at least some embodiments, superpixels and superpoints are utilized to improve the knowledge distillation process.

[0029] 4 shows a flow chart for a method 300 for pre-training a radar data processing model in a pre-training phase that leverages superpixels and superpoints. It should be understood that method 300 is one example embodiment of the process of block 220 in method 200 of FIG.

[0030] Method 300 begins by (i) segmenting each image from each training data pair into superpixels, and (ii) segmenting each radar point cloud from each training data pair into superpoints (block 310). In particular, in a pre-training phase, each training data pair <P t ,I t For each image I t of,

number

[0031] Similarly, in the pre-training phase, for each training data pair <P t ,I t For each radar point cloud P t of,

number

[0032] As will be described in further detail below, superpixels and superpoints can be defined in different ways to enhance the knowledge distillation process. In a first approach to defining superpixels and superpoints, the processor 110 determines the segmented images and thus applies a machine learning model, such as an image segmentation algorithm, e.g., the graph-based Felzenszwalb and Huttenlocher (FH) segmentation algorithm, or a pixel clustering algorithm, e.g., the Simple Linear Iterative Clustering (SLIC) algorithm, to each image I. t 5A-5C show examples of segmenting an image into superpixels. In particular, FIG. 5A shows an original image 400 captured by a camera in an urban environment with cars and buildings. FIG. 5B shows a segmented image 410 generated by segmenting the image 400 into superpixels using the FH algorithm. FIG. 5C shows a segmented image 420 generated by segmenting the image 400 into superpixels using the SLIC algorithm.

[0033] The processor 110 then determines the segmented radar point cloud and thus, for example, the mapping εp Using the t For each point in each image I t 5C, which shows radar points (illustrated as dots) projected onto the segmented image 420. Processor 110 defines a superpoint for each of the segmented radar point clouds as a group of radar points that map to the same superpixel in the respective segmented image. In other words, each superpoint

number

number

number

[0034] In a second approach to defining superpixels and superpoints, contrary to the first approach, the processor 110 determines a segmented radar point cloud and then applies a point clustering algorithm, e.g., a machine learning model such as the DBScan algorithm, to each radar point cloud P. t Define super points by applying

[0035] The processor 110 then initially defines superpixels of the segmented image by applying an image segmentation algorithm or pixel clustering algorithm in a manner similar to that described with respect to the first approach. However, these initially defined superpixels are not necessarily clustered in a single pixel array, but are instead clustered in a single pixel array, such that each of the initially defined superpixels corresponds to a pixel cluster of each of the radar point clouds P. t The superpixels are further merged by processor 110 by matching each superpixel in each segmented image with a respective superpoint in the radar point cloud. Processor 110 defines a final superpixel of each segmented image as a group of initially defined superpixels that map to the same superpoint in the respective segmented radar point cloud. Alternatively, in a second approach, the application of an image segmentation algorithm or pixel clustering algorithm can be omitted, and instead, processor 110 performs a pixel clustering algorithm on each image I to define the superpixels. t For each original pixel in t In other words, we simply match each final superpixel

number

number

number

[0036] Using the second approach to define superpixels and superpoints, clustering methods are utilized to differentiate radar points that belong to different objects. If radar points that mostly belong to the same object are clustered into the same group, the aligned superpixels are combined together to form an object-level representation, thus overcoming the problem of image over-segmentation. However, due to the sparsity of radar point clouds, the shape of the points is arbitrary with density variations and some outliers that introduce extra difficulties in selecting a suitable clustering method.

[0037] In a third approach for defining superpixels and superpoints, similar to the second approach, the processor 110 determines the segmented radar point clouds and therefore defines the superpoints by applying a point clustering algorithm, e.g., a machine learning model such as the DBScan algorithm. However, the point clustering algorithm or other machine learning model is not applied to each radar point cloud P t Instead of only applying the same to each radar point cloud P, the processor 110 t For each radar point cloud P t A predetermined number of adjacent radar point clouds captured at a time point immediately before or after the time point t at which the radar point cloud P t-1 And the radar point cloud P immediately after t+1The first approach combines the first and second radar point clouds to generate one combined radar point cloud. This combined radar point cloud helps to overcome the sparsity problem of the radar point cloud. The processor 110 determines the segmented radar point cloud and therefore defines superpixels by applying a point clustering algorithm, e.g., a machine learning model such as the DBScan algorithm or the Hungarian method, to the combined radar point cloud. The processor 110 then defines superpixels of the segmented image using a process similar to that described with respect to the second approach.

[0038] 4, the method 300 continues by (i) determining an image feature map based on each image using the second neural network, and (ii) determining a radar feature map based on each radar point cloud using the first neural network (block 320). In particular, similar to that described above, in a pre-training phase, each training data pair <P t ,I t For each radar point cloud P t is provided as an input to the encoder 40 of the radar data processing model 30, and each image I t as an input to the pre-trained image model 20. Using the pre-trained image model 20, the processor 110 t Based on the D-dimensional image feature map h ω Similarly, using the encoder 40 of the radar data processing model 30, the processor 110 determines the radar point cloud P t Based on the D-dimensional radar feature map f θ (P) is also determined.

[0039] The method 300 continues by determining a contrast loss between the image feature map and the radar feature map in a superpixel-to-superpoint manner (block 330). In particular, the processor 110 determines a contrast loss between the image feature map and the radar feature map f θ and image feature map h ω Based on the mapping ε p The contrast loss is determined using the superpixel-to-superpoint method. However, to provide a more useful contrast loss, the processor 110 determines the contrast loss in a superpixel-to-superpoint manner. It should be appreciated that this superpixel-level contrast loss has several advantages. The superpixel-level contrast loss gives the same weight to all regions in the contrast loss regardless of the sampling density of the radar points. Furthermore, with the superpixel-level contrast loss, the contrast loss is small enough between the corresponding superpixel and superpoint, but large enough to differentiate features from another superpixel of different categories in the same scene. Determining the contrast loss between two feature maps in a superpixel-to-superpoint manner helps ensure that the knowledge distillation is at the object level. Compared to pixel-level contrast learning, determining the contrast loss in a superpixel-to-superpoint manner does not face the problem of contrast between nearly identical points inside a superpixel, which may create a “false negative” problem for self-supervised learning. Conversely, scene-level contrastive losses, which typically contrast a global representation of the entire radar point cloud with the corresponding entire camera image, cannot differentiate between object-level differences and offer limited diversity since most scenes have objects of the same class, e.g., cars, pedestrians, etc.

[0040] Thus, the processor 110 generates a segmented radar feature map f θ (i.e., the respective superpoint feature outputs for each superpoint in each radar point cloud) and the segmented image feature map hω The processor 110 matches each superpoint in each radar point cloud with each superpixel in each image and determines a contrast loss based on the matched superpoint feature output and the superpixel feature output. In particular, in at least one embodiment, for a given camera image I c For each superpixel, the processor 110 computes a contrastive loss from each corresponding superpixel and superpoint. To do this, the processor 110 first needs to obtain the superpoint feature output of each superpoint and the superpixel feature output of each superpixel.

[0041] Once the superpixels and superpoints are defined, the processor 110 uses the defined superpixels and superpoints to compute the original radar feature map f θ Let us denote the original image feature map h ω In particular, we can align the original image feature map h ω and the original radar feature map f θ Based on this, the processor 110 generates a segmented image feature map h ω and a segmented radar feature map f with dimensions N'×D. θ In other words, the dimensions of the image feature map output by the pre-trained image model 20 are determined.

number

number

[0042] The processor 110 processes each image I t For each superpixel in, we give the original image feature map h ω Each superpixel feature output based on the features in

number

number

[0043] The processor 110 calculates each radar point cloud P t For each superpoint in, we denote the original radar feature map f θ Output each superpoint feature based on the features in

number

number

number

[0044] Therefore, the output features of the kth matching superpixel-to-superpoint region are defined as

number

number

[0045] The processor 110 outputs the matched SuperPoint features.

number

number

number

number

[0046] The method 300 continues by training the first neural network based on the contrast loss (block 340). In particular, during each training cycle and / or training data pair, <P t ,I t After each batch of >, the processor 110 refines one or more components of the radar data processing model 30 based on the contrast loss L. The one or more components of the radar data processing model 30 that are refined include at least the encoder 40 of the radar data processing model 30. In at least some embodiments, during such a refinement process, model parameters (e.g., model coefficients, machine learning model weights, etc.) of the radar data processing model 30 are modified or updated based on the contrast loss L (e.g., using stochastic gradient descent, etc.). The contrast loss is calculated by multiplying the corresponding superpoint features by the contrast loss L.

number

number

number

[0047] In some embodiments, the parameters (e.g., kernel weights, model coefficients, etc.) of the pre-trained image model 20 are frozen during the pre-training phase while training the radar data processing model 30. In some embodiments, only the parameters of the encoder of the pre-trained image model 20 are frozen during the pre-training phase, and the parameters of the decoder of the pre-trained image model 20 are trained together with the encoder 40 of the radar data processing model 30.

[0048] In this manner, the radar data processing model 30 is pre-trained in a self-supervised manner to extract latent features from radar point clouds without requiring human annotation of the radar point clouds used in training. Instead, the output features from the pre-trained image model 20 essentially serve as soft labels for the training process.

[0049] 3, the method 200 continues by receiving a plurality of annotated radar point clouds having labels corresponding to radar data processing tasks (block 230). In particular, the processor 110 receives the plurality of annotated radar point clouds having labels corresponding to radar data processing tasks (block 230). <P t ,C t and / or the database 102 receives a plurality of annotated radar point clouds. <P t ,C t Each annotated radar point cloud stores the

number

[0050] In general, because radar data processing model 30, and in particular encoder 40 of radar data processing model 30, have already been trained in a self-supervised manner, the number of annotated radar point clouds in the plurality of annotated radar point clouds is small compared to the amount that would be required to train a conventional radar data processing model. In this manner, the annotated training data set can be constructed in a low-resource setting and at low cost by manually labeling radar point clouds.

[0051] The method 200 continues by further training the first neural network to perform radar data processing tasks in a fine-tuning phase based on a plurality of annotated radar point clouds (block 240). In particular, in the fine-tuning phase, <P t ,C t For each radar point cloud P t is provided as an input to the encoder 40 of the radar data processing model 30, which similarly generates the radar feature map f θ Each radar point cloud P t Generate a latent representation of the radar feature map f θ are provided to a decoder 50 of the radar data processing model 30, which outputs the final model output, e.g., the radar point cloud P t The processor 110 generates an object classification or semantic segmentation of the points in . The processor 110 outputs the final model output and the associated ground truth labels C t and determining a training loss based on a comparison with. In at least one embodiment, the training loss is a cross-entropy loss. Finally, the processor 110 refines the radar data processing model 30 based on the training loss.

[0052] It should be appreciated that after the pre-training phase, the radar data processing model 30 will already be sufficiently trained to extract useful latent representations of each radar point cloud as a result of the knowledge distillation process. However, the decoder 50 of the radar data processing model 30 must be further trained to perform the specified downstream tasks and to generate the desired final output. Thus, in at least one embodiment, during the fine-tuning phase, the parameters of the encoder 40 (e.g., kernel weights, model coefficients, etc.) are frozen and only the parameters of the decoder 50 are refined during the fine-tuning phase.

[0053] The associated ground truth label C t In some embodiments where x is a classification label, such as for a class of object detection, the processor 110 may calculate the training loss as a categorical cross-entropy loss:

number

[0054] Experimental Results The performance of the system was tested on a challenging object detection task to demonstrate the improvements provided by the training workflow 10. A prototype of the system was implemented and selected to test on autonomous driving data to ensure real-world performance.

[0055] The experiment was conducted based on the nuScenes dataset, which is the first dataset that contains aligned camera and radar sensors and sufficient radar measurements (18 features per point, including Doppler speed, RCS, depth, etc.) for autonomous driving. Compared to other datasets in the same domain, nuScenes has the largest sample size and the most comprehensive radar data, which is useful in the context of this specification because models generally based on LiDAR do not perform well, and the model needs to use different features specific to radar data to perform downstream tasks. The nuScenes dataset contains 700 scenes for training and 150 scenes for validation. In this experiment, the pre-trained model is trained using 700 training scenes, while the downstream model is trained using 600 scenes and evaluated based on the remaining 100 scenes as mini-validation to select parameters, and the performance of the entire pipeline is tested using 150 validation scenes.

[0056] Despite providing rich radar measurements such as RCS and Doppler velocity, the raw radar data for the nuScenes dataset is not ground truth labeled. This, on the other hand, reflects the difficulty of manually labeling each radar point. To carry out the experiments, ground truth labels were exploited from the aligned LiDAR data collected together with the camera and radar. LiDAR has an order of magnitude more points compared to radar, which is sufficient to find a close enough match for the radar points. Each radar point was transformed from the radar frame to the global ego frame and its coordinates were compared with the transformed LiDAR point on the ego frame. The closest point was selected and the LiDAR label was applied as the label of this radar point. To ensure that the matched radar and LiDAR points belong to the same object and to ensure that enough radar points are selected for the experiments, it was necessary to carefully select the distance threshold between the matched radar and LiDAR points. When calculating the distance between the radar point and the LiDAR point, it was noted that the depth range and resolution (distance to the front) at the radar point is much greater than the other two axes, and to best balance this fact, the relative distance is calculated using a compensation factor that multiplies the depth by 0.13.

[0057] Another challenge was choosing an appropriate relative distance, for which visual validation was performed on multiple keyframes, and a threshold of 1.5m was selected to best balance accuracy and availability of radar points. However, the radar and LiDAR points were mostly aligned, with most matched points being within 10cm of each other. In this data set, the LiDAR only has a range of 105m, compared to a depth range of 250m for the radar. As a result, radar points outside the LiDAR range were discarded as they did not have reference points within the distance threshold. 100 frames were manually checked to ensure that the labels annotated using this method were accurate.

[0058] Performing object detection on sparse radar point clouds is a challenging task. In our experiments, object detection was performed on three categories of targets used in NVRadarNet: cars, pedestrians, and bicycles.

[0059] In at least some embodiments, the backbone of the radar data processing model 30 described herein is a sparse residual U-Net architecture. A 3x3x3 kernel was applied for all sparse convolutions. For input, a sparse occupancy grid of 3D radar data is obtained, derived from voxels in a cylindrical coordinate system that is better adapted to the radar data. The radar data has input features of depth, RCS and Doppler velocity, which are fed into the radar data processing model 30, which outputs a size of 64 latent features to align with the image model. To balance the input features, the RCS was multiplied by 1000 and the Doppler velocity on each axis was multiplied by 10000. Self-supervised learning relies heavily on data augmentation. On the image side, two augmentation methods were used: random crop and random flip. On the point side, a random flip was applied around the forward-facing direction.

[0060] The neural network was trained using one GPU for 50 epochs using SGD with an initial learning rate of 0.05, momentum of 0.9, dampening of 0.1, and weight decay of 0.0001. A batch size of 32 and a dropout rate of 0.5 were used.

[0061] Although there is prior work in knowledge distillation for LiDAR point clouds, they perform poorly when applied to radar point clouds, mainly due to sparsity. A baseline was implemented as a comparison with our 3D pre-training approach, i.e., a fully connected neural network (FCN) for object detection tasks.

[0062] To compare training workflow 10 with a conventional CNN network using all the training data, a fully connected neural network (FCN) was constructed consisting of three linear layers followed by linear activation (ReLU) and dropout layers. The performance of this FCN method was compared with downstream results using training workflow 10. All baseline models were trained using one GPU using the same nuScenes training set and tested under the nuScenes validation set.

[0063] The performance of FCN and training workflow 10 on the nuScenes validation dataset for the object detection task is shown in Tables I and II.

[0064] Table I below shows the performance of the FCN network using valid radar points on the nuScenes dataset (100% training set). [Table 1]

[0065] Table II below shows the performance of the training workflow 10 using valid radar points on the nuScenes dataset. [Table 2]

[0066] As can be observed, the knowledge distillation process described herein outperforms conventional FCN approaches. Furthermore, the amount of annotated data required in downstream tasks is significantly reduced by using the knowledge distillation process described herein.

[0067] Embodiments within the scope of the present disclosure may include non-transitory computer-readable storage media or machine-readable media for carrying or having computer-executable instructions (also referred to as program instructions) or data structures stored therein. Such non-transitory computer-readable storage media or machine-readable media may be any available medium that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such non-transitory computer-readable storage media or machine-readable media may include RAM, ROM, EEPROM, CD-ROM, or other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to carry or store desired program code means in the form of computer-executable instructions or data structures. Combinations of the above should also be included within the scope of non-transitory computer-readable storage media or machine-readable media.

[0068] Computer-executable instructions include, for example, instructions and data that cause a general purpose computer, special purpose computer, or special purpose processing device to perform a particular function or group of functions. Computer-executable instructions also include program modules that are executed by a computer in stand-alone or networked environments. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of the program code means for executing steps of the methods disclosed herein. A particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps.

[0069] While the present disclosure has been illustrated and described in the drawings and the foregoing description, these drawings and the foregoing description should be regarded as illustrative in nature and not as restrictive in nature, it being understood that only preferred embodiments have been presented and that all changes, modifications and further applications that fall within the spirit of the disclosure are intended to be protected.

Claims

1. 1. A method for training a first neural network to perform a radar data processing task, the method comprising: receiving, by a processor, a plurality of training data pairs, each training data pair in the plurality of training data pairs including a respective image and a respective radar point cloud captured synchronously with respect to a same scene; training, by the processor, in a first phase based on the plurality of training data pairs, the first neural network to extract features from radar point clouds using a second neural network pre-trained to extract features from images; receiving, by the processor, a plurality of annotated radar point clouds having labels corresponding to radar data processing tasks; further training, by the processor, the first neural network to perform the radar data processing tasks in a second phase based on the plurality of annotated radar point clouds; A method comprising:

2. the first neural network includes an encoder followed by a decoder; training the first neural network in the first phase includes training the encoder of the first neural network based on the plurality of training data pairs using the second neural network; 2. The method of claim 1 , wherein training the first neural network in the second phase includes training the decoder of the first neural network based on the plurality of annotated radar point clouds.

3. The method of claim 2 , wherein parameters of the encoder of the first neural network are frozen while training the decoder of the first neural network in the second phase.

4. The method of claim 1 , wherein parameters of the second neural network are frozen while training the first neural network in the first phase.

5. Training the first neural network in the first phase includes: determining a first feature output based on each radar point cloud in the plurality of training data pairs using the first neural network; determining a second feature output based on each image in the plurality of training data pairs corresponding to the respective radar point cloud using the second neural network; refining the first neural network based on the first feature output and the second feature output; The method of claim 1 , comprising:

6. 6. The method of claim 5, wherein training the first neural network in the first phase includes determining a mapping between (i) points of a radar point cloud in the plurality of training data pairs and (ii) pixels of images in the plurality of training data pairs.

7. Training the first neural network in the first phase includes: determining a contrast loss based on the first feature output and the second feature output and based on the mapping; and refining the first neural network based on the contrast loss; and The method of claim 6, comprising:

8. Training the first neural network in the first phase includes: Segmenting the respective radar point clouds into superpoints, each superpoint including a subset of points from the respective radar point cloud; Segmenting each of the images into superpixels, each superpixel comprising a subset of pixels of the respective image; The method of claim 7, comprising:

9. Training the first neural network in the first phase includes: determining a respective superpoint feature output for each superpoint in the respective radar point cloud; determining a respective superpixel feature output for each superpixel in the respective image; determining the contrast loss based on the respective superpoint feature output of the respective superpoint in the respective radar point cloud and the respective superpixel feature output of the respective superpixel in the respective image; The method of claim 8 , comprising:

10. Training the first neural network in the first phase includes: matching each superpoint in the respective radar point cloud with a respective superpixel in the respective image; determining the contrast loss based on the respective superpoint feature output and the respective superpixel feature output of each matched superpoint and superpixel; 10. The method of claim 9, comprising:

11. determining the respective superpoint feature output further comprises determining the respective superpoint feature output as a weighted average of features in the first feature output for a point of the respective superpoint; The method of claim 9 , wherein the average is weighted depending on the average distance of each point to each other point within the respective superpoint.

12. 10. The method of claim 9, wherein determining the respective superpixel feature output further comprises determining the respective superpixel feature output as an average of features in the second feature output for pixels of the respective superpixel.

13. 9. The method of claim 8, wherein segmenting the respective images into superpixels further comprises defining each superpixel of the respective images by applying at least one of an image segmentation algorithm and a pixel clustering algorithm to the respective images.

14. Segmenting each of the radar point clouds into superpoints comprises: matching each point in the respective radar point cloud with a respective superpixel in the respective image; defining a respective superpoint of said respective radar point cloud as a group of radar points that match the same superpixel in said respective image; The method of claim 13, comprising:

15. 9. The method of claim 8, wherein segmenting the respective radar point clouds into superpoints further comprises defining each superpoint of the respective radar point cloud by applying a point clustering algorithm to the respective radar point cloud.

16. Segmenting each of the images into superpixels comprises: matching each superpixel in the respective image with a respective superpoint in the respective radar point cloud; defining a respective superpixel of said respective image as a group of pixels that match the same superpoint in said respective radar point cloud; The method of claim 15 further comprising:

17. Segmenting each of the images into superpixels comprises: - initially defining superpixels for each image by applying at least one of an image segmentation algorithm and a pixel clustering algorithm to the respective image; matching each initially defined superpixel in the respective image with a respective superpoint in the respective radar point cloud; defining each superpixel of each of the images as a group of initially defined superpixels that match the same superpoint in each of the radar point clouds; The method of claim 15 further comprising:

18. 16. The method of claim 15, wherein applying a point clustering algorithm to the respective radar point clouds further comprises generating a respective combined radar point cloud by combining the respective radar point cloud with at least one additional radar point cloud acquired at an immediately preceding or subsequent time compared to the time the respective radar point cloud was acquired.

19. 16. The method of claim 15, wherein applying a point clustering algorithm to the respective radar point clouds further comprises defining a respective superpoint of the respective radar point cloud by applying a point clustering algorithm to the respective combined radar point clouds.

20. A non-transitory computer readable medium having program instructions stored thereon that, when executed by a processor, receiving a plurality of training data pairs, each training data pair in the plurality of training data pairs including a respective image and a respective radar point cloud captured synchronously with respect to a same scene; training the first neural network to extract features from radar point clouds using a second neural network pre-trained to extract features from images in a first phase based on the plurality of training data pairs; receiving a plurality of annotated radar point clouds having labels corresponding to radar data processing tasks; further training the first neural network to perform the radar data processing tasks in a second phase based on the plurality of annotated radar point clouds; A non-transitory computer-readable medium for causing the processor to perform the method.

Images