System and method for camera-to-radar knowledge refinement

Abstract

Systems and methods for camera-to-radar knowledge refinement. A workflow for training a radar data processing model is described herein in which the requirements for annotated training data are reduced. The training workflow utilizes an image sensor (such as a camera) synchronized with a radar sensor to capture synchronized images and radar point cloud data. The training workflow utilizes a self-supervised knowledge refinement process to pre-train a radar data processing model on unannotated synchronized images and radar point cloud data using a pre-trained image model. Subsequently, the radar data processing model is finely tuned using the limited annotated radar point cloud data set, thereby greatly reducing the manual annotation burden.

Description

Technical Field

[0001] The apparatus and methods disclosed in this document relate to machine learning, and more particularly to training radar data processing models. Background Art

[0002] Unless otherwise indicated herein, the materials described in this section are not admitted to be prior art for inclusion in this section.

[0003] The tremendous development of millimeter wave radar has led to its widespread integration into commercial vehicles to collect accurate distance, direction, and speed information from objects in the surrounding environment, which is an important component of autonomous driving systems. These capabilities make radar a perfect candidate for object detection, which is one of the most critical tasks in autonomous driving solutions, especially in extreme conditions such as fog, rain, and other visual obstruction scenarios. Radar can achieve more accurate object detection and image segmentation by utilizing measurements of distance, direction, and speed relative to the target. Similarly, radar systems also have great potential to be adopted in a variety of other scenarios, such as indoor human detection, smart vacuum cleaners, or object detection for outdoor self-driving.

[0004] However, due to the sparsity limitation of radar points, existing radar-based methods require a lot of manual effort to annotate a large number of radar point clouds and build large datasets, making the use of existing radar-based methods quite expensive. What is needed is a method to reduce the burden of manual annotation on radar point clouds so that radar-based methods can be deployed at a lower cost. Summary of the invention

[0005] A method for training a first neural network to perform a radar data processing task is described herein. The method includes receiving, with a processor, a plurality of training data pairs, each respective training data pair in the plurality of training data pairs including a respective image and a respective radar point cloud, which are captured synchronously with each other of the same scene. The method further includes, in a first stage, training, with the processor, the first neural network based on the plurality of training data pairs to extract features from the radar point cloud using a second neural network pre-trained to extract features from the image. The method further includes receiving, with the processor, a plurality of annotated radar point clouds, the annotated radar point clouds having labels corresponding to the radar data processing task. The method further includes, in a second stage, further training, with the processor, the first neural network to perform the radar data processing task based on the plurality of annotated radar point clouds.

[0006] A non-transitory computer-readable medium is described herein. The non-transitory computer-readable medium stores program instructions that, when executed by a processor, cause the processor to receive a plurality of training data pairs, each corresponding training data pair in the plurality of training data pairs comprising a corresponding image and a corresponding radar point cloud, which are captured synchronously with each other of the same scene. The program instructions, when executed by the processor, further cause the processor to train a first neural network based on the plurality of training data pairs in a first stage to extract features from the radar point cloud using a second neural network pre-trained to extract features from the image. The program instructions, when executed by the processor, further cause the processor to receive a plurality of annotated radar point clouds, the annotated radar point clouds having labels corresponding to a radar data processing task. The program instructions, when executed by the processor, further cause the processor to further train the first neural network based on the plurality of annotated radar point clouds in a second stage to perform the radar data processing task. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The foregoing aspects and other features of the method are explained in the following description, taken in conjunction with the accompanying drawings.

[0008] Figure 1 A workflow for training radar data processing models is summarized, where the requirement for annotated training data is reduced.

[0009] Figure 2 An exemplary embodiment of a computing device that may be used to train a radar data processing model is shown.

[0010] Figure 3 A flow chart of a method for training a radar data processing model configured to perform a radar data processing task is shown.

[0011] Figure 4 A flow chart of a method for pre-training a radar data processing model in a pre-training phase is shown, the method utilizing superpixels and superpoints.

[0012] Figure 5A-Figure 5C An exemplary segmentation of an image into superpixels is shown. DETAILED DESCRIPTION

[0013] For the purpose of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the accompanying drawings and described in the following written specification. It should be understood that no limitation of the scope of the present disclosure is intended thereby. It should be further understood that the present disclosure includes any changes and modifications to the illustrated embodiments, and includes further applications of the principles of the present disclosure that would normally occur to a person skilled in the art to which the present disclosure belongs.

[0014] Overview

[0015] Figure 1 A workflow 10 for training a radar data processing model 30 is summarized, wherein the requirement for annotated training data is reduced. In the illustrated embodiment, the radar data processing model 30 is a U-net based model having an encoder 40 followed by a decoder 50. In particular, the encoder 40 is configured to extract features from a radar point cloud, and the decoder 50 is configured to perform downstream tasks on the radar point cloud based on the extracted features. The training workflow 10 utilizes an image sensor, such as a camera, which is synchronized with a radar sensor to capture synchronized image and radar point cloud data. The training workflow 10 utilizes a self-supervised knowledge distillation process to pre-train the radar data processing model 30 on unannotated synchronized image and radar point cloud data before fine-tuning with a limited set of annotated radar point cloud data, thereby greatly reducing the manual annotation burden.

[0016] The training workflow 10 consists of the following two stages: pre-training and fine-tuning. In the pre-training stage, the encoder 40 of the radar data processing model 30 is pre-trained using the pre-trained image model 20. In particular, pairs of aligned images and radar point clouds are input into the pre-trained image model 20 and the encoder 40 of the radar data processing model 30, respectively. The encoder 40 of the radar data processing model 30 is pre-trained by implementing matching between 2D pixel features extracted from aligned images and 3D point features extracted from radar point clouds. In the fine-tuning stage, the decoder 50 of the radar data processing model 30 is trained using a small annotated radar point cloud dataset with labels corresponding to a certain downstream task to be learned, such as target detection or semantic segmentation.

[0017] Therefore, the training workflow 10 advantageously transfers knowledge from existing camera image-based models and significantly reduces the manual effort required to label radar point cloud data while achieving comparable performance. In particular, the radar data processing model 30 is pre-trained on a large amount of unannotated data and only requires a small set of annotated data for fine-tuning later.

[0018] The training workflow 10 can be used for a wide range of applications, including autonomous driving, surveillance and public safety applications, intrusion detection, and vulnerable pedestrian detection. A system that captures co-located camera and radar data can be used for training, and after training, the radar can be operated independently using the proposed method to detect objects of interest and distances from the radar sensor. In autonomous vehicle applications, such radar-only operation can help avoid obstacles in low visibility scenarios. For surveillance and public safety applications, it can notify officers or security personnel when an intrusion is detected, or trigger an alarm.

[0019] Exemplary Hardware Embodiment

[0020] Figure 2 An exemplary embodiment of a computing device 100 is shown, which can be used to train a radar data processing model 30 to perform radar data processing tasks, such as target detection or semantic segmentation. Similarly, the computing device 100 can be used to operate a previously trained radar data processing model 30 to perform radar data processing tasks. The computing device 100 includes a processor 110, a memory 120, a display screen 130, a user interface 140, and at least one network communication module 150. It will be appreciated that the illustrated embodiment of the computing device 100 is only an exemplary embodiment, which only represents any of the various modes or configurations of a server, a desktop computer, a laptop computer, a mobile phone, a tablet computer, or any other computing device operable in the manner described herein. In some embodiments, the computing device 100 communicates with a database 102, which can be hosted by another device or stored in the memory 120 of the computing device 100 itself.

[0021] The processor 110 is configured to execute instructions to operate the computing device 100 to implement features, functions, characteristics, and / or the like as described herein. To this end, the processor 110 is operably connected to the memory 120, the display screen 130, and the network communication module 150. The processor 110 generally includes one or more processors, which can operate in parallel or otherwise cooperate with each other. Those of ordinary skill in the art will recognize that a "processor" includes any hardware system, hardware mechanism, or hardware component that processes data, signals, or other information. Therefore, the processor 110 may include a system with a central processing unit, a graphics processing unit, a plurality of processing units, a dedicated circuit for implementing a function, a programmable logic, or other processing system.

[0022] The memory 120 is configured to store data and program instructions that, when executed by the processor 110, enable the computing device 100 to perform the various operations described herein. As will be appreciated by one of ordinary skill in the art, the memory 120 may be any type of device capable of storing information accessible by the processor 110, such as a memory card, ROM, RAM, hard drive, magnetic disk, flash memory, or any of a variety of other computer-readable media used as data storage devices.

[0023] Display screen 130 may include any of various known types of displays (such as an LCD or OLED screen) configured to display a graphical user interface. User interface 140 may include various interfaces for operating computing device 100, such as buttons, switches, a keyboard or other keypad, a speaker, and a microphone. Alternatively or additionally, display screen 130 may include a touch screen configured to receive touch input from a user.

[0024] The network communication module 150 may include one or more transceivers, modems, processors, memories, oscillators, antennas, or other hardware conventionally included in communication modules to enable communication with various other devices. In particular, the network communication module 150 generally includes an Ethernet adapter or A module configured to enable communication with a wired or wireless network and / or a router (not shown) configured to enable communication with various other devices. Additionally, the network communication module 150 may include module (not shown), and one or more cellular modems configured to communicate with a wireless telephone network.

[0025] In at least some embodiments, the memory 120 stores program instructions for the radar data processing model 30, which are configured to perform radar data processing tasks once training is performed. In at least some embodiments, the database 102 stores a plurality of training data pairs, each corresponding training data pair including a corresponding image and a corresponding radar point cloud, which are synchronously captured relative to the scene. Additionally, in at least some embodiments, the database 102 stores a plurality of annotated radar point clouds with labels corresponding to the radar data processing tasks to be learned.

[0026] Methods for training radar data processing models

[0027] Various operations and processes for operating the computing device 100 to develop and train the radar data processing model 30 for performing radar data processing tasks are described below. In these descriptions, the statement that the method, processor, and / or system is performing a certain task or function refers to the controller or processor (e.g., the processor 110 of the computing device 100) executing programming instructions stored in a non-transitory computer-readable storage medium (e.g., the memory 120 of the computing device 100) operably connected to the controller or processor to manipulate data or operate one or more components in the computing device 100 or the database 102 to perform the task or function. Additionally, the steps of the method can be performed in any feasible time order, regardless of the order shown in the figure or the order in which the steps are described.

[0028] Figure 3A flow chart of a method 200 for training a radar data processing model configured to perform a radar data processing task is shown. A key insight and advantage of method 200 is that target-level representations can be transferred between models using a contrastive learning algorithm. In particular, through self-supervised knowledge distillation, the encoder 40 of the radar data processing model 30 learns to extract potential representations of radar point cloud data by leveraging existing image models and using aligned radar and image data. Additionally, method 200 advantageously utilizes superpixels and superpoints. In particular, method 200 groups points and pixels belonging to the same target and performs contrastive learning to refine knowledge from camera to radar, rather than performing per-point or per-pixel distillation. Method 200 uses contrastive loss to implement matching between pairs of corresponding 3D radar point features and 2D image pixel features.

[0029] The method 200 begins by receiving a plurality of training data pairs, each corresponding training data pair comprising an image and a radar point cloud captured synchronously with respect to a scene (block 210). In particular, the processor 110 receives and / or the database 102 stores the plurality of training data pairs. <P t , I t >. Each training data pair consists of a radar point cloud, labeled It is captured by a radar sensor, where t indicates the radar point cloud P t The time of capture, i is the radar point cloud P t The index of each specific radar data point in the radar point cloud P t and Q is the total number of points in the radar point cloud P t The dimensions of each radar data point in . Each training data pair also includes The corresponding image of I is captured synchronously by a camera or other 2D image sensor (e.g., a depth camera, an infrared camera, etc.) and the corresponding radar point cloud radar at the same time t, where M represents the image I t It should be appreciated that the plurality of training data pairs need not be manually annotated with any ground truth labels associated with the radar data processing task.

[0030] Method 200 continues as follows: In a pre-training phase, based on a plurality of training data pairs, a first neural network is trained to extract features from a radar point cloud using a second neural network that is pre-trained to extract features from an image (block 220). In particular, in the pre-training phase, processor 110 trains radar data processing model 30, denoted by f θ (·) to extract features from radar point clouds using a pre-trained image model 20, denoted as h ω (·), which is pre-trained to be based on multiple training data pairs <P t , I t> Extract features from the image. By combining the feature output f from the pre-trained image model 20 θ (P) and the corresponding image model output h ω (I) Alignment, training the radar data processing model 30 in a self-supervised manner. In at least one embodiment, during the training of the radar data processing model 30 in the pre-training stage, the parameters (eg, kernel weights, model coefficients, etc.) of the pre-trained image model 20 are frozen.

[0031] In some embodiments, the pre-trained image model 20 is a neural network that is configured to take 3D RGB image data as input and output a D-dimensional image feature map h ω , where M represents the number of pixels in the image. In some embodiments, the pre-trained image model 20 has an encoder-decoder architecture (not shown). In one embodiment, the encoder of the pre-trained image model 20 is a multi-layer convolutional network (e.g., output size 2048), and the decoder of the pre-trained image model 20 is a simple convolutional layer (e.g., output size 64). In one embodiment, the pre-trained image model 20 is a ResNet-50, which is pre-trained with MoCov2 on ImageNet in a self-supervised manner, followed by point-by-point projection using a linear layer and l2 normalization. However, in some embodiments, it should be appreciated that a variety of pre-existing models for extracting features from 2D image data can be utilized.

[0032] In some embodiments, the radar data processing model 30 is a neural network configured to take useful input from Q as input Q' and output a D-dimensional radar feature map f θ , where N is the total number of points in the radar point cloud, and Q is the dimension of each radar data point in the radar point cloud. In at least one embodiment, the radar data processing model 30 has a sparse residual U-Net architecture followed by point-wise projection with a linear layer and l2 normalization.

[0033] In the pre-training phase, for each training data pair <P t , I t >, the processor 110 converts the corresponding radar point cloud P t The encoder 40 of the radar data processing model 30 is provided as input and the corresponding image I t As input, it is provided to the pre-trained image model 20. Using the pre-trained image model 20, the processor 110 generates a t Determine the D-dimensional image feature map h ω (I). Similarly, using the encoder 40 of the radar data processing model 30, the processor 110 processes the radar point cloud P t Determine the D-dimensional radar feature map fθ (P). The processor 110 is based on the radar characteristic map f θ (P) and image feature map h ω (I) to determine the training loss. In at least one embodiment, the training loss is a contrast loss. Finally, the processor 110 improves the radar data processing model 30 (e.g., updates model coefficients, weights, parameters, etc.) based on the determined training loss.

[0034] In at least some embodiments, processor 110 determines a plurality of training data pairs <P t , I t >The radar point cloud P t points and multiple training data pairs <P t , I t >Image I t In particular, the first step in knowledge extraction is to align the radar point cloud with the camera image. Based on the known relative poses of the radar sensor and the camera sensor, the processor 110 aligns each radar point p i Projected onto the camera frame to obtain image I t More specifically, processor 110 constructs a map It is taken as each 3D radar point p i , and output image I t The processor 110 uses the mapping ε to p to calculate the training loss during the pre-training phase.

[0035] It should be appreciated that simply distilling from the knowledge on a pixel-to-radar point basis may not transfer useful knowledge to the radar data processing model 30, thus hindering the goal of using an existing pre-trained image model 20 to pre-train the radar data processing model 30 without any manual annotations. This is because the radar point cloud from a radar sensor has a different resolution than an image from a camera sensor. In addition, radar is able to recognize object-level information, rather than pixel-level information. Therefore, for the knowledge distillation process, it is advantageous to instead distill the object-level, rather than overly coarse scene-level knowledge, from the pre-trained image model 20 to the radar data processing model 30. To this end, in at least some embodiments, superpixels and superpoints are utilized to improve the knowledge distillation process.

[0036] Figure 4 FIG. 3 is a flow chart showing a method 300 for pre-training a radar data processing model in a pre-training phase, which utilizes superpixels and superpoints. It should be appreciated that the method 300 is Figure 3 The process of block 220 in method 200 is an exemplary embodiment of the present invention.

[0037] The method 300 begins by segmenting (i) the corresponding image from the corresponding training data pair into superpixels, and (ii) the corresponding radar point cloud from the corresponding training data pair into superpoints (block 310). In particular, in the pre-training phase, for each training data pair <P t , I t >, the processor 110 converts the corresponding image I t Split into The segmented image is determined by using a plurality of superpixels of M', where M' is the total number of superpixels. As used herein, a "superpixel" refers to a group of pixels or a subset of pixels within an image. In at least some embodiments, the pixels of the superpixel are arranged continuously with the image. In some embodiments, the superpixel includes visually similar pixels, for example, pixels having colors and / or brightness within a threshold similarity to each other. It should be appreciated that such superpixels generally provide more meaningful target-level information than individual pixels.

[0038] Similarly, in the pre-training phase, for each training data pair <P t , I t >, the processor 110 converts the corresponding radar point cloud P t Split into The segmented radar point cloud is determined by using a plurality of superpoints N', where N' is the total number of superpoints. As used herein, a "superpoint" refers to a group of points within a point cloud. In some embodiments, superpoints include points corresponding to the same target. In this manner, it should be appreciated that, similar to superpixels, superpoints generally provide more meaningful target-level information than individual points.

[0039] As further described below, superpixels and superpoints may be defined in various ways to enhance the knowledge distillation process. In a first method for defining superpixels and superpoints, processor 110 determines a segmented image and thereby extracts the superpixels and superpoints from the image by applying a machine learning model to the corresponding image. t To define superpixels, the machine learning model is such as an image segmentation algorithm, for example, the graph-based Felzenszwalb and Huttenlocher (FH) segmentation algorithm; or a pixel clustering algorithm, for example, the Simple Linear Iterative Clustering (SLIC) algorithm. Figures 5A-5C An exemplary segmentation of an image into superpixels is shown. In particular, Figure 5A A raw image 400 captured by a camera in an urban environment with cars and buildings is shown. Figure 5B A segmented image 410 generated by segmenting the image 400 into superpixels using the FH algorithm is shown. Figure 5C A segmented image 420 generated by segmenting the image 400 into superpixels using the SLIC algorithm is shown.

[0040] Next, the processor 110 determines the segmented radar point cloud and thus divides the corresponding radar point cloud P t Each point in the image I t The corresponding superpixels in p ) to define the superpoint. Again refer to Figure 5C , radar points (illustrated as small dots) are shown projected onto the segmented image 420. The processor 110 defines each superpoint of the segmented radar point cloud as a set of radar points that are mapped to the same superpixel in the corresponding segmented image. In other words, each superpoint Contained in the corresponding superpixel It should be appreciated that due to the sparsity of radar point clouds, some superpixels will not be mapped to any point from the radar point cloud. Therefore, in at least some embodiments, such superpixels corresponding to empty superpoints and / or not mapped to any point are not used to determine the training loss.

[0041] In a second method for defining superpixels and superpoints, in contrast to the first method, the processor 110 determines a segmented radar point cloud and thus classifies the corresponding radar point cloud P by applying a machine learning model (such as a point clustering algorithm, e.g., the DBScan algorithm) to the corresponding radar point cloud P. t To define the superpoint.

[0042] Next, the processor 110 initially defines superpixels of the segmented image by applying an image segmentation algorithm or a pixel clustering algorithm, as similarly discussed with respect to the first method. However, the processor 110 performs the above steps by comparing each initially defined superpixel with the corresponding radar point cloud P. t The processor 110 further merges these initially defined superpixels by matching the corresponding superpoints in the segmented radar point cloud. The processor 110 defines each final superpixel of the segmented image as a set of initially defined superpixels that map to the same superpoint in the corresponding segmented radar point cloud. Alternatively, the application of the image segmentation algorithm or the pixel clustering algorithm may be omitted in the second method, and instead, the processor 110 simply merges the corresponding image I t Each original pixel in the corresponding radar point cloud P t In other words, each final superpixel Contains the set of matched pixels (or superpixels as initially defined) that are mapped to the corresponding superpoints It should be appreciated that due to the sparsity of radar point clouds, some superpixels will not be mapped to any point from the radar point cloud. Therefore, in at least some embodiments, such superpixels corresponding to empty superpoints and / or not mapped to any point are not used to determine the training loss.

[0043] Using the second method for defining superpixels and superpoints, a clustering method is used to distinguish radar points belonging to different targets. When radar points that mainly belong to the same target are clustered into the same group, the aligned superpixels are combined together to form an object-level representation, thus overcoming the over-segmentation problem of the image. However, due to the sparsity of radar point clouds, the shapes of points are arbitrary, with varying densities, and some outliers introduce additional difficulties in choosing the right clustering method.

[0044] In the third method for defining superpixels and superpoints, similar to the second method, the processor 110 determines the segmented radar point cloud and thus defines superpoints by applying a machine learning model (such as a point clustering algorithm, e.g., the DBScan algorithm). However, the processor 110 does not apply the point clustering algorithm or other machine learning model only to the corresponding radar point cloud P. t , but by transforming the corresponding radar point cloud P t The combined radar point cloud is generated by combining with a predetermined number of adjacent radar point clouds, wherein the predetermined number of adjacent radar point clouds are obtained by combining with the corresponding radar point cloud P t The time t is compared to the time captured immediately before or immediately after, for example, the radar point cloud P immediately before t-1 And the radar point cloud P immediately following t+1 This combined radar point cloud helps overcome the sparsity problem of the radar point cloud. The processor 110 determines the segmented radar point cloud and thus defines superpoints by applying a machine learning model (such as a point clustering algorithm, e.g., DBScan algorithm or Hungarian method) to the combined radar point cloud. Next, the processor 110 defines superpixels of the segmented image using the same process discussed with respect to the second method.

[0045] Continue to refer Figure 4 , the method 300 continues by determining (i) an image feature map based on the corresponding image using the second neural network, and (ii) a radar feature map based on the corresponding radar point cloud using the first neural network (block 320). In particular, as similarly discussed above, in the pre-training phase, for each training data pair <P t , I t >, the processor 110 provides the corresponding radar point cloud P t As input to the encoder 40 of the radar data processing model 30, and providing the corresponding image I t As the input of the pre-trained image model 20. Using the pre-trained image model 20, the processor 110 t Determine the D-dimensional image feature map h ω (I). Similarly, using the encoder 40 of the radar data processing model 30, the processor 110 processes the radar point cloud Pt Determine the D-dimensional radar feature map f θ (P).

[0046] The method 300 continues by determining a contrast loss between the image feature map and the radar feature map on a superpixel-to-superpoint basis (block 330). θ And the image feature map h ω , and use the mapping ε p To determine the contrast loss. However, in order to provide a more useful contrast loss, the processor 110 determines the contrast loss on the basis of superpixels to superpoints. It should be appreciated that this superpixel-level contrast loss has several advantages. Superpixel-level contrast loss gives the same weight to all regions in the contrast loss, regardless of the radar point sampling density. Additionally, the superpixel-level contrast loss makes the contrast loss between the corresponding superpixel and superpoint small enough, but large enough to distinguish the feature from another superpixel in a different class in the same scene. Determining the contrast loss between the two feature maps on the basis of superpixels to superpoints helps to ensure that knowledge distillation is at the target level. Compared with pixel-level contrast learning, determining the contrast loss on the basis of superpixels to superpoints does not face the contrast problem between almost identical points within superpixels, which will produce a "missing" problem for self-supervised learning. In contrast, determining the contrast loss at the scene level (which typically compares the global representation of the entire radar point cloud with the entire corresponding camera image) is unable to distinguish differences at the target level and provides limited diversity because most scenes have targets of the same class, such as cars, pedestrians, etc.

[0047] Therefore, the processor 110 generates the radar characteristic map f based on the segmentation θ The superpoints (i.e., the corresponding superpoint feature output of each superpoint in the corresponding radar point cloud) and the segmented image feature map h ω The processor 110 matches each superpoint in the corresponding radar point cloud with the corresponding superpixel in the corresponding image and determines the contrast loss based on the matched superpoint feature output and superpixel feature output. In particular, in at least one embodiment, for a given camera image I c , the processor 110 calculates a contrast loss from each corresponding superpixel and superpoint. To do this, the processor 110 first needs to obtain the superpoint feature output of each superpoint and the superpixel feature output of each superpixel.

[0048] When the superpixels and superpoints are defined, the processor 110 may use the defined superpixels and superpoints to transform the original radar feature map f having the original dimension N×D into θ With the original image feature map h having the original dimension M×D ωIn particular, based on the original image feature map h ω and the original radar feature map f θ , the processor 110 determines a segmented image feature map h having dimensions M'×D ω and the segmented radar feature map f with dimension N'×D θ In other words, the image feature map output by the pre-trained image model 20 The dimension of is decomposed into a dimension of M×3→M′×D, and the radar characteristic map output by the encoder 40 is The dimension of is decomposed into a matrix having dimensions N×Q'→N'×D.

[0049] The processor 110 performs the following operations based on the original image feature map h ω The features of the pixels corresponding to the corresponding superpixels in the corresponding image I t Each superpixel in determines the corresponding superpixel feature output To determine the segmentation image feature map h ω In one embodiment, processor 110 uses average pooling to determine the superpixel feature output for each superpixel That is, as the average of the feature outputs of each pixel in the superpixel.

[0050] The processor 110 performs the following operations based on the original radar characteristic image f θ The features of the points corresponding to the corresponding superpoints are the corresponding radar point clouds P t Each superpoint in determines the corresponding superpoint feature output To determine the segmented radar feature map f θ However, due to the sparse nature of radar point clouds, simply performing average pooling to determine the feature output for each superpoint The spatial information between the points may not be fully utilized. Instead, in one embodiment, the processor 110 uses weighted average pooling to determine the super-point feature output of each super-point For example, a weighted average of the features of each point based on the average distance to other points within the superpoint, thereby giving higher weights to points that are closer to other points in the superpoint than to points that are farther away from other points in the superpoint.

[0051] Therefore, the output feature of the kth matching superpixel to superpoint region can be expressed as:

[0052]

[0053] Where C denotes the corresponding training data pair, and Indicate the corresponding matching superpixel and superpoint pairs within the training data.

[0054] The processor 110 outputs the matched super-point features And superpixel feature output Determining instance contrast loss In particular, in one embodiment, processor 110 determines the contrast loss as follows:

[0055]

[0056] where τ is the temperature, and P is the set of all superpixel superpoint matches with positive radar points such that In one embodiment, processor 110 only considers matches with at least one radar point and does not use additional empty points as negatives. In some embodiments, due to the sparsity of radar points, during the pre-training phase, processor 110 uses equation (3) to determine the contrastive loss with a batch of multiple shuffled scenes. to ensure that enough samples from all classes are involved.

[0057] Method 300 continues with: training the first neural network based on the contrastive loss (block 340). In particular, during each training cycle and / or during each batch of training data pairs <P t , I t >After that, the processor 110 performs the comparison based on the contrast loss To improve one or more components of the radar data processing model 30. The one or more components of the radar data processing model 30 that are improved include at least the encoder 40 of the radar data processing model 30. In at least some embodiments, during such improvement process, based on the contrast loss (e.g., using stochastic gradient descent, etc.), the model parameters (e.g., model coefficients, machine learning model weights, etc.) of the radar data processing model 30 are modified or updated. The contrast loss is the corresponding super-point feature and superpixel features matches between , and not with other similar superpixel features even on the same camera images.

[0058] In some embodiments, during the training of the radar data processing model 30 in the pre-training stage, the parameters (e.g., kernel weights, model coefficients, etc.) of the pre-trained image model 20 are frozen. In some embodiments, during the pre-training stage, only the parameters of the encoder of the pre-trained image model 20 are frozen, and the parameters of the decoder of the pre-trained image model 20 are trained together with the encoder 40 of the radar data processing model 30.

[0059] In this way, the radar data processing model 30 is pre-trained in a self-supervised manner to extract latent features from radar point clouds without the need for manual annotation of the radar point clouds used in training. Instead, the output features from the pre-trained image model 20 are essentially used as soft labels for the training process.

[0060] return Figure 3 , the method 200 continues as follows: receiving a plurality of annotated radar point clouds, the annotated radar point clouds having labels corresponding to radar data processing tasks (block 230). In particular, the processor 110 receives and / or the database 102 stores a plurality of annotated radar point clouds <P t , C t >. Each annotated radar point cloud includes radar point clouds of the same form as discussed above, denoted as It is captured by a radar sensor, where t indicates the time when the radar point cloud was captured, i is the index of each specific radar data point in the radar point cloud, N is the total number of points in the radar point cloud, and Q is the dimension of each radar data point in the radar point cloud. Each annotated radar point cloud further includes an associated ground truth label C t , where label c i The form of corresponds to the specific downstream task for which the radar data processing model 30 is trained, such as target detection or semantic segmentation. In one embodiment, each annotated radar point cloud P t Each radar point C includes a corresponding classification label indicating, for example, whether the point corresponds to an object of a specific class or indicates a segmentation index, such that the associated ground truth label C t Take Form C t =(c i ) i＝1，2，...，n , where c i is the point p i In P t The classification labels in .

[0061] In general, because the radar data processing model 30 (and in particular the encoder 40 of the radar data processing model 30) has been pre-trained in a self-supervised manner, the number of annotated radar point clouds in the plurality of annotated radar point clouds is small compared to the number that would be required to train a conventional radar data processing model. In this way, an annotated training dataset can be constructed by manually labeling radar point clouds at a lower resource setting and at a lower cost.

[0062] The method 200 continues as follows: In a fine-tuning phase, the first neural network is further trained to perform a radar data processing task based on a plurality of annotated radar point clouds (block 240). In particular, in the fine-tuning phase, for each annotated radar point cloud <P t , Ct >, the processor 110 converts the corresponding radar point cloud P t The encoder 40 provided as input to the radar data processing model 30 uses the radar characteristic map f θ The corresponding radar point cloud P is generated in the form of t The potential representation of is similarly discussed above. The radar signature f θ is provided to the decoder 50 of the radar data processing model 30, which generates the final model output, such as the radar point cloud P t The processor 110 performs object classification or semantic segmentation of the points in the image based on the final model output and the associated ground truth label C. t The training loss is determined by comparing the training loss. In at least one embodiment, the training loss is a cross entropy loss. Finally, the processor 110 improves the radar data processing model 30 based on the training loss.

[0063] It should be appreciated that after the pre-training phase, the radar data processing model 30 will have been well trained to extract useful latent representations of the corresponding radar point cloud as a result of the knowledge distillation process. However, the decoder 50 of the radar data processing model 30 must still be trained to perform the specified downstream tasks and generate the desired final output. Therefore, in at least one embodiment, the parameters of the encoder 40 (e.g., kernel weights, model coefficients, etc.) are frozen during the fine-tuning phase, and only the parameters of the decoder 50 are improved during the fine-tuning phase.

[0064] In some embodiments, where the associated ground truth label C t For categorical labels, such as for object detection classes, processor 110 determines the training loss as a categorical cross entropy loss as follows:

[0065]

[0066] Wherein, if the class label c is the correct classification of the radar point, y is a binary indicator (0 or 1), and p is the predicted probability of the class of the radar point. Similar to the pre-training stage, in at least some embodiments, the processor 110 determines the cross entropy loss θ for a batch of multiple shuffled frames using equation (4) to ensure that sufficient samples from all classes are involved.

[0067] Experimental Results

[0068] To demonstrate the improvements provided by the training workflow 10, the performance of the system was tested on a challenging object detection task. To ensure real-world performance, a prototype of the system was implemented and selected for testing on autonomous driving data.

[0069] Experiments are conducted on the nuScenes dataset, which is the first dataset containing aligned camera and radar sensors and sufficient radar measurements for autonomous driving (18 features per point, including Doppler velocity, RCS, depth, etc.). Compared to other datasets in the same field, nuScenes has the largest sample size and the most comprehensive radar data, which is useful because the model needs to use different features specific to radar data for downstream tasks, as usually lidar-based models do not work well in this context. The nuScenes dataset contains 700 scenes for training and 150 scenes for validation. In this experiment, the pre-trained model is trained with 700 training scenes, while the downstream model is trained with 600 scenes and evaluated on the remaining 100 scenes as a small validation for selecting parameters, and the performance of the entire pipeline is tested with 150 validation scenes.

[0070] Despite the availability of rich radar measurements (such as RCS and Doppler velocity), the raw radar data on the nuScenes dataset does not come with ground truth labels, which in turn reflects the difficulty of manually labeling each radar point. For the experiments, the ground truth labels leveraged aligned lidar data collected alongside the camera and radar. Lidar has more than an order of magnitude more points than radar, making it sufficient to find sufficiently close radar point matches. Each radar point is transformed from the radar frame to the global ego frame and its coordinates are compared with the transformed lidar point on the ego frame. The closest point is selected and the lidar label is applied as the label for this radar point. In order to guarantee that the matched radar and lidar points belong to the same object and to ensure that enough radar points are selected for our experiments, it is necessary to carefully choose the distance threshold between the matched radar and lidar points. Note that when calculating the distance between radar and lidar points, the range and resolution of the depth on the radar points (distance to the front) is significantly larger than the two other axes, and to best balance this fact, a compensation factor of 0.13 is multiplied by the depth to calculate the relative distance.

[0071] Another challenge is choosing an appropriate relative distance, so a 1.5m threshold was chosen to best balance accuracy and availability of radar points when visually verified on multiple keyframes. However, the radar and lidar points are mostly aligned, with most matching points within 10cm of each other. In this dataset, the lidar only has a range of 105m, compared to the radar’s depth range of 250m. As a result, radar points outside the lidar range are discarded because they have no reference points within the distance threshold. 100 frames were manually checked to ensure that the labels annotated using this method were accurate.

[0072] Performing object detection on sparse radar point clouds is a challenging task. In the experiments, object detection is performed on the three categories used in NVRadarNet: cars, pedestrians, and bicycles.

[0073] In at least some embodiments, the backbone of the radar data processing model 30 described herein is a sparse residual U-Net architecture. A 3×3×3 kernel is applied to all sparse convolutions. As for the input, it takes a sparse occupancy grid of 3D radar data, obtained from voxels in cylindrical coordinates that are more suitable for radar data. The radar data has input features of depth, RCS, and Doppler velocity, which are fed into the radar data processing model 30 and output the size of 64 potential features to align with the image model. To balance the input features, the RCS is multiplied by 1000 and the Doppler velocity on each axis is multiplied by 10000. Self-supervised learning relies heavily on data enhancement. On the image side, two enhancement methods are used: random cropping and random flipping. On the point side, random flipping is applied around the forward-facing direction.

[0074] The neural network was trained for 50 epochs using 1 GPU, using SGD with an initial learning rate of 0.05, a momentum of 0.9, a damping of 0.1, and a weight decay of 0.0001. The batch size used was 32 and the dropout rate was 0.5.

[0075] Although there are prior works on knowledge distillation on LiDAR point clouds, they perform poorly when applied to LiDAR point clouds mainly due to sparsity. A baseline is implemented as a comparison with our 3D pre-training method: a fully connected neural network (FCN) for the object detection task.

[0076] To compare the training workflow 10 with a conventional CNN network using all the training data, a fully connected neural network (FCN) was built consisting of three linear layers followed by a linear activation layer (ReLU) and a dropout layer. The performance of this FCN approach was compared with downstream results using training workflow 10. All baseline models were trained using the same nuScenes training set, using 1 GPU, and tested on the nuScenes validation set.

[0077] The performance of FCN and training workflow 10 on the nuScenes validation dataset for the object detection task is shown in Tables I and II.

[0078] Table I below shows the performance of the FCN network using valid radar points on the nuScenes dataset (100% training set):

[0079] FCN QUR wxya 28.3 72.1

[0080] Table II below shows the performance of the training workflow 10 using valid radar points on the nuScenes dataset.

[0081] Training Workflow10 5％ 10％ 20％ 100％ QUR 32.5 32.3 33.5 35.9 wxya 74.7 74.9 75.2 76.2

[0082] As can be observed, the knowledge distillation process described in this article performs better than conventional FCN methods. Additionally, using the knowledge distillation process described in this article, the amount of annotated data required in downstream tasks is greatly reduced.

[0083] Embodiments within the scope of the present disclosure may also include non-transitory computer-readable storage media or machine-readable media for carrying or storing computer-executable instructions (also referred to as program instructions) or data structures thereon. Such non-transitory computer-readable storage media or machine-readable media may be any available medium that can be accessed by a general-purpose or special-purpose computer. As an example and not limitation, such non-transitory computer-readable storage media or machine-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that may be used to carry or store computer-executable instructions or data structure forms of desired program code components. The above combination should also be included within the scope of non-transitory computer-readable storage media or machine-readable media.

[0084] Computer executable instructions include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a specific function or group of functions. Computer executable instructions also include program modules that are executed by computers in an independent environment or a network environment. In general, program modules include routines, programs, objects, components, and data structures, etc., which perform specific tasks or implement specific abstract data types. Computer executable instructions, associated data structures, and program modules represent examples of program code components for performing the steps of the methods disclosed herein. A specific sequence of such executable instructions or associated data structures represents an example of corresponding actions for implementing the functions described in such steps.

[0085] While the present disclosure has been illustrated and described in the drawings and foregoing description, it should be properly considered as illustrative rather than restrictive. It should be understood that only preferred embodiments have been presented and all changes, modifications and further applications within the spirit of the present disclosure are desired to be protected.

Claims

1. A method for training a first neural network to perform a radar data processing task, the method comprising: receiving, with a processor, a plurality of training data pairs, each respective training data pair of the plurality of training data pairs comprising a respective image and a respective radar point cloud, the respective image and the respective radar point cloud being captured synchronously with each other of the same scene; In a first stage, based on the plurality of training data pairs, training with a processor a first neural network to extract features from a radar point cloud using a second neural network pre-trained to extract features from an image; receiving, with a processor, a plurality of annotated radar point clouds having labels corresponding to radar data processing tasks; and In a second stage, the first neural network is further trained with a processor to perform radar data processing tasks based on the plurality of annotated radar point clouds.

2. The method according to claim 1, wherein: The first neural network includes an encoder followed by a decoder, Training the first neural network in the first stage includes training an encoder of the first neural network using the second neural network and based on the plurality of training data pairs; and Training the first neural network in the second stage includes training a decoder of the first neural network based on the plurality of annotated radar point clouds.

3. The method of claim 2, wherein during training of the decoder of the first neural network in the second stage, parameters of the encoder of the first neural network are frozen.

4. The method of claim 1, wherein during the training of the first neural network in the first stage, the parameters of the second neural network are frozen.

5. The method according to claim 1, wherein training the first neural network in the first stage comprises: using a first neural network, determining a first feature output based on corresponding radar point clouds in the plurality of training data pairs; determining, using a second neural network, a second feature output based on respective images in the plurality of training data pairs corresponding to respective radar point clouds; and The first neural network is improved based on the first feature output and the second feature output.

6. The method according to claim 5, wherein training the first neural network in the first stage comprises: A mapping between (i) points of the radar point cloud in the plurality of training data pairs and (ii) pixels of the images in the plurality of training data pairs is determined.

7. The method according to claim 6, wherein training the first neural network in the first stage comprises: determining a contrastive loss based on the first feature output and the second feature output and based on the mapping; and Improving the first neural network based on contrastive loss.

8. The method according to claim 7, wherein training the first neural network in the first stage comprises: segmenting the corresponding radar point cloud into superpoints, each superpoint comprising a subset of points from the corresponding radar point cloud; and The corresponding image is segmented into superpixels, each superpixel comprising a subset of pixels from the corresponding image.

9. The method according to claim 8, wherein training the first neural network in the first stage comprises: determining a corresponding super-point feature output for each corresponding super-point in the corresponding radar point cloud; determining a corresponding superpixel feature output for each corresponding superpixel in the corresponding image; The contrastive loss is determined based on the corresponding super-point feature output of each super-point in the corresponding radar point cloud and the corresponding super-pixel feature output of each super-pixel in the corresponding image.

10. The method according to claim 9, wherein training the first neural network in the first stage comprises: Matching each superpoint in the corresponding radar point cloud with the corresponding superpixel in the corresponding image; and A contrastive loss is determined based on the corresponding superpoint feature output and the corresponding superpixel feature output of each matched superpoint and superpixel.

11. The method according to claim 9, wherein determining the corresponding super-point feature output further comprises: The corresponding superpoint feature output is determined as a weighted average of the features in the first feature outputs for the points of the corresponding superpoint, the average being weighted depending on the average distance of each point from every other point in the corresponding superpoint.

12. The method according to claim 9, wherein determining the corresponding superpixel feature output further comprises: The corresponding superpixel feature output is determined as the average of the features in the second feature outputs of the pixels of the corresponding superpixel.

13. The method according to claim 8, segmenting the corresponding image into superpixels further comprises: Each superpixel of the corresponding image is defined by applying at least one of an image segmentation algorithm and a pixel clustering algorithm to the corresponding image.

14. The method according to claim 13, segmenting the corresponding radar point cloud into super points further comprises: Match each point in the corresponding radar point cloud with the corresponding superpixel in the corresponding image; and Each superpoint of the corresponding radar point cloud is defined as a set of radar points that match the same superpixel in the corresponding image.

15. The method according to claim 8, segmenting the corresponding radar point cloud into super points further comprises: Each superpoint of the corresponding radar point cloud is defined by applying a point clustering algorithm to the corresponding radar point cloud.

16. The method according to claim 15, segmenting the corresponding image into superpixels further comprises: Matching each superpixel in the corresponding image with the corresponding superpoint in the corresponding radar point cloud; and Each superpixel of the corresponding image is defined as a set of pixels that match the same superpoint in the corresponding radar point cloud.

17. The method according to claim 15, segmenting the corresponding image into superpixels further comprises: Initially defining superpixels of the corresponding image by applying at least one of an image segmentation algorithm and a pixel clustering algorithm to the corresponding image; matching each initially defined superpixel in the corresponding image with a corresponding superpoint in the corresponding radar point cloud; and Each superpixel of the corresponding image is defined as a set of initially defined superpixels that match the same superpoint in the corresponding radar point cloud.

18. The method of claim 15, wherein applying a point clustering algorithm to the corresponding radar point cloud further comprises: A respective combined radar point cloud is generated by combining the respective radar point cloud with at least one further radar point cloud that was captured at a time immediately before or immediately after the time at which the respective radar point cloud was captured.

19. The method of claim 15, wherein applying a point clustering algorithm to the corresponding radar point cloud further comprises: Each superpoint of the corresponding radar point cloud is defined by applying a point clustering algorithm to the corresponding combined radar point cloud.

20. A non-transitory computer readable medium storing program instructions which, when executed by a processor, cause the processor to receiving a plurality of training data pairs, each respective training data pair of the plurality of training data pairs comprising a respective image and a respective radar point cloud, the respective image and the respective radar point cloud being captured synchronously with each other of the same scene; In a first stage, based on the plurality of training data pairs, a first neural network is trained to extract features from a radar point cloud using a second neural network pre-trained to extract features from an image; receiving a plurality of annotated radar point clouds having labels corresponding to radar data processing tasks; and In a second stage, the first neural network is further trained to perform a radar data processing task based on the plurality of annotated radar point clouds.

Images