Upsampling radar via multimodal neural fields

Abstract

Synchronized camera and radar images are captured from a scene. A shared spatial representation of the scene is generated by encoding spatial features into a spatial hash table of a shared geometry encoder. The shared representation is decoded using a geometry decoder to produce camera occupancy and radar occupancy values. Camera density and radar density values ​​are generated by applying a density decoding function to the spatial representation. Predicted camera images are rendered from the camera density and reflectivity values ​​using a volumetric rendering process. Predicted radar images are rendered from the radar density and reflectivity values ​​using a radar-specific volumetric rendering process. The shared geometry encoder is based on a multimodal loss function.

Images