System and method for context aware two-dimensional object annotations via native labelling and cross-modal transfers

Abstract

A method for identifying and annotating at least one object in one or more input scenes includes the following steps. First, the input scenes containing the at least one object are received and stored. Then, a three-dimensional, 3D model is generated from these scenes, incorporating the at least one object. Using an artificial intelligence, AI-based system, the objects are identified in the 3D model, including their type and each instance in the 3D model. The identified objects in the 3D model are then projected onto the input scenes to create two-dimensional, 2D annotations on each instance of the identified objects. An occlusion score is determined for each instance of the annotated objects in each of the input scenes. Finally, the input scenes with the annotated objects having zero or minimal occlusion scores and that provide distinct views of the annotated objects are selected and presented to users for further analysis.