AI-driven benchmark ground value generation for object detection and tracking on image sequences

Abstract

An AI-driven benchmark ground truth generation is provided for object detection and tracking on image sequences. The storage device maintains raw image data of a video with frame sequences, as well as frame annotations indicating objects identified in the corresponding frames. The processor, based on the annotations of keyframes in the raw image data, determines the task type and proxy type of the keyframes; receives annotations of objects identified in the keyframes of the raw image data according to the keyframe task type and proxy type; selects review keyframes based on the confidence level of the keyframe annotations; based on the annotations of intermediate frames in the raw image data, determines the task type and proxy type of the intermediate frames; and receives annotations of objects identified in the intermediate frames of the raw image data according to the intermediate frame task type and proxy type.

Description

Technical Field

[0001] This disclosure relates to aspects of artificial intelligence (AI) driven ground truth generation for object detection and tracking on image sequences. Background Technology

[0002] Compared to annotation prototypes used in public datasets, benchmark ground truth used in safety-critical systems typically requires high accuracy in geometric annotation and complex attributes. For example, pedestrian annotation prototypes used in public datasets require loose bounding boxes to cover the visible portion of a pedestrian. However, annotations for safety-critical systems often require pixel-accurate estimated bounding boxes along with body centerlines and additional attributes such as body pose and head angle. Due to this complexity and verbose requirements, along with the wide variety of scenarios, it takes too long for human annotators to understand all the requirements needed for the annotation task. This, coupled with the high learning curve for understanding the requirements, hinders the outward expansion of the number of annotators. Furthermore, the high cost of purely human annotation is a barrier to generating large amounts of annotated data, which is a prerequisite for data-driven machine learning algorithms such as deep learning. Summary of the Invention

[0003] In one or more illustrative examples, a system for generating high-precision benchmark ground truth data for human-machine collaboration in object identification, localization, and tracking in an image sequence includes: a user interface; a storage device configured to maintain raw image data including video with a frame sequence, and frame annotations indicating objects identified in the respective frames; and a processor communicating with the storage device and the user interface. The processor is programmed to: determine the task type and proxy type of keyframes for annotations of keyframes in the raw image data; receive annotations of objects identified in keyframes of the raw image data based on the keyframe task type and keyframe proxy type; select review keyframes based on the confidence level of the keyframe annotations; determine the task type and proxy type of intermediate frames for annotations of intermediate frames in the raw image data; and receive annotations of objects identified in intermediate frames of the raw image data based on the intermediate frame task type and intermediate frame proxy type.

[0004] In one or more illustrative examples, a method for generating high-precision benchmark ground truth data for human-computer collaboration in object identification, localization, and tracking in an image sequence includes: maintaining raw image data comprising a video with a frame sequence, and frame annotations indicating aspects of objects identified in the respective frames, said objects including one or more of pedestrians, cyclists, animals, vehicles, animals in an indoor environment, and moving objects, said annotations including one or more of the geometry around the object, the centerline of the object, or the direction of travel of the object; annotations for keyframes of the raw image data; determining the task type and agent type of the keyframes, said task type including purely human. The process involves: receiving annotations for objects identified in keyframes of the original image data, based on the keyframe task type, the AI ​​interaction task type, or the human task with pre-annotated machine learning data; selecting keyframes for review based on the confidence level of the annotations; determining the task type and proxy type of intermediate frames for annotations of the original image data; and receiving annotations for objects identified in intermediate frames of the original image data, based on the intermediate frame task type and proxy type.

[0005] In one or more illustrative examples, a computer-readable medium includes instructions that, when executed by a processor, cause the processor to maintain raw image data comprising a video having a sequence of frames, and frame annotations indicating aspects of objects identified in the respective frames, the objects including one or more of pedestrians, cyclists, animals, vehicles, animals in an indoor environment, and moving objects, the annotations including one or more of the geometry around the object, the centerline of the object, or the direction of travel of the object; annotation of keyframes for the raw image data; determining a task type and a proxy type for the keyframes, the task type including one of a purely human annotation task type, an AI interaction task type, or a human task with machine learning pre-annotation, the proxy type including those with average annotation skill. The process involves: receiving annotations of objects identified in keyframes of the original image data, based on the keyframe task type and keyframe agent type; selecting review keyframes based on the confidence level of the annotations, the confidence level being based on one or more of the following: (i) the performance of the worker performing the annotation task, (ii) the worker's overall performance across multiple annotation tasks, (iii) the prediction score determined based on the machine identification of the annotations, or (iv) an analysis of the image quality of the original image data; determining the task type and agent type of intermediate frames for annotations of intermediate frames of the original image data; and receiving annotations of objects identified in intermediate frames of the original image data, based on the intermediate frame task type and intermediate frame agent type. Attached Figure Description

[0006] Figure 1 The illustration shows an example annotation system for capturing and annotating image data;

[0007] Figure 2 This illustration shows an example of a data diagram used for annotation of image data;

[0008] Figure 3 The diagram illustrates an example workflow for the annotation task;

[0009] Figure 4 An example of a qualification assessment task is illustrated;

[0010] Figure 5 The diagram illustrates an example of a user interface for performing manual annotations;

[0011] Figure 6 An example of another aspect of the user interface for performing manual annotations is illustrated;

[0012] Figure 7 An example of pedestrian direction annotation is shown in the illustration;

[0013] Figure 8The illustration shows an example of pedestrian ID matching;

[0014] Figure 9 The illustration shows an example of a frame of raw image data related to pedestrian ID matching;

[0015] Figure 10 The illustration shows an example of a manually annotated user interface for performing AI assistance;

[0016] Figure 11 The diagram illustrates the architecture of an AI-based annotator;

[0017] Figure 12 The illustration shows an example of the process used to extract image patches for training;

[0018] Figure 13 An example of the final review task for a set of frames is illustrated;

[0019] Figure 14 The diagram illustrates a sample workflow for two-level review;

[0020] Figure 15 The illustration shows an example of review questions used to perform the final review; and

[0021] Figure 16 The illustration shows an example of a process for breaking down and executing annotation tasks into multiple tasks. Detailed Implementation

[0022] Embodiments of this disclosure are described herein. However, it will be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The figures are not necessarily to scale; some features may be enlarged or minimized to show details of particular components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to adopt the embodiments in various ways. As will be understood by those skilled in the art, various features illustrated and described with reference to any of the figures may be combined with features illustrated in one or more other figures to produce embodiments not explicitly illustrated or described. The combinations of illustrated features provide representative embodiments for typical applications. However, various combinations and modifications of features consistent with the teachings of this disclosure may be desired for particular applications.

[0023] This disclosure relates to systems and methods for generating high-precision benchmark ground truth data for human-machine collaboration to perform object detection / localization / tracking tasks on large-scale image sequences. Target objects for annotation include pedestrians, cyclists, animals, various types of vehicles in outdoor environments, and people, animals, and any moving objects in indoor environments. The disclosed method enables the annotation task to be decomposed into multiple tasks automated in a workflow, and each task to be dynamically assigned to one or more human annotators or machines to efficiently deliver benchmark ground truth. The benchmark ground truth generated from the annotation process is used to retrain the machine learning model used in the annotation process to improve machine predictions over time. The annotation process is divided into two main steps: keyframe annotation and intermediate frame annotation, which have different tasks in the workflow but share similar tasks.

[0024] The machine learning model used for benchmark truth generation (optionally with humans in the loop) can be used for other purposes, such as predictions for object recognition services.

[0025] To address the scalability and efficiency of annotation tasks, the system in this disclosure decomposes complex annotation tasks into multiple micro / machine tasks. Each micro-task is designed to be performed by any worker that has passed basic training and qualification without having memorized all the requirements. The machine tasks are designed to apply cutting-edge machine learning models to make the annotation process efficient. To further improve efficiency over time, machine learning models from previous annotations are used as is or after appropriate retraining (e.g., applying transfer learning). Depending on the characteristics of the images collected from various cameras, mounting locations, and deployment environments, machine learning models from previous annotations are used as is or after appropriate retraining (e.g., applying transfer learning).

[0026] As explained in detail herein, this disclosure provides for human-machine collaboration in annotation of large-scale data. Efficiency in large-scale data annotation can be provided by integrating machine learning models and humans in the annotation process loop and by improving machine predictions over time using data from previous batches. Manual / human annotation can be time-consuming and expensive. Therefore, this disclosure provides systems and methods for reducing human annotation effort by increasing the number of accurate machine annotations that do not require geometric annotation adjustments.

[0027] Additionally, the described systems and methods offer a reduction in cognitive load in complex annotation tasks. Indeed, for novice human annotators, learning complex annotation requirements to be able to annotate an entire video without errors can be extremely time-consuming. The proposed systems and methods reduce the learning time for human annotators by dividing the entire annotation work into multiple machine tasks that can be performed by many people. Therefore, the systems and methods are scalable for rapidly recruiting and training human annotators.

[0028] Regarding machine and interactive annotation user interfaces, machine learning models are used to annotate tasks within workflows, either automatically generating annotations via an interactive UI or collaborating with human annotators. Regarding quality control in crowdsourced tasks, quality control mechanisms are embedded in the design of tasks and workflows.

[0029] As described herein, efficient and scalable benchmark ground truth generation systems and methods produce high-precision (pixel-level accuracy) annotations for developing object detection / localization and object tracking. This disclosure provides systems and methods for generating high-precision benchmark ground truth data for human-machine collaboration to perform object detection / localization / tracking tasks on large-scale image sequences. As some examples, objects used for annotation include pedestrians, cyclists, animals, various types of vehicles in outdoor environments, and people, animals, and any moving objects in indoor environments.

[0030] Figure 1 An example annotation system 100 for capturing and annotating image data 132 is illustrated. Annotation system 100 includes a server 102 that hosts an annotation web application 124 accessible to client device 104 via network 122. Server 102 includes a processor 106 operatively connected to storage device 110 and network device 118. Server 102 further includes an image data input source 130 for receiving image data 132. Client device 104 includes a processor 108 operatively connected to storage device 112, a display device 114, a human-machine interface (HMI) control 116, and network device 120. It should be noted that the example annotation system 100 is an example, and other systems 100 can be used. For example, although only one client device 104 is shown, systems 100 including multiple client devices 104 are also contemplated. As another possibility, although the example implementation is shown as a web-based application, alternative systems can be implemented as standalone systems or client-server systems with thick client software.

[0031] Each of the processor 106 of server 102 and the processor 108 of client device 104 may include one or more integrated circuits that implement the functionality of a central processing unit (CPU) and / or a graphics processing unit (GPU). In some examples, processors 106, 108 are system-on-a-chip (SoC) integrating the functionality of a CPU and a GPU. The SoC may optionally include other components, such as, for example, including storage device 110 and network device 118 or 120 into a single integrated device. In other examples, the CPU and GPU are connected to each other via peripheral connectivity devices such as PCI express or other suitable peripheral data connections. In one example, the CPU is a commercially available central processing device implementing an instruction set such as one of the x86, ARM, Power, or MIPS instruction set families.

[0032] Regardless of the specific circumstances, during operation, processors 106 and 108 execute stored program instructions retrieved from storage devices 110 and 112, respectively. Accordingly, the stored program instructions include software that controls the operation of processors 106 and 108 to perform the operations described herein. Storage devices 110 and 112 may include both non-volatile memory and volatile memory devices. Non-volatile memory includes solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when annotation system 100 is deactivated or loses power. Volatile memory includes static and dynamic random access memory (RAM) that stores program instructions and data during operation of annotation system 100.

[0033] The GPU of client device 104 may include hardware and software for displaying at least two-dimensional (2D) and optionally three-dimensional (3D) graphics to the client's display device 114. Display device 114 may include an electronic display screen, projector, printer, or any other suitable device for reproducing the graphics display. In some examples, the processor 108 of client 104 uses the hardware functionality in the GPU to execute software programs to accelerate the execution of machine learning or other computational operations described herein.

[0034] The HMI control 116 of client 104 may include any of a variety of devices that enable client device 104 of annotation system 100 to receive control input from workers or other users. Examples of suitable input devices for receiving human interface input may include keyboards, mice, trackballs, touchscreens, voice input devices, graphic tablets, etc.

[0035] Network devices 118 and 120 may each include any of a variety of devices that enable server 102 and client device 104 to send and / or receive data from external devices via network 122. Examples of suitable network devices 118 and 120 include network adapters or peripheral interconnect devices that receive data from another computer or external data storage device, which can be useful for receiving large datasets in an efficient manner.

[0036] The annotation web application 124 is an example of a software application executed by server 102. When executed, the annotation web application 124 can use various algorithms to perform aspects of the operations described herein. In the example, as discussed above, the annotation web application 124 may include instructions executable by processor 106 of server 102. Computer-executable instructions can be compiled or interpreted from computer programs created using various programming languages ​​and / or techniques, which include, without limitation and individually or in combination, JAVA, C, C++, C#, VISUAL BASIC, JAVASCRIPT, PYTHON, PERL, PL / SQL, etc. Generally, processor 106 receives instructions, for example, from storage device 110, computer-readable media, etc., and executes these instructions to perform one or more processes, including one or more processes described herein. Such instructions and other data can be stored and transmitted using various computer-readable media.

[0037] Web client 126 may be a web browser or other web-based client executed by client device 104. When executed, web client 126 may allow client device 104 to access annotation web application 124 to display the user interface of annotation web application 124. Web client 126 may further provide input received via HMI control 116 to annotation web application 124 on server 102 via network 122.

[0038] In artificial intelligence (AI) or machine learning systems, model-based inference refers to inference methods that operate based on a machine learning model 128 of the worldview being analyzed. Generally, the machine learning model 128 is trained to learn a function that provides a precise correlation between input and output values. At runtime, the machine learning engine uses the knowledge encoded in the machine learning model 128 to derive conclusions such as diagnoses or predictions from observed data. An example machine learning system may include the TensorFlow AI engine, made available by Alphabet Inc. of Mountain View, California, although other machine learning systems may be used additionally or alternatively. As discussed in detail herein, the annotation web application 124 and the machine learning model 128 can be configured to identify and annotate features of image data 132 for use in efficient and scalable benchmark ground truth generation systems and methods, resulting in high-precision (pixel-level accuracy) annotations for developing object detection / localization and object tracking.

[0039] Image data source 130 may be a camera, for example, mounted on a moving object such as a car, wall, or utility pole, or mounted in a mobile device, configured to capture image data 132. In another example, image data input 132 may be an interface, such as a network device 118, or an interface to storage device 110, used to retrieve previously captured image data 132. Image data 132 may be video, such as an image sequence. Each image in image data 132 may be referred to herein as a frame. For privacy reasons, faces and license plates may be blurred from image data 132 for certain annotation tasks.

[0040] Figure 2Example 200 illustrates a data diagram for annotating image data 132. As shown, raw image data 132, such as video, is stored in a data lake (e.g., storage device 110, drive, or other storage device). In annotation task workflow 202, raw image data 132 is provided to human annotation task 204 to create annotations 206. Annotations 206 may include, for example, weak annotations 208, machine learning annotations 210, and final annotations 212. Additional metadata 214 may also be stored regarding the raw image data 132. For example, the additional metadata 214 may include weather conditions during which the raw image data 132 was captured, the geographical location where the raw image data 132 was captured, the time during which the raw image data 132 was captured, etc. As discussed in further detail below, a training data selector 216 may be used to select raw image data 132 and annotations 206 from storage 110, as shown at 218. At 220, the machine learning algorithm receives the selected raw image data 132 and annotations, and at 222, a revision of the training model is created. The trained model is then used by annotation task workflow 202 to provide machine learning annotations 210. The machine learning annotations 210 can also be supervised by human annotation task 204.

[0041] Figure 3 The diagram illustrates Example 300 of the annotation task workflow. Generally, the annotation workflow is divided into two stages: 1) keyframe annotation and 2) intermediate frame annotation. If the keyframe interval is one, all frames are annotated as keyframes without intermediate frame annotation. For keyframe annotation, the keyframe interval is selected, then the annotation type and task are determined, and then the keyframe annotations can be reviewed. For intermediate frame annotation, the intermediate frame annotations are first generated by the machine, then a human annotator verifies the correctness of the machine annotations and provides feedback to the machine by correcting the annotations.

[0042] Regarding keyframe annotation, a keyframe interval selection is performed. In the example, the keyframe interval can be selected or configured as a static value in the annotation system 100 (e.g., the keyframe interval is 5). In another example, the keyframe interval can be defined based on a formula or function with parameters important in the domain. For example, the keyframe interval selection function in the autonomous driving domain can be represented using parameters such as car speed, steering wheel, road type (city, highway), etc., and / or scene similarity / difference across neighboring frames. The keyframe interval can be dynamically selected by the annotator to maximize their efficiency and annotation accuracy when they interact with the underlying machine, which automatically generates annotations for subsequent frames after the annotator's input.

[0043] Regarding annotation types and tasks, the annotation system 100 can be dynamically configured for different annotation tasks depending on efficiency or other reasons. The annotation system 100 can also be configured to assign a maximum and minimum number of tasks to an annotator.

[0044] Regarding manual or purely human annotation, annotations can be primarily performed by one or more human annotators. Depending on the task complexity, human annotators can be qualified for a given task type by completing online training courses and passing a qualification test.

[0045] Figure 4 Example 400 of the qualification task is illustrated. As shown, Example 400 illustrates the annotation task interface, which demonstrates aspects of bounding box training, where it is the turn of a human operator to draw bounding boxes around the same pedestrian shown in the previous example. User interface buttons are provided in the displayed user interface to allow the human to draw the boxes. Once the boxes have been drawn, the "Check My Work" control can be selected to allow the user to check the work.

[0046] The annotation task interface may include an instruction pane where instructions can be provided, and an annotation pane where the annotator can draw geometry over the target object on the image / frame. Geometry may include bounding boxes, centerlines, cuboids, L-shapes, single-point or multi-point shapes, lines, or freehand drawing forms. It should be noted that these are common examples of shapes that can be used in annotation tasks, but different shapes can be used. Annotation tasks may include providing various properties, such as a person's body posture, head angle, etc. Note that these are specific requirements that may only be used in certain examples, and some implementations may not have these properties for pedestrian annotation. Furthermore, different object types may have different properties. For example, pedestrians may have walking direction or body posture properties, but vehicles may not.

[0047] Annotation tasks can include object matching tasks, which require identifying the same objects across different frames. If an annotator is required to draw geometry for the same object across frames, an object ID can be assigned to each annotation across frames. If frames are divided into multiple annotation tasks, an ID matching task can be created to find the same objects across frames. Optionally, ID matching can be initially performed by a machine before creating ID matching tasks for human annotators. About Figure 5 , Figure 6 and Figure 7 An example of manual annotation is shown. About Figure 8 and Figure 9 The illustration shows an example of ID matching.

[0048] Figure 5Example 500 of a user interface for performing manual annotation is illustrated. In the example, the user interface may be provided to a display 114. The title of the user interface indicates that the user interface is used to draw bounding boxes around a person for five frames of raw image data 132. As shown, the manual annotation user interface includes an instruction pane that instructs the user to look at the frames to the right and select a pedestrian that is larger than the ruler shown below and has no yellow box around it. If such a pedestrian is found, the user can select a control to indicate that the pedestrian to be annotated has been found. If not, the user can select a control to indicate that all pedestrians have boxes. The user interface may also include an annotation page that displays the frames of the raw image data 132, from which the user can attempt to identify pedestrians.

[0049] Figure 6 Example 600 illustrates another aspect of a user interface for performing manual annotation. As indicated, the purpose of the user interface is to identify accurate bounding boxes and centerlines for all pedestrians. Actions that can be used to identify a new pedestrian include clicking the "New Pedestrian" control in the user interface and then selecting the outermost points of the four sides of the pedestrian's body. If the pedestrian is partially occluded, points should be entered to estimate the outermost point of coverage. Next, the user can adjust the centerline to cross the center of the pedestrian's hip. This process can be repeated until all pedestrians have accurate boxes and centerlines. Additional attention can be given to dark areas in the image and to small pedestrians. In one example, the human operator can be paid per entered box or per identified pedestrian. If so, an indication of the human operator's payment can be included in the user interface.

[0050] In the example, if the machine learning aspect of the annotation system 100 determines that there are more than a predetermined number of pedestrians in the image (e.g., 20 pedestrians), the user interface can provide the user with the option to stop and submit after a portion of the pedestrians have been located (e.g., once 20 pedestrians have been located).

[0051] In the example, if the image contains a large number of pedestrians, it can be divided into different patches for individual manual annotation. A user can initially view one patch and can click on an additional patch to show that portion of the image. In some implementations, once a new patch is shown, the user may not be able to return to a previous patch.

[0052] Figure 7Example 700 of pedestrian direction annotation is illustrated. In addition to the bounding box and center line, additional attributes of the pedestrian, such as the direction the pedestrian is walking, can be annotated in the image. As shown, the user interface header indicates the user interface for identifying pedestrian direction. Additionally, the annotation user interface includes an instruction pane that instructs the user to identify which direction best describes the direction the pedestrian highlighted in the annotation pane is walking. Selections can include the pedestrian walking to the left, walking to the right, walking (roughly) towards the driver, or walking (roughly) away from the driver. The user interface can further query which angle best reflects the angle the highlighted pedestrian is walking at and can provide some example angles. Additionally, the instruction pane indicates that if no highlighted pedestrian is present in the current frame, any answer can be selected, and the reviewer's work will be reviewed by another worker. The user interface may also include an annotation page displaying the frame of the original image data 132, from which the user can attempt to identify the pedestrian's walking direction.

[0053] Figure 8 Example 800 of pedestrian ID matching is illustrated. As shown, the user interface provides selection of one or more pedestrian sets identified in a base frame (shown on the left). The user interface also provides a matching frame (shown on the right) from which the user can map the same pedestrians. In this way, the same ID can be used for the same pedestrian across frames.

[0054] Figure 9 The illustration shows example 900 of raw image data frames for pedestrian ID matching. As shown, worker A requests five keyframes to annotate a pedestrian. (If there are n pedestrians, then for the same keyframes, there are a total of n different tasks to be completed using the workers). Also shown, worker B requests the next set of five keyframes to annotate the same pedestrian. Between the two sets of frames, there should be a match for the same pedestrian to generate a consistent pedestrian ID.

[0055] The approach shifts to AI-assisted annotation, a type of annotation designed to be completed interactively by the annotator and the machine. Human annotators can provide weak annotations on the target object (e.g., a single point click on the center of the target object, a coarse bounding box covering the target object, etc.). The machine task can provide refined / accurate geometric annotations (e.g., a precise bounding box on a target pedestrian). If the machine-generated geometric annotations are inaccurate and / or out of tolerance, the annotator can simply provide feedback by correcting any incorrect parts via a user interface. Weak annotations, the original machine-predicted annotations, and the human-corrected annotations can be saved to Annotation System 100 for online or offline retraining of the machine.

[0056] Machine-generated annotations can be implemented through various schemes. In one example, a scheme can be employed that takes an image patch as input and estimates the tight bounding boxes around the main objects in the image patch as output. The input image patch can be a cropped region from the original image, based on the object location estimated by an object detector or tracker. Deep convolutional network architectures can be utilized, which efficiently reduce errors in computer vision pre-annotation and are easily adapted to datasets different from the training data. This latter property can be useful in annotation systems 100, as they often encounter data with different metadata characteristics (e.g., different camera parameters, different road and weather conditions, etc.).

[0057] Different geometric shape annotations can also leverage other machine learning algorithms. In the example, to achieve accurate bounding boxes, machine-generated semantic segmentation annotations can be used by selecting the outermost point of the target fragment's (x, y) coordinates. Similarly, the center of the body annotation can be generated by utilizing predicted body keypoints.

[0058] Figure 10 Example 1000 of a user interface for performing AI-assisted manual annotation is illustrated. As shown, the user interface can indicate that the goal is to annotate accurate bounding boxes and centerlines for all pedestrians with AI assistance. Steps that can be performed to do this may include first clicking two points, the top left and bottom right, to cover the pedestrian. The AI ​​can then generate bounding boxes. The AI ​​can then be taught, if necessary, by the user clicking the correct outermost point(s) of the pedestrian. If the pedestrian is partially occluded, points should be entered to estimate the outermost point of coverage. The AI ​​can then generate centerlines. If necessary, the AI ​​can be taught by the user clicking to correct the centerlines. This process can be repeated until all pedestrians have accurate boxes and centerlines. Additional attention can be given to dark areas in the image and to small pedestrians. In one example, a human operator can be paid per entered box or per identified pedestrian. If so, an indication of the human operator's payment can be included in the user interface. Additionally, the user interface can provide an indication of how much the AI ​​has learned from the user.

[0059] Besides manual annotation and AI-assisted annotation, a third type of annotation is machine-initiated annotation with pre-annotations. This type of annotation can use an object detector to detect objects with object classes / categories. These detected objects can be input into the AI ​​as image patches cropped from video frames based on pre-annotated bounding boxes from computer vision algorithms (object tracking or object detection). Before cropping the image, the four sides of the pre-annotated bounding boxes can be expanded to ensure that the visible portion of the object is included in the image patch. The AI ​​can then predict the precise bounding boxes of the objects. The video annotation scheme is one example, but the AI ​​scheme can be applied to any annotation system 100 that utilizes computer vision pre-annotations, or to make the coarse bounding boxes drawn by the annotator more accurate. In such an annotation pipeline, the input video sequence of raw image data 132 is first divided into keyframes (sampled every K frames, where K can be determined by the speed of the moving car and environmental changes) and intermediate frames. The pre-annotations in the keyframes can be initialized by an object detector and then refined by the AI. Such keyframe pre-annotations can be reviewed by a human worker and then corrected or redrawn to ensure that the keyframe annotations are accurate. Annotated keyframes can be used to populate the pre-annotations of intermediate frames using an object tracker. The pre-annotations of the intermediate frames can then be refined by AI and corrected by human annotators. The result can be that detected objects input to the AI ​​are refined to tight bounding boxes of the detected objects. In any case, the pre-annotations from step 1 can be verified and / or corrected by one or more human annotators.

[0060] Before generating intermediate frame annotations, keyframe annotations can be reviewed by one or more different human annotators with high-quality profiles, resulting in high-quality annotations. The review process can be a single, consolidated task, or it can be broken down into multiple steps performed by more than one human annotator. Review can be performed on all annotations, or alternatively, only annotations with low confidence levels can be reviewed. Figure 6 An example is shown in the illustration.

[0061] The process moves from keyframe annotations to intermediate frame annotations, which can initially be generated by the machine. Human annotators can then verify the correctness of the machine annotations and provide feedback to the machine by correcting them.

[0062] Geometry and object ID generation can be performed by a machine. Pre-labeled locations / positions of target objects can be identified based on the positions of the start and end keyframes of the selected frames. This position can be calculated using interpolation and computer vision techniques such as kernel correlation filters. The generated annotations can have the same ID.

[0063] Patches of the target object (pre-labeled from a machine learning algorithm with extra regions) can be created to obtain estimates of fine-grained (high-precision) geometric shape annotations. A detailed description of the example estimator is discussed in this paper.

[0064] Figure 11 The diagram illustrates the architecture 1100 of the AI-based annotator. As shown, the first set of layers is a feature extractor, followed by three fully connected layers. In this example, the output dimension is four, corresponding to the (x, y) coordinates of the bottom left and top right corners of the bounding box. Here, the feature extractor refers to the convolutional layer in a well-known deep neural network architecture used in computer vision, such as VGG16 (in Karen Simonyan and Andrew Zisserman, *Very deep convolutional networks for large-scale image recognition*). arXiv preprint arXiv:1409.1556 (described in detail in 2014), ResNet50 (in Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deep residual learning for image recognition,” IEEE Conference Proceedings on Computer Vision and Pattern Recognition ( Proceedings of the IEEE conference on computer vision and pattern recognition (described in ), pp. 770–778, 2016, and MobileNet (Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications.” arXiv preprint arXiv:1704.04861 (2017), each of which is incorporated into this paper in its entirety by reference.

[0065] Regarding the overall architecture, the AI-based annotator first learns and extracts features from the image, and then uses the extracted features to estimate the four limiting coordinates of the main object in the image patch. Therefore, the architecture is divided into a feature extractor and a coordinate estimator. For the feature extractor, the design goal is to represent the main object in the image patch. In some examples, this can be achieved using feature extractors that have proven effective in various computer vision tasks, such as the first thirteen layers of VGG16, the first forty-nine layers of ResNet50, and the first eighteen layers of MobileNet. Another design metric for the feature extractor is transferability, as data annotation systems often encounter images with different characteristics; therefore, the AI ​​should be easily transferable to different datasets.

[0066] Regarding direct transferability, at the beginning of annotating a new set of images, the AI ​​model used in the annotation pipeline will have already been trained using different datasets. The AI ​​can estimate the four limit coordinates of an object as accurately as possible, although more error is expected due to the AI ​​model being trained on different datasets. That is, the AI ​​model should have the lowest possible variance while maintaining its estimation power. Given the same amount of training data and training strategy, reducing variance requires reducing the number of parameters.

[0067] Regarding the minimum amount of data required for adaptation, while AI models are designed to be directly transferable, they will never perform as well as if trained on the same dataset. A common practice in data annotation is to retrain or fine-tune the computer vision algorithm used in the system after annotating a portion of the data. It can be seen that if the algorithm can be fine-tuned earlier in the annotation process, it can assist the human annotator more, and therefore the cost of annotating the entire dataset is lower. Therefore, AI models should require a minimum amount of data to be fine-tuned to a new dataset.

[0068] Moving to the coordinate estimator, the top layer of the AI ​​model is configured to learn the mapping between extracted image features and the four extreme coordinates of the main object in the image patch. The estimator can be a deep neural network with an architecture that includes a feature extractor and a coordinate estimator. The feature extractor can be consistent with architectures proven useful in the computer vision literature. Examples of such feature extractors are VGG16, ResNet50, and MobileNet. The coordinate estimator can be configured to learn the mapping between extracted image features and the four extreme coordinates of the main object in the image patch. To learn such a non-linear mapping, more than one layer of regression is needed because the coordinate estimator is inherently more difficult than object detectors, which typically have only one fully connected layer after pooling. A loss function can also be defined. For the loss function, the AI ​​aims to make the bounding box boundaries as accurate as possible, meaning as few pixels as possible between the object extreme points and the bounding box. For example, L1 distance is a natural choice for measuring estimation performance. For optimization, Huber loss can be used. To simulate the error in the pre-labels used to train such an estimator, error statistics generated in the pre-labels can be collected, and such errors can be injected when generating the image patch for training. For the centerline of the body annotation, in order to use the pedestrian target object type, the body's centerline can be predicted by leveraging a machine model for body keypoints and / or by centerline interpolation based on two keyframes. Attribute annotation generation can also be performed by a machine. Attributes can be generated based on interpolation of two keyframes.

[0069] The first step in training an AI model is to estimate the distribution of errors that the AI ​​model will correct. Such an error distribution helps the coordinate estimator locate the main object in the image patch and helps the training process simulate the errors the AI ​​model needs to correct in the annotation pipeline. Furthermore, training with the error distribution, rather than the actual errors from the computer vision algorithm, better isolates the AI ​​model from how accurately the computer vision algorithm performs in the annotation pipeline and improves the transferability of the AI ​​model to new datasets.

[0070] By using Figure 11Using the annotation pipeline in the example, the errors corrected by the AI ​​model can include object detector errors and object tracker errors. Ideally, statistics from both algorithms could be collected, but to reduce training effort, the AI ​​model can be trained first using the worse error. In this case, since the object tracker is initialized every K frames, and the objects of interest (vehicles and pedestrians) typically do not have sudden changes in motion, the bounding box boundary errors of the object tracker should be smaller than those of the object detector, provided K is not too large. Therefore, the AI ​​can operate the object detector on the entire training dataset and match the bounding boxes to ground truth to collect the bounding box boundary error statistics introduced by the object detector.

[0071] Figure 12 Example 1200 illustrates the process for extracting image patches used for training. After obtaining the computer vision error statistics, the next step is to extract the image patch containing each object and the ground truth coordinates of the object within each patch—the image patch will be the input to the AI ​​model, and the ground truth coordinates will be used to calculate the loss value. As shown in Example 1200, given an image of a fully visible object and a ground truth bounding box, the four sides of the ground truth bounding box are first expanded at a fixed ratio to ensure that the object is fully included in the image patch; then, based on the error statistics collected from the object detector, the four sides are randomly shifted (each side can be shifted inward or outward depending on the amount drawn from the distribution); the image patch is then cropped, and the patch is normalized to a fixed size as input to train the AI ​​model. The normalization process maintains the aspect ratio of the original patch and uses 0 to padded empty pixels for all channels. Note that the training patch is generated dynamically (on the fly) during each epoch of training, so the size of the cropped patch may differ when the object is used multiple times during training.

[0072] For human annotators, certain validation tasks may exist. These tasks can be broken down into three distinct steps: i) removing any incorrect annotations that cover one or more non-target objects, ii) adding new annotations that do not cover the target objects, and iii) adjusting the geometry annotations if the machine-generated annotations do not meet accuracy requirements. If a machine confidence level exists, the validation task may target annotations with low confidence levels.

[0073] Then, a final review task can be performed on all frames. The review process can be a single, consolidated task, or it can be broken down into multiple steps, which can be performed by more than one human annotator. The review can be targeted at annotations with low confidence levels. Machine-based interactive review can be utilized. After review, geometry and ID tracking of all target object annotations can again be performed by the machine. Additional annotations (e.g., object grouping) can be added by the machine by calculating the overlap of bounding boxes and attributes of two or more objects.

[0074] Figure 13 Example 1300 illustrates the final review task for a set of frames. As shown, three frames are in progress, while three other frames are waiting. Of the three frames in progress, the first frame is indicated as having completed bounding box annotation and ID generation, while the second and third frames are only indicated as having completed bounding box annotation.

[0075] Results from the annotation system can be used to perform continuous training of machine learning models. Annotation system 100 stores all annotation results (including intermediate and final ones) to continuously train the machine learning algorithms used during the annotation process. (Reference: Back) Figure 2 The continuous training pipeline architecture has access to a data lake (storage repository) containing raw image data 132, all annotation data, and metadata. A training data selector determines which data to apply to the next training cycle. The training data selector has functions and logic programmed to statistically analyze the distribution of metadata, calculate the difference between machine annotations and final annotations, and select target training data based on the analysis results to maximize learning in the machine learning algorithm. For example, if the IoU (joint intersection) between the final annotation and the machine annotation for objects with height > 500 pixels in a night scene is < 70%, then frames with those annotations can be selected as target training data.

[0076] Regarding quality control of crowdsourced workers, the annotation system 100 can categorize human annotators into two distinct roles based on their quality profiles. One role is that of the average worker—a worker who has undergone training and qualification (if necessary) to perform annotation tasks. The other role is that of a trusted node / worker who has completed a significant amount of work in the past, and this trusted worker is responsible for reviewing the tasks of other workers.

[0077] The annotation system 100 can have three different review processes. Depending on the task type, one or more of these different review processes can be applied. The first of these processes is a two-level review among workers themselves for the same task. The second of these processes is an independent review / verification task. The third of these review tasks is a final review task performed by experts (for keyframes and final results).

[0078] Figure 14 The diagram illustrates a sample workflow of two-level review 1400. In two-level review, worker (human annotator) review occurs after each worker (e.g., worker A) submits his / her task. Annotation system 100 can assign the review and annotation task to another worker (e.g., worker B) to review worker A's accuracy and provide feedback before worker B moves on to his / her own annotation task.

[0079] Referring more specifically to Example 1400, if worker B is not a trusted node / worker, the annotation system 100 can create a review task for a trusted worker to review worker B's task. If the review of worker B's task is negative, the annotation system 100 can send the task back to the original worker (worker B) and request him / her to revise it. If worker B does not provide a revision before the deadline, the task can be rejected, and another review and annotation task can be created. Otherwise, worker B's task can be approved. If the review of worker B's task is positive, worker B's result is valid.

[0080] If the review of worker A's task is negative, the annotation system 100 can send the task back to the original worker (worker A) for revision. If worker A does not provide a revision before the deadline, the task can be rejected, and another review and annotation task can be created. Otherwise, worker A's task can be approved. After a task is approved or rejected, the worker's quality profile is updated.

[0081] For independent review / verification tasks, for annotations primarily performed by machines, independent tasks where the review is done by only a small number of workers (n) can be utilized instead of a two-level review process. For final review tasks performed by experts (e.g., for final results on intermediate frames and keyframes): before publishing / finalizing the baseline truth, experts (whose quality profile is higher than or equal to the minimum quality condition of the trusted node) can be engaged to correct any incorrect annotations.

[0082] Figure 15The illustration shows an example 1500 of a review question used to perform a final review. As shown, the user interface is asking the worker to look at the frame on the right (the annotation pane) and identify pedestrians where there are no bounding boxes around them. Instructions may continue to remind the worker to pay specific attention to dark areas and small pedestrians. The review question might ask how many pedestrians lack bounding boxes around them. In answering this question, the annotation system 100 may receive additional input regarding the quality of the annotation.

[0083] Figure 16 An example of a process 1600 for decomposing and executing an annotation task into multiple tasks is illustrated. In the example, process 1600 can be executed by an annotation web application 124 within the context of annotation system 100. Process 1600 may include flows for keyframe annotation and flows for intermediate frame annotation.

[0084] Regarding keyframes, the process can begin at operation 1602, where the annotation web application 124 selects keyframes for annotation. In the example, the annotation web application 124 can identify keyframes in the input video sequence of the raw image data 132.

[0085] At operation 1604, the annotation web application 124 can identify the type of annotation task to be performed, as well as the type of proxy for the annotation. The task type can include, for example, a purely human annotation task, an AI interaction task, or a human task with pre-annotated machine learning data. The proxy type can include, for example, a worker with average annotation skills, a worker with expert skills, or a machine. In some examples, these identifiers can be executed based on user input. In other examples, these identifiers can be executed by the annotation web application 124 based on the raw image data 132 available for annotation.

[0086] At point 1606, the web application 124 can launch a test instance for the selected frame to execute the annotations. (See above for reference.) Figure 4 , 5 Sections 6, 7, 8, 10, 13, and 15 discuss in detail the example user interface for annotation and / or annotation review. The review can then be performed by the indicated agent type for the indicated task type. Once the review is complete or aborted, control can return to operation 1604 at operation 1608 to select another annotation task.

[0087] Additionally or alternatively, process 1600 may continue from operation 1608 to operation 1610, wherein frames of the annotated raw image data 132 are selected for expert review. This selection may be performed based on a confidence level of the annotation. For example, the confidence level may be based on one or more performances of workers performing the annotation task (or the overall performance of workers in all annotation tasks), the prediction score of the machine determined based on the annotated machine identification, an analysis of the image quality of the raw image data 132, and / or based on other difficulties in performing the annotation (e.g., the lack of ability of human operators or machines to identify objects in the raw image data 132).

[0088] At operation 1612, the annotation web application 124 initiates a review task instance for selected frames of the annotated raw image data 132. In this example, the review can be AI-assisted or purely human. (See above reference.) Figure 4 , 5 Sections 6, 7, 8, 10, 13, and 15 discuss in detail the example user interface for annotation and / or annotation review.

[0089] At operation 1614, the annotation web application 124 completes the review of the selected frame and target object. For example, the annotation web application 124 can confirm that the task type review has been completed for all objects, for all objects with a confidence level below the threshold, etc.

[0090] Next, at 1616, the web application annotation 124 determines whether there are any additional annotations and tasks to be performed for the keyframe. If yes, control is transferred to operation 1604. If not, process 1600 ends.

[0091] Regarding annotations for intermediate frames, at operation 1618, the annotation web application 124 can automatically generate annotations for the intermediate frames. At operation 1620, similar to operation 1604, but for intermediate frames, the annotation web application 124 can select the task type and the proxy type of the intermediate frame. For example, the task can be a purely human task or an AI interaction task.

[0092] At operation 1622, similar to operation 1606, but for intermediate frames, the commenting web application 124 starts a task instance for the intermediate frame. Depending on the task type and proxy type, the intermediate frame can be inspected accordingly. Following operation 1622, at operation 1624, the commenting web application 124 determines if there are any additional reviews to be performed on the intermediate frame. For example, there might be several intermediate frames that need to be reviewed using different task types. If so, control transfers to operation 1620. If not, once the annotation and review of the keyframe, as well as the intermediate frames, are complete, control transfers to operation 1626 to indicate the completion of the annotation.

[0093] Generally, the processes, methods, or algorithms disclosed herein can be delivered to / implemented by a processing device, controller, or computer, which may include any existing programmable electronic control unit or dedicated electronic control unit. Similarly, processes, methods, or algorithms can be stored in many forms as data and instructions executable by a controller or computer, including but not limited to information permanently stored on non-writable storage media such as ROM devices, and information reproducibly stored on writable storage media such as floppy disks, magnetic tapes, CDs, RAM devices, and other magnetic and optical media. Processes, methods, or algorithms can also be implemented in a software executable object. Alternatively, suitable hardware components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or combinations of hardware, software, and firmware components, can be used to embody processes, methods, or algorithms, wholly or partially.

[0094] While exemplary embodiments have been described above, these embodiments are not intended to describe all possible forms covered by the claims. The language used in this specification is descriptive rather than restrictive, and it should be understood that various changes may be made without departing from the spirit and scope of this disclosure. As previously stated, features of various embodiments may be combined to form other embodiments of the invention that may not be explicitly described or illustrated. While various embodiments may have been described as providing an advantage over or preferred over other embodiments or prior art implementations in one or more desired features, those skilled in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system properties depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, lifecycle cost, merchantability, appearance, packaging, size, suitability, weight, manufacturability, ease of assembly, etc. Accordingly, any embodiment described as less desirable than other embodiments or prior art in one or more features is not outside the scope of this disclosure and may be desirable for a particular application.

Claims

1. A system for generating high-precision benchmark ground truth data for human-computer collaboration in object identification, localization, and tracking in image sequences, comprising: User interface; A storage device configured to maintain raw image data including video with frame sequences, and frame annotations indicating objects identified in the respective frames; and The processor, which communicates with the storage device and the user interface, is programmed to... For the annotation of keyframes in the original image data, the task type and proxy type of the keyframes are determined. The task type includes one of the following: a purely human annotation task, an AI interactive task, or a human task with machine learning pre-annotation. The proxy type includes one of the following: a worker with average annotation skills, a worker with expert skills, or a machine using a machine learning model. Based on the keyframe task type and keyframe proxy type, receive annotations of objects identified in the keyframes of the original image data. Based on the confidence level of the keyframe annotations, keyframes are selected for review, wherein the confidence level is based on one or more of the following: (i) the performance of the worker performing the annotation task, (ii) the worker's overall performance across multiple annotation tasks, (iii) the prediction score determined based on the machine identifier of the annotations, or (iv) the image quality analysis of the original image data, the annotation of intermediate frames for the original image data, the determination of the task type and agent type of the intermediate frames, and Based on the intermediate frame task type and intermediate frame proxy type, receive annotations for objects identified in intermediate frames of the original image data; The processor is programmed to generate pre-labels using an object tracker and annotations of objects identified in keyframes when the task type of the intermediate frame is a human task with machine learning pre-labels, and to refine the pre-labels using AI.

2. The system according to claim 1, wherein, The processor is further programmed to detect objects with precise, tight boundary geometry using the machine learning model when operating with an agent type that utilizes a machine learning model, the machine learning model having a deep convolutional network architecture including a feature extractor configured to identify features of the object, and a coordinate estimator configured to identify the object coordinates using the identified features.

3. The system of claim 1, wherein the objects used for annotation include one or more of pedestrians, cyclists, animals, vehicles, animals in an indoor environment, or moving objects.

4. The system of claim 1, wherein the annotation includes one or more of the following: the geometry around the object, the bounding box around the object, the centerline of the object, object-specific attributes, or the direction of travel of the object.

5. The system of claim 1, wherein the processor is further programmed to: Select frames from the raw image data and their corresponding manual annotations; The training of the machine learning model was revised, and the machine learning model was configured to use manual annotations to identify objects in the frame; and Provide machine learning annotations for the frames to be received via a user interface for manual correction.

6. The system of claim 5, wherein the raw image data is associated with additional metadata, the additional metadata including one or more elements of contextual information specifying one or more of the weather conditions during which the raw image data was captured, the geographic location during which the raw image data was captured, or the time during which the raw image data was captured, and the metadata is used as input to aid in the revised training of the machine learning model.

7. The system of claim 5, wherein the manual corrections received via the user interface are used as at least part of the manual annotations to revise the training of the machine learning model.

8. The system of claim 5, wherein the manual annotation of the frame includes a click that identifies the estimated center of an object regardless of whether the object is occluded, and the machine learning annotation includes the boundary geometry around the object, as identified by the center.

9. The system of claim 5, wherein the manual annotation of the frame includes an estimated outermost point identifying the object regardless of whether the object is occluded, and the machine learning annotation includes the centerline of the object as identified by the outermost point.

10. The system according to claim 1, wherein, The processor is further programmed to receive verification input from the user interface as a review of the annotations on the keyframes. The verification input includes one or more of the following: (i) manually deleting incorrect annotations that cover non-target objects, (ii) manually adding new annotations that do not cover target objects, and (iii) adjusting the geometry of annotations for machine-generated annotations that fail to meet accuracy requirements.

11. A method for generating high-precision benchmark ground truth data for human-computer collaboration in object identification, localization, and tracking in image sequences, comprising: The maintenance includes raw image data of video with frame sequences, as well as frame annotations indicating objects identified in the respective frames, including one or more of pedestrians, cyclists, animals, vehicles, animals in an indoor environment, and moving objects, and the annotations include one or more of the geometry around the object, the centerline of the object, or the direction of travel of the object; For the annotation of keyframes in the original image data, the task type and proxy type of the keyframes are determined. The task type includes one of the following: a purely human annotation task type, an AI interaction task type, or a human task with machine learning pre-annotation. The proxy type includes one of the following: a worker with average annotation skills, a worker with expert skills, or a machine using a machine learning model. Based on the keyframe task type and keyframe proxy type, receive annotations for objects identified in the keyframes of the original image data. Based on the confidence level of the annotation of keyframes, select review keyframes, wherein the confidence level is based on one or more of the following: (i) the performance of the worker performing the annotation task, (ii) the overall performance of the worker across multiple annotation tasks, (iii) the prediction score determined based on the machine identifier of the annotation, or (iv) the image quality analysis of the raw image data. Annotations for intermediate frames of the original image data determine the task type and proxy type of the intermediate frames; and Based on the intermediate frame task type and intermediate frame proxy type, receive annotations for objects identified in intermediate frames of the original image data; Specifically, when the task type of the intermediate frame is a human task with machine learning pre-annotations, pre-annotations are generated using an object tracker and annotations of objects identified in the keyframes, and the pre-annotations are refined by AI.

12. The method of claim 11, wherein the processor is further programmed to: Select frames from the raw image data and their corresponding manual annotations; The training of the machine learning model was revised, and the machine learning model was configured to use manual annotations to identify objects in the frame; and Provide machine learning annotations for the frames to be received via a user interface for manual correction.

13. The method of claim 12, wherein the original image data is associated with additional metadata, the additional metadata including one or more elements of contextual information specifying one or more of the weather conditions during which the original image data was captured, the geographic location during which the original image data was captured, or the time during which the original image data was captured, and the metadata is used as input to aid in the revised training of the machine learning model.

14. The method of claim 12, wherein the manual corrections received via the user interface are used as at least part of the manual annotations to revise the training of the machine learning model.

15. The method according to claim 12, wherein, The manual annotation of the frame includes a click that identifies the estimated center of the object regardless of whether the object is occluded, and the machine learning annotation includes the geometry around the object as identified by the center.

16. The method of claim 12, wherein the manual annotation of the frame includes identifying the estimated outermost point of the object regardless of whether the object is occluded, and the machine learning annotation includes the centerline of the object as identified by the outermost point.

17. The method according to claim 11, wherein, The processor is further programmed to receive verification input from the user interface as a review of the annotations on the keyframes. The verification input includes one or more of the following: (i) manually deleting incorrect annotations that cover non-target objects, (ii) manually adding new annotations that do not cover target objects, and (iii) adjusting the geometry of annotations for machine-generated annotations that fail to meet accuracy requirements.

18. A computer-readable medium including instructions that, when executed by a processor, cause the processor to: maintain raw image data comprising a video having a sequence of frames, and frame annotations indicating aspects of objects identified in the respective frames, the objects including one or more of pedestrians, cyclists, animals, vehicles, animals in an indoor environment, and moving objects, the annotations including one or more of geometry around the object, a centerline of the object, or a direction of travel of the object; For the annotation of keyframes in the original image data, the task type and proxy type of the keyframes are determined. The task type includes one of the following: a purely human annotation task type, an AI interaction task type, or a human task with machine learning pre-annotation. The proxy type includes one of the following: a worker with average annotation skills, a worker with expert skills, or a machine using a machine learning model. Based on the keyframe task type and keyframe proxy type, receive annotations for objects identified in the keyframes of the original image data. The confidence level of the annotation based on the keyframes is selected for review, and the confidence level is based on one or more of the following: (i) the performance of the worker performing the annotation task, (ii) the overall performance of the worker across multiple annotation tasks, (iii) the prediction score determined based on the machine identifier of the annotation, or (iv) the image quality analysis of the raw image data. Annotations on intermediate frames of the original image data determine the task type and proxy type of the intermediate frames; and Based on the intermediate frame task type and intermediate frame proxy type, receive annotations for objects identified in intermediate frames of the original image data; in, When the task type of the intermediate frame is the same as that of a human task with machine learning pre-annotations, pre-annotations are generated using an object tracker and annotations of objects identified in the keyframes, and then refined using AI.

19. The medium of claim 18, further comprising instructions that, when executed by a processor, cause the processor to: Select frames from the raw image data and their corresponding manual annotations; Provide machine learning annotations for the frames to facilitate receiving manual corrections via a user interface; and The training of the machine learning model is revised to identify objects by utilizing manual corrections received via a user interface as at least part of the manual annotations. The raw image data is associated with additional metadata, which includes one or more elements of contextual information that specify one or more of the weather conditions during which the raw image data was captured, the geographic location where the raw image data was captured, or the time during which the raw image data was captured. This metadata is used as input to aid in the revised training of the machine learning model.

20. The medium of claim 18, further comprising instructions that, when executed by a processor, cause the processor to receive verification input from a user interface as a review of annotations for keyframes, the verification input including one or more of the following: (i) manually deleting incorrect annotations that cover non-target objects, (ii) manually adding new annotations that do not cover target objects, and (iii) adjusting the geometry of annotations for machine-generated annotations that fail to meet accuracy requirements.

Images