Tagging methods and systems for machine learning purposes

Abstract

The present invention generally relates to systems and methods for tagging (annotating) data for further use in supervised or semi-supervised machine learning. More specifically, the present invention relates to facilitating tagging of data, where the subject of tagging is not presented in a detailed manner. The present invention represents a technical means for effectively detecting and correcting inaccurately tagged data, helping tagging experts reduce the number of tagging errors. The present invention represents a method for facilitating tagging of data for machine learning purposes, thereby improving the quality of training datasets and, consequently, the reliability of the prediction or classification output of ML models.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Patent Application No. 63 / 436,609, entitled "TAGGING METHOD FOR MACHINE LEARNING PURPOSES," filed January 1, 2023, which is incorporated herein by reference in its entirety.

[0002] The present invention relates generally to systems and methods for tagging (annotating) data for further use in supervised or semi-supervised machine learning. More particularly, the present invention relates to facilitating tagging of data, where the subject of tagging is not presented in a detailed manner. [Background technology]

[0003] As is known, developing mathematical models that can learn from data and make predictions about the data is a common purpose of machine learning. In particular, supervised and semi-supervised machine learning involves model training using a so-called "training dataset" (or "supervised dataset"), fine-tuning using a "validation dataset", and testing using a "test dataset". The term "training dataset" generally refers to a set of example input and output vector (or scalar) pairs. The model iteratively analyzes the input data in the training dataset to generate results, and then compares the results with the output data corresponding to each input data in the training dataset, which is the target result. Based on the comparison, the supervised learning algorithm determines the optimal combination of variables that provides the highest predictive reliability. Ultimately, a well-trained model will always show sufficiently reliable results when analyzing unknown data.

[0004] As a result, the quality of a training dataset is reasonably considered to be an important aspect of machine learning. However, in practice, the work of acquiring, tagging (labeling), and preparing a training dataset proves to be tedious and expensive. This work requires complex coordination and combinations between machine learning processes, human resources, and tagging tools. The process of creating a training dataset becomes even more difficult when the task involves analyzing ambiguous data, which is difficult for humans to carefully classify and tag, let alone an ML model that must be trained to do so.

[0005] In particular, some problems can arise when tagging different types of geographic data. For example, the task of tagging multi-level overpasses (multi-level transport routes, e.g., pedestrian and vehicle bridges, interchanges, etc.) on satellite imagery and classifying them by type in order to create ML models for controlling autonomous driverless cars can be considered as one such type of task. On satellite imagery, causeways are viewed from above, making it difficult to reliably distinguish between multi-level and single-level overpasses.

[0006] Nevertheless, a well-trained ML model can potentially show more reliable results than a human observer, as it can reveal deeply hidden features in the input data that turn out to be highly related to the target output data.

[0007] In addition to the problem of the quality of the training dataset, there is also the problem of quantity. In practice, it is extremely difficult to provide a sufficient amount of training data to achieve reliable training results. Therefore, this aspect becomes essential, especially when considered together with the fact that preparing a training dataset is cumbersome and expensive. Summary of the Invention [Problem to be solved by the invention]

[0008] Therefore, there is a need for a technological means to effectively detect and correct inaccurately tagged data and assist tagging professionals in reducing the number of future tagging errors. In other words, there is a need for a method to facilitate data tagging for machine learning purposes, thereby improving the quality of training datasets and, consequently, the reliability of the prediction or classification output of ML models. [Means for solving the problem]

[0009] In order to overcome the above-mentioned drawbacks of the prior art, the following invention is provided.

[0010] In a general aspect, the present invention may be directed to a method, by at least one processor, of creating a labeled dataset for training a machine learning (ML) model, the method comprising: receiving an annotation task representing a requirement to annotate an input data element; dividing the annotation task into a plurality of annotation subtasks, each representing a requirement to annotate a respective unannotated portion of the input data element; assigning one or more annotation subtasks of the plurality of subtasks to one or more taggers; obtaining respective annotated portions of the input data element based on a plurality of annotations provided by the taggers corresponding to the assigned annotation subtasks; and forming the labeled dataset by aggregating the annotated portions.

[0011] In another general aspect, the invention may be directed to a system for creating a labeled dataset, the system including: a non-transitory memory device having stored thereon a module of instruction code; and at least one processor associated with the memory device and configured to execute the module of instruction code, wherein upon execution of the module of instruction code, the at least one processor is configured to: receive an annotation task representing a requirement to annotate an input data element; divide the annotation task into a plurality of annotation subtasks, each representing a requirement to annotate a respective un-annotated portion of the input data element; assign one or more annotation subtasks of the plurality of subtasks to one or more taggers; obtain respective annotated portions of the input data element based on a plurality of annotations provided by the taggers corresponding to the assigned annotation subtasks; and form the labeled dataset by aggregating the annotated portions.

[0012] In yet another general aspect, the invention may be directed to a method, by at least one processor, of creating a labeled dataset for training a machine learning (ML) model, the method including: receiving an annotation task representing a requirement to annotate an input data element; receiving the input data element; dividing the annotation task into a plurality of annotation subtasks, each representing a requirement to annotate a respective non-annotated portion of the input data element; and performing an iterative process, each iteration inferring a pre-trained ML-based model on one or more non-annotated portions of the input data element to annotate the one or more non-annotated portions of the input data element according to one or more pre-defined criteria. calculating a confidence value representing the reliability of the input data elements being appropriate for the class; assigning at least one annotation subtask of the plurality of subtasks to at least one tagger based on the calculated confidence value; obtaining annotated portions of each of the input data elements based on the plurality of annotations provided by the at least one tagger corresponding to the assigned at least one annotation subtask; forming an intermediate version of the labeled dataset by aggregating the annotated portions; and supplementary training of the pre-trained ML-based model using the intermediate version of the labeled dataset as supervised data to recalculate the confidence value.

[0013] In some embodiments, the steps of assigning one or more annotation subtasks and obtaining the respective annotated portions are performed as an iterative process comprising a sequence of at least two iterations.

[0014] In some embodiments, the sequence of at least two iterations further includes at least one initial iteration further including forming an intermediate version of the labeled dataset by aggregating the annotated portions and training an ML model using the intermediate version of the labeled dataset as supervised data to calculate confidence values ​​representing the confidence that one or more portions of the input data elements are appropriate for one or more predefined classes; and at least one intermediate iteration further including inferring the trained ML model on one or more unannotated portions of the input data elements to calculate respective confidence values ​​and assigning at least one annotation subtask of the plurality of subtasks to one or more tagging modules based on the calculated confidence values.

[0015] In some embodiments, the sequence of at least two iterations further includes at least one initial iteration further including performing a multi-level quality assurance (QA) procedure on the plurality of annotations to obtain a plurality of respective QA scores, and at least one intermediate iteration further including assigning one or more annotation subtasks of the plurality of subtasks to a particular tagger based on the QA scores.

[0016] In some embodiments, the at least one initial iteration further includes forming an intermediate version of the labeled dataset by aggregating the annotated portions, and training an ML model using the intermediate version of the labeled dataset as supervised data to calculate confidence values ​​representing the confidence that one or more portions of the input data elements are appropriate for one or more predefined classes, and the at least one intermediate iteration further includes inferring the trained ML model on one or more unannotated portions of the input data elements to calculate respective confidence values, and performing a multi-level quality assurance (QA) procedure on the multiple annotations based on the calculated confidence values.

[0017] In some embodiments, the method further includes receiving at least one supplementary data element that relates to one or more portions of the input data element by at least one common characterizing feature, and performing a multi-level quality assurance (QA) procedure on the plurality of annotations based on the at least one respective supplementary data element.

[0018] In some embodiments, performing the multi-level QA procedure includes receiving annotations for at least one annotation subtask via a first user interface (UI) for each first-level tagger, receiving at least one supervised feedback data element for the annotation via a second UI for a second-level tagger, and calculating a QA score for the annotation based on the supervised feedback data elements.

[0019] In some embodiments, the method further includes receiving at least one approval feedback data element for the annotation via a third UI related to the third-level tagger, and further calculating a QA score for the annotation based on the approval feedback data element.

[0020] In some embodiments, the input data elements correspond to a particular geographic region and include a plurality of overpass data elements, each portion of the input data elements corresponding to a sub-region of the geographic region and including a subset of the plurality of overpass data elements.

[0021] In some embodiments, the annotation includes an indication of at least one overpass data element as representing a multi-level overpass or a single-level overpass.

[0022] In some embodiments, the one or more predefined classes are selected from a first class representing the presence of a multi-level overpass in the portion and a second class representing the absence of a multi-level overpass in the portion.

[0023] In some embodiments, the at least one processor may be further configured to assign one or more annotation subtasks and obtain respective annotated portions in an iterative process comprising a sequence of at least two iterations.

[0024] In some embodiments, the at least one processor may be further configured to receive at least one supplementary data element that relates to one or more portions of the input data element by at least one common characterizing feature, and to perform a multi-level quality assurance (QA) procedure on the plurality of annotations based on the at least one respective supplementary data element.

[0025] In some embodiments, the at least one processor may be further configured to perform a multi-level quality assurance (QA) procedure by receiving, for at least one annotation subtask, annotations via a first user interface (UI) for a respective first-level tagger, receiving, for the annotations, at least one supervised feedback data element via a second UI for a second-level tagger, and calculating a QA score for the annotation based on the supervised feedback data element.

[0026] In some embodiments, the at least one processor may be further configured to perform a multi-level quality assurance (QA) procedure by receiving at least one approval feedback data element for the annotation via a third UI related to the third-level tagger, and further calculating a QA score for the annotation based on the approval feedback data element. [Brief explanation of the drawings]

[0027] The subject matter of the present invention is particularly pointed out and distinctly claimed in the concluding portion of the specification, however, both the organization and method of operation, together with other objects, features and advantages thereof, may best be understood by reading the following detailed description when read in conjunction with the accompanying drawings.

[0028] [Figure 1] FIG. 1 is a block diagram illustrating a computing device that may be included in a tagging system for machine learning purposes, according to some embodiments.

[0029] [Figure 2] FIG. 1 is a block diagram illustrating the interconnection of the tagging system with other machine learning aspects, according to some embodiments.

[0030] [Figure 3A] FIG. 1 is a block diagram illustrating a tagging system for machine learning purposes, according to some embodiments.

[0031] [Figure 3B] FIG. 3B is a sequence diagram illustrating the operational mode process of the tagging system for machine learning purposes according to the embodiment of FIG. 3A.

[0032] [Figure 4A] FIG. 1 is a block diagram illustrating a tagging system for machine learning purposes according to an alternative embodiment.

[0033] [Figure 4B] FIG. 4B is a sequence diagram illustrating the operational mode process of the tagging system for machine learning purposes according to the embodiment of FIG. 4A.

[0034] [Figure 5A] FIG. 10 is a block diagram illustrating a tagging system for machine learning purposes according to yet another alternative embodiment.

[0035] [Figure 5B] FIG. 5B is a sequence diagram illustrating the operational mode process of the tagging system for machine learning purposes according to the embodiment of FIG. 5A.

[0036] [Figure 6] FIG. 1 is a flow diagram illustrating a method for training a machine learning (ML) model, according to some embodiments.

[0037] [Figure 7] FIG. 1 is a flow diagram illustrating a method for training a machine learning (ML) model, according to another embodiment.

[0038] It should be understood that for simplicity and clarity of the drawings, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements, and the letters "A," "B," and "C" may be changed according to the respective figure numbers. DETAILED DESCRIPTION OF THE INVENTION

[0039] Those skilled in the art will appreciate that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Accordingly, the foregoing embodiments are to be considered in all respects as illustrative and not limiting of the invention described herein. The scope of the present invention is, therefore, indicated by the appended claims, rather than the foregoing description, and all changes that come within the meaning and range of equivalency of the claims are intended to be embraced therein.

[0040] In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the present invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, detailed descriptions of well-known methods, procedures, and components are omitted so as not to obscure the present invention. Some features or elements described with respect to one embodiment may be combined with features or elements described with respect to other embodiments. For clarity, identical or similar features or elements may not be repeatedly described.

[0041] Although embodiments of the invention are not limited in this respect, for example, discussions utilizing terms such as "processing," "computing," "calculating," "determining," "establishing," "analyzing," "examining," etc. may refer to operations and / or processes of a computer, computing platform, computing system, or other electronic computing device that manipulate and / or transform data represented as physical (e.g., electronic) quantities in the computer's registers and / or memory into other data that is also represented as physical quantities in the computer's registers and / or memory or other non-transitory information storage medium capable of storing instructions for performing operations and / or processes.

[0042] Although embodiments of the invention are not limited in this respect, the terms "plurality" and "a plurality" as used herein may include, for example, "a large number" or "two or more." The terms "plurality" and "a plurality" may be used throughout this specification to describe two or more components, devices, elements, units, parameters, etc. Terms as used herein may include one or more items.

[0043] Unless explicitly stated, the method embodiments described herein are not limited to a particular order or sequence. Furthermore, some of the described method embodiments or elements thereof may occur or be performed simultaneously, at the same time, or together.

[0044] In some embodiments of the present invention, the ML model may be an artificial neural network (ANN).

[0045] A neural network (NN) or artificial neural network (ANN), e.g., a neural network implementing machine learning (ML) or artificial intelligence (AI) functions, may refer to an information processing paradigm that may include nodes, called neurons, organized into layers with links between them. The links can transfer signals between neurons and may be associated with weights. A NN may be configured or trained for a specific task, e.g., pattern recognition or classification. Training a NN for a specific task may involve adjusting these weights based on examples. Each neuron in an intermediate or final layer receives an input signal, e.g., a weighted sum of output signals from other neurons, and may process the input signal using a linear or nonlinear function (e.g., an activation function). The results of the input and intermediate layers are transferred to other neurons, and the results of the output layer may be provided as the output of the NN. Typically, the neurons and links in a NN are represented by mathematical structures, such as activation functions, matrices of data elements, and weights. A processor, e.g., a CPU, or a graphics processing unit (GPU), or a dedicated hardware device, can perform the relevant calculations.

[0046] It will be apparent to those skilled in the art that various ML models can be implemented without departing from the essence of the present invention. It should also be understood that in some embodiments, the ML model may be a single ML model or a set (ensemble) of ML models that collectively achieve the same functionality as a single ML model. Therefore, in consideration of the scope of the present invention, the above-described variations should be considered equivalent.

[0047] It should also be understood that in the context of this specification, the terms "tagging," "labeling," and "annotating," as well as derivatives of these terms, may be used interchangeably.

[0048] The following description of the claimed invention is provided in accordance with the above-mentioned task of tagging geographic data, e.g., multi-level overpasses (multi-level transportation routes, e.g., pedestrian and motor vehicle bridges, interchanges, tunnels, etc.) on satellite imagery corresponding to a particular geographic region. Accordingly, in some aspects, the following description refers to training an ML model that classifies incoming samples of input data elements (e.g., satellite imagery corresponding to a particular geographic region and fragments or portions thereof) according to one or more predefined classes (e.g., presence / absence of an overpass, type of overpass such as "roadway over driveway," "roadway over sidewalk," "roadway over waterway," etc.).

[0049] Supervised and semi-supervised training of ML models faces the following obstacles in practice. Generally, such training requires creating a training dataset by manually tagging input data elements and distinguishing them by various types. Tagging geographic data has several problems, such as the need to divide large satellite images into portions (subregions) and the aspect of detecting and tagging geographic elements with specific characteristics of interest (e.g., terrain, traffic, vegetation, pictures, overpasses, etc.). Also, when it comes to data such as satellite images, it is difficult to provide appropriate annotations, for example, to identify multi-level overpasses, distinguish them from single-level crossroads, determine their types, etc. This aspect can be important because errors in the training dataset dramatically reduce the reliability of the classification output provided by the trained ML model.

[0050] To overcome this problem, embodiments of the present invention provide a multi-level tagging approach, as described herein.

[0051] This particular embodiment is provided so that the description is fully illustrative and is not intended to limit the scope of protection claimed by the present invention.

[0052] Those skilled in the art will understand that the implementations of the claimed invention according to this task are provided as non-exclusive examples, and that other actual implementations may be covered by the claimed invention.

[0053] Reference is now made to FIG. 1, which is a block diagram illustrating a computing device that may be included in a tagging system for machine learning purposes, according to some embodiments.

[0054] Computing device 1 may include a processor or controller 2, which may be, for example, a central processing unit (CPU) processor, chip, or any suitable computing or calculation device, an operating system 3, a memory device 4, instruction code 5, a storage system 6, input devices 7, and output devices 8. The processor 2 (or one or more controllers or processors, possibly across multiple units or devices) may be configured to perform methods described herein and / or to execute or function as various modules, units, etc. More than one computing device 1 may be included in a system according to embodiments of the present invention, and one or more computing devices 1 may function as components of a system according to embodiments of the present invention.

[0055] Operating system 3 may be or include any code segment (e.g., similar to instruction code 5 described herein) designed and / or configured to perform tasks including coordinating, scheduling, arbitrating, supervising, controlling, or managing the operation of computing device 1, such as, for example, scheduling the execution of software programs or tasks or enabling software programs or other modules or units to communicate. Operating system 3 may be a commercial operating system. Note that operating system 3 may be an optional component, and for example, in some embodiments, a system may include a computing device that does not require or include an operating system 3.

[0056] Memory device 4 may be or include, for example, random access memory (RAM), read-only memory (ROM), dynamic RAM (DRAM), synchronous DRAM (SD-RAM), double data rate (DDR) memory chips, flash memory, volatile memory, non-volatile memory, cache memory, buffer, short-term memory unit, long-term memory unit, or other suitable memory or storage unit. Memory device 4 may be or include multiple, possibly different, memory units. Memory device 4 may be a non-transitory computer or processor-readable medium or a non-transitory computer storage medium, such as RAM. In one embodiment, a non-transitory storage medium, such as memory device 4, a hard disk drive, or another storage device, may store instructions or code that, when executed by a processor, cause the processor to perform the methods described herein.

[0057] The instruction code 5 may be any executable code, such as an application, a program, a process, a task, or a script. The instruction code 5 may be executed by the processor or controller 2, possibly under the control of the operating system 3. For example, the instruction code 5 may be an application that may provide a tool for manual data tagging, or that may be configured to achieve automatic or semi-automatic data tagging, as well as to train an ML model, as described further herein. For clarity, although a single item of instruction code 5 is shown in FIG. 1 , systems according to some embodiments of the present invention may include multiple modules of instruction code similar to the instruction code 5 that may be loaded into the memory device 4 and cause the processor 2 to perform the methods described herein.

[0058] Storage system 6 may be or include, for example, flash memory as known in the art, memory within or embedded in a microcontroller or chip as known in the art, a hard disk drive, a CD-R (CD-Recordable) drive, a Blu-ray Disc (BD), a Universal Serial Bus (USB) device, or other suitable removable and / or non-removable storage unit. Various types of data sets may be stored in storage system 6 and loaded from storage system 6 into memory device 4 where the data sets may be processed by processor or controller 2.

[0059] 1 may be omitted. For example, memory device 4 may be a non-volatile memory having the storage capacity of storage system 6. Thus, although shown as a separate component, storage system 6 may be embedded in or included within memory device 4.

[0060] Input device(s) 7 may be or include any suitable input device, component, or system, such as a detachable keyboard or keypad, a mouse, etc. Output device(s) 8 may include one or more (possibly detachable) displays or monitors, speakers, and / or any other suitable output device(s). As shown in blocks 7 and 8, any applicable input / output (I / O) devices may be connected to computing device 1. For example, a wired or wireless network interface card (NIC), a universal serial bus (USB) device, or an external hard drive may be included in input device(s) 7 and / or output device(s) 8. It will be appreciated that any suitable number of input devices 7 and output devices 8 may be operably connected to computing device 1, as shown in blocks 7 and 8.

[0061] It will be apparent that the storage system 6 , the input devices 7 and the output devices 8 may have both embedded and external implementations with respect to the computing device 1 .

[0062] Systems according to some embodiments of the present invention may include components such as, but not limited to, multiple central processing units (CPUs) or any other suitable general-purpose or special-purpose processors or controllers (e.g., similar to element 2), multiple input units, multiple output units, multiple memory units, and multiple storage units.

[0063] Reference is now made to FIG. 2, which is a block diagram illustrating the interconnection of the tagging system with other machine learning aspects, according to some embodiments.

[0064] As can be seen, the tagging system (e.g., tagging system 10) represents an aspect of ML that is inherently integrated with other ML disciplines. In some embodiments, tagging system 10 may be configured to receive an input data element (e.g., an image from satellite imagery dataset 20A corresponding to a particular geographic region and including multiple overpass data elements (e.g., overpass data element 20A1)). Tagging system 10 may further be configured to receive supplemental data elements (e.g., supplemental data elements from supplemental dataset 21A) that relate to one or more portions of the input data element by some common characterizing feature (e.g., a photograph of an overpass related to a respective portion of the satellite image by location (GPS) data). The supplemental data elements may include an overpass data element that represents the same overpass as the input data element (e.g., overpass data element 21A1 that represents the same overpass as overpass data element 20A1).

[0065] As can be seen, based on the provided supplementary data elements, it is much easier to identify that the indicated overpass is multi-level (roadway over private road) than based on the input data elements, thereby significantly reducing the possibility of making an error during tagging of the corresponding part of the input data elements.

[0066] In some embodiments, the tagging system 10 may output, as a result of the tagging, an aggregated, labeled satellite image dataset 70A. The satellite image dataset 70A may be presented in either a final version or an intermediate version.

[0067] Additionally, the satellite imagery dataset 70A can be used as supervised data to train an ML model (e.g., overpass classification ML model 90) to calculate a confidence value representing the reliability that one or more portions of the input data elements (e.g., images from the satellite imagery dataset 20A) fit into one or more predefined classes. The classes may include, for example, a first class representing that a multi-level overpass is present in this portion of the input data elements and a second class representing that a multi-level overpass is not present in this portion.

[0068] It should be understood that in the context of the present invention, the term "confidence value" or "confidence score" refers to a well-known concept in ML-based classification activities. For example, a "confidence value" may represent the reliability of the results of a classification ML-based model, i.e., the reliability that an input data element (or a portion thereof) fits into one or more predefined classes, in the form of a value between 0 and 1, with "1" being 100 percent reliability and "0" being 0 percent reliability. Those skilled in the art will understand what a "confidence value" represents and how it can be calculated.

[0069] Additionally, the trained ML model 90 may be further inferred on one or more unannotated portions of the input data element to calculate respective confidence values, which may be further forwarded as feedback to the system 10 for use in supporting the actions of taggers on new incoming input data elements, as described in further detail herein.

[0070] As can be seen, technical improvements in tagging may be provided based on the synergistic effects of various ML aspects (e.g., supplementary data, feedback from trained ML models, etc.).

[0071] Reference is now made to FIG. 3A, which is a block diagram illustrating tagging system 10 according to some embodiments, and FIG. 3B, which is a sequence diagram illustrating the operation of system 10.

[0072] 3A and 3B are directed to a system 10 that provides technical means for performing tagging with a multi-level quality assurance (QA) procedure. In the illustrated embodiment, the tagging and QA procedure are performed manually by a tagger via a user interface (UI).

[0073] According to some embodiments of the present invention, system 10 may be implemented as software modules, hardware modules, or any combination thereof. For example, system 10 may be or include a computing device such as element 1 of FIG. 1. System 10 may also be adapted to execute one or more modules of instruction code 5 to perform tagging of input data elements and provide further instructions to ML model training module 80 to train overpass classification ML model 90 based on the labeled data. In some embodiments, the input data elements may be images from a satellite imagery dataset 20A corresponding to a particular geographic region.

[0074] The arrows may represent the flow of one or more data elements into and out of system 10 and / or between modules or elements of system 10. For clarity, some arrows may be omitted.

[0075] In some embodiments, system 10 is scalable and may include a variable number n of modules that may vary according to the particular purpose and task for which a particular embodiment is directed. For clarity, such elements are designated using the prefixes "first," "second," and "n," correspondingly.

[0076] In some embodiments, the system 10 may include a data query splitting module 30, a task management module 40, a user interface (UI) module 60, a first level tagging module 50, a second level tagging module 51, and an n-level tagging module 52.

[0077] In some embodiments, task management module 40 may be configured for receiving 400B an annotation task 10A representing a requirement to annotate an input data element. For example, the input data element may be one or more satellite images of a satellite imagery dataset 20A stored in a repository of main data supplier 20. In some embodiments, annotation task 10A may include a request to system 10 indicating the presence / absence of an overpass data element representing a multi-level overpass or a single-level overpass, the type of overpass data element, such as “roadway over driveway,” “roadway over sidewalk,” or “roadway over waterway.” Annotation task 10A may be provided by a user of system 10, for example, via a network interface. Task management module 40 may be further configured to forward instructions 401B generating a request 300B to receive the input data element to data query splitting module 30.

[0078] In some embodiments, the data query splitting module 30 may be configured to generate a request 300B to receive an input data element. In the embodiment described herein, the input data element is a satellite image from the satellite image dataset 20A. The data query splitting module 30 may be further configured to forward the request 300B to the main data supplier 20. The data query splitting module 30 may be further configured to receive a response 200B from the main data supplier 20, which may include a satellite image from the satellite image dataset 20A, and to forward an acknowledgement 301B to the task management module 40.

[0079] In some embodiments, task management module 40 may be further configured to perform division 402B of annotation task 10A into multiple annotation subtasks 40A, each representing a requirement to annotate a respective non-annotated portion of the input data element. In some embodiments, each portion of the input data element (e.g., satellite image portion 30A) corresponds to a subregion of the geographic region indicated on the input data element and includes a subset of the multiple overpass data elements.

[0080] The task management module 40 may be further configured to transfer instructions 403B to the data query splitting module 30 to split the input data elements (satellite images of the received satellite image dataset 20A) into respective non-annotated portions (satellite image portions 30A), and the data query splitting module 30 may be configured to perform the splitting 302B of the received satellite images into portions 30A in accordance with the instructions 403B.

[0081] In some embodiments, the task management module 40 may be further configured to perform step 404B of assigning one or more annotation subtasks 40A of the plurality of subtasks 40A to one or more taggers by each first-level tagging module 50, and the data query division module 30 may be configured to perform transmission 303B of the satellite image portions 30A to the first-level tagging modules 50 according to the assigned subtasks 40A.

[0082] In some embodiments, the first-level tagging module 50 may be configured to perform annotation 500B of the satellite image portion 30A, for example, an indication of the presence / absence of an overpass data element, an indication of an overpass type such as “roadway over driveway,” “roadway over sidewalk,” “roadway over waterway,” etc. In some embodiments, the first-level tagging module 50 may be configured to perform annotation 500B via a UI module 60, which may be configured to provide corresponding UI functions to the first-level taggers who are annotators. The UI module 60 may be further configured to receive, via a first UI, multiple annotations 500B from the first-level taggers corresponding to the assigned annotation subtasks 40A, the first UI being related to each first-level tagger. The first-level tagging module 50 may be configured to obtain respective annotated portions of the input data element (e.g., labeled satellite image portion 50A) based on the multiple annotations 500B provided by the first-level taggers corresponding to the assigned annotation subtasks 40A.

[0083] It should be understood that in order to annotate a substantial amount of data with a limited number of taggers, some operations that the system 10 is configured to perform should be performed repeatedly (iteratively). Accordingly, in some embodiments, the system 10 may be configured to perform some operations as an iterative process. Such operations may include at least assigning 404B, by each first-level tagging module 50, one or more annotation subtasks 40A of the plurality of subtasks 40A to one or more taggers, and obtaining respective annotated portions of the input data elements (e.g., labeled satellite image portions 50A) based on the plurality of annotations 500B provided by the first-level taggers corresponding to the assigned annotation subtasks 40A. The iterative process may include at least two iterations.

[0084] In some embodiments, assigning one or more annotation subtasks and obtaining each annotated portion is performed as an iterative process involving a sequence of at least two iterations, which are referred to herein as "initial," "intermediate," and "final" to distinguish them by their order of application.

[0085] To effectively detect and correct inaccurately tagged data and assist tagging specialists in reducing the number of future tagging errors, in some embodiments, system 10 may be configured to perform a multi-level quality assurance (QA) procedure on the plurality of annotations 500B. The multi-level quality assurance (QA) procedure may be performed during at least one initial iteration of the iterative process. The multi-level QA procedure is performed by tagging modules 51, 52, as described further herein.

[0086] In some embodiments, the first level tagging module 50 may be configured to perform the transmission 501B of the labeled satellite image portions 50A to the second level tagging module 51.

[0087] In some embodiments, the second-level tagging module 51 may be configured to perform supervised tagging 510B of the annotations 500B, for example, to indicate whether the annotations performed by the first-level taggers are correct or incorrect. The second-level tagging module 51 may be configured to perform supervised tagging 510B via a UI module 60, which may be configured to provide corresponding user interface functionality to the second-level taggers who perform the supervision. In some embodiments, the UI module 60 may be further configured to receive at least one supervised feedback data element 51A′ for the annotations 500B via a second UI for each second-level tagger. In some embodiments, the second-level tagging module 51 may be configured to generate inspected labeled satellite image portions 51A as a result of the annotations 500B and supervised tagging 510B, and to transmit 511B of the inspected labeled satellite image portions 51A to the n-th level tagging module 52.

[0088] In some embodiments, the nth-level tagging module 52 may include a third-level tagging module, which may be configured to perform approval 520B of the annotations 500B and supervised tagging 510B, for example, to indicate whether the annotations performed by the first-level taggers are approved. In some embodiments, the third-level tagging module of the nth-level tagging module 52 may be configured to perform approval 520B via the UI module 60, which may be configured to provide corresponding user interface functionality to the third-level taggers who perform the approval. In some embodiments, the UI module 60 may be further configured to receive at least one approval feedback data element 52A′ for the annotations 500B via a third UI for each third-level tagger. In some embodiments, the third-level tagging module of the nth-level tagging module 52 may be configured to generate approved labeled satellite image portions 52A as a result of the annotations 500B, the supervised tagging 510B, and the approval 520B.

[0089] Furthermore, the use of QA scoring of the taggers' work is proposed to reduce the amount of tagging errors and thereby improve the quality of the training dataset. Accordingly, in some embodiments, the second-level tagging module 51 may be configured to calculate a QA score for the annotation 500B based on the supervised feedback data elements 51A′ and transmit the QA score to a third-level tagging module of a respective n-th level tagging module 52. In some embodiments, the third-level tagging module of the n-th level tagging module 52 may further be configured to calculate a QA score for the annotation 500B based on the approval feedback data elements 52A′. In some embodiments, the third-level tagging module may further be configured to transmit (not shown) multiple QA scores to the task management module 40.

[0090] For example, in some embodiments, the supervised feedback data element 51A′ and the acknowledgement feedback data element 52A′ may represent a value for the difficulty of an incorrectly or correctly completed subtask, which may be assessed by the second-level tagging module 51 or the nth-level tagging module 52, respectively (e.g., by the corresponding tagger) based on their own perspective on the difficulty of the subtask or based on a respective calculated confidence value (described further herein) that the input data element portion corresponding to the respective subtask is appropriate for at least one pre-defined class. Thus, the more difficult the correctly completed subtask, the higher the calculated QA score, and the easier the incorrectly completed subtask, the lower the calculated QA score (as indicated by the respective supervised data element and / or acknowledgement data element).

[0091] In some embodiments, task management module 40 may be further configured to perform step 404B of assigning one or more subsequent annotation subtasks 40A of the plurality of subtasks 40A to one or more specific first-level tagging modules 50 based on the QA score. Assignment 404B may be performed during at least one intermediate or final iteration of the iterative process. Thus, in this manner, annotation subtasks 40A may be assigned by UI module 60 to specific first-level taggers who perform annotation 500B using specific first-level tagging modules 50 based on the QA score.

[0092] For example, the task management module 40 may be further configured to perform the assignment 404B of one or more subsequent annotation subtasks 40A in the following manner: annotation subtasks 40A (calculation 900B as described further below) corresponding to portions of the input data elements for which the ML model 90 has calculated a low confidence value, e.g., a confidence value less than 0.5, may be assigned to a first-level tagging module with a high QA score, as they are considered to be more capable of performing a “difficult” task; whereas annotation subtasks 40A (calculation 900B as described further below) corresponding to portions of the input data elements for which the ML model 90 has calculated a high confidence value, e.g., a confidence value higher than 0.5, may be assigned to a first-level tagging module with a low QA score, as the classification is more likely to be correct and such subtasks are considered “easy” and therefore do not require high quality and reliability tagging.

[0093] In some embodiments, execution of a multi-level quality assurance (QA) procedure on the multiple annotations may be performed based on the calculated confidence values ​​(calculation 900B as described further below). For example, subtasks corresponding to tagging input data element portions classified by the ML model 90 with a “high” confidence value (e.g., greater than 0.8) are considered “easy” to tag, and subtasks corresponding to tagging input data element portions classified by the ML model 90 with a “low” confidence value (e.g., less than 0.8) are considered “difficult” to tag. Thus, incorrect execution of an “easy” subtask (presence of a tagging error) may signal the second-level tagging module 51 (or the n-th-level tagging module 52) to calculate a lower QA score for each annotation 500B than in the case of incorrect execution of a “difficult” subtask, i.e., the QA score is reduced less by an error in an “easy” subtask than by an error in a “difficult” subtask. This approach can also work for successful execution of “easy” and “difficult” subtasks, respectively.

[0094] The described multi-level QA procedure collectively facilitates the process of data tagging, thereby improving the quality of the training dataset and, consequently, the reliability of the classification output of the ML model.

[0095] In some embodiments, system 10 may be further configured to form a labeled dataset (e.g., labeled satellite image dataset 70A) by aggregating the annotated portions (e.g., satellite image portions 52A).

[0096] As mentioned above, in practice, it is difficult to prescribe a specific amount of training data that is sufficient to achieve reliable training results. To overcome this problem and to prioritize subtasks and provide subtask allocation load balancing, the following solution is proposed.

[0097] In some embodiments, system 10 may be further configured to form an intermediate version of the labeled dataset (e.g., labeled satellite image dataset 70A) by aggregating annotated portions (e.g., satellite image portions 52A) during at least one initial iteration.

[0098] In some embodiments, the third-level tagging module of the nth-level tagging module 52 may be further configured to perform sending 521B of the aggregated labeled satellite image dataset 70A to the ML model training module 80. In some embodiments, the ML model training module 80 may be further configured to perform supervised or semi-supervised training 800B of an ML model (e.g., overpass classification ML model 90) using an intermediate version of the labeled dataset (e.g., labeled satellite image dataset 70A) to calculate a confidence value during at least one initial iteration. The confidence value may represent the confidence that one or more portions of the input data elements (satellite images) fit into one or more predefined classes. The predefined classes may include a first class representing the presence of a multi-level overpass in this portion and a second class representing the absence of a multi-level overpass in this portion. In some embodiments, the predefined classes may further include the type of multi-level overpass, such as, for example, “roadway over driveway,” “roadway over sidewalk,” “roadway over waterway,” etc.

[0099] In some embodiments, task management module 40 may be further configured, during at least one intermediate iteration, to send to the trained ML model (e.g., overpass classification ML model 90) instructions 405B for inferring the overpass classification ML model 90 for one or more unannotated portions of the input data element (e.g., satellite image portion 30A). Data query segmentation module 30 may be further configured to perform transmission 304B of satellite image portion 30A to overpass classification ML model 90. In some embodiments, overpass classification ML model 90 may be further configured to perform calculation 900B of a confidence value that one or more portions of the input data element (e.g., satellite image portion 30A) are appropriate for one or more predefined classes (e.g., presence / absence of a multi-level overpass, type of multi-level overpass such as "roadway over driveway," "roadway over sidewalk," "roadway over waterway," etc.).

[0100] In some embodiments, the overpass classification ML model 90 may be further configured to generate, during at least one intermediate iteration, overpass classification data 90A including results of the calculation 900B. The overpass classification ML model 90 may be further configured to perform transmission 901B of the overpass classification data 90A to the task management module 40. In some embodiments, the task management module 40 may be further configured, during at least one intermediate iteration, to perform step 404B of assigning at least one annotation subtask of the plurality of subtasks 40A to at least one first-level tagging module 50 based on the calculated confidence value.

[0101] In some embodiments, task management module 40 may be configured with instructions such that if additional positive annotations are needed based on overpass classification data 90A, task management module 40 performs, during at least one intermediate iteration, assignment 404B of annotation subtasks 40A according to input data element portions having high confidence values ​​for a positive classification output. In some embodiments, task management module 40 may be configured with instructions such that if additional negative annotations are needed based on overpass classification data 90A, task management module 40 performs, during at least one intermediate iteration, assignment 404B of annotation subtasks 40A according to input data element portions having low confidence values ​​for a positive classification output.

[0102] Also, in some embodiments, task management module 40 may be configured with instructions such that task management module 40 performs assignment 404B of annotation subtasks 40A to input data element portions that have low confidence values ​​for positive and / or negative classification outputs in order to focus the tagging process and subsequent learning process, during at least one intermediate iteration, on types of input data element portions that ML model 90 cannot classify with high confidence, thereby improving the efficiency of the training process.

[0103] Reference is now made to FIG. 4A, which is a block diagram illustrating tagging system 10 according to an alternative embodiment, and FIG. 4B, which is a sequence diagram illustrating the operation of system 10. As shown in FIG.

[0104] The embodiment depicted in Figures 4A and 4B is similar in general aspects to the embodiment depicted in Figures 3A and 3B, respectively, except as further described.

[0105] The embodiment depicted in Figures 4A and 4B provides an additional way to reduce the amount of tagging errors caused by the type of input data. As explained with reference to Figure 2 above, a set of additional, more detailed types of supplementary data may become available that are interconnected with the data of the main dataset and are easier to annotate. For example, there may be a dataset of photos and videos that are related to one or more portions of the input data elements (e.g., satellite imagery) by at least one common characterizing feature (e.g., by GPS coordinates). Clearly, it is much easier to identify overpass objects based on such supplementary data.

[0106] The idea of ​​this aspect of the claimed invention is therefore to develop a method for creating labeled datasets for training an ML model to classify incoming samples of less detailed types of input data, which method involves supervised data tagging based on more detailed types of data.

[0107] In some embodiments, the task management module 40 may be further configured to forward to the data query splitting module 30 instructions 401B for generating a request 300B to receive the input data element (e.g., a satellite image from the satellite imagery dataset 20A), the supplemental data element from the first type of supplemental dataset 21A (e.g., a photo of an overpass with corresponding location data), and the supplemental data element from the nth type of supplemental dataset (e.g., a video recording from a dash cam with corresponding location data). The data query splitting module 30 may be configured to generate a request 305B to receive the supplemental data element from the first type of supplemental dataset 21A and forward the request 305B to the first type of supplemental data supplier 21. The data query splitting module 30 may be configured to generate a request 306B to receive the supplemental data element from the nth type of supplemental dataset 22A and forward the request 306B to the nth type of supplemental data supplier 22. The data query splitting module 30 may be further configured to receive a response 210B from the first type supplemental data supplier 21, where the response 210B may include a first type supplemental data element (e.g., a photo of an overpass with corresponding location data) from the first type supplemental data set 21A that is related to the corresponding portion of the input data element (e.g., a satellite image from the satellite imagery dataset 20A) by at least one common characterizing feature (e.g., location data). In some embodiments, the data query splitting module 30 may be further configured to receive a response 220B from the nth type supplemental data supplier 22, where the response 220B may include an nth type supplemental data element (e.g., a video recording from a dash cam with corresponding location data) from the nth type supplemental data set 22A that is related to the corresponding portion of the input data element (e.g., a satellite image from the satellite imagery dataset 20A) by at least one common characterizing feature (e.g., location data). In some embodiments, the data query splitting module 30 may be further configured to forward the receipt confirmation 301B to the task management module 40.

[0108] In some embodiments, the data query splitting module 30 may be further configured to perform, according to the assigned subtask 40A, sending 307B of the first type of supplementary data element 31A (e.g., a photo of the overpass with corresponding location data) to the second-level tagging module 51.

[0109] In some embodiments, the system 10 is further configured to perform a multi-level quality assurance (QA) procedure on the plurality of annotations 500B based on at least one respective supplementary data element (e.g., a first type supplementary data element 31A and a w type supplementary data element 32A).

[0110] In particular, the second-level tagging module 51 may be configured to perform supervised tagging 510B of the annotations 500B and to perform an indication of whether the annotations performed by the first-level taggers are correct or incorrect, for example, by using the first-type supplemental data elements 31A (photographs of the overpass with corresponding location data). In some embodiments, the second-level tagging module 51 may be configured to perform supervised tagging 510B via the UI module 60, which may be configured to provide corresponding user interface functionality, including presentation of the first-type supplemental data elements 31A, to the supervised second-level taggers. In some embodiments, the UI module 60 may be further configured to receive, for the annotations 500B, at least one supervised feedback data element 51A′ based on the first-type supplemental data elements 31A via a second UI for each second-level tagger.

[0111] In some embodiments, the data query splitting module 30 may be further configured to perform transmission 308B of the nth type supplemental data element 32A (video recording from the dash cam with corresponding location data) to the nth level tagging module 52 according to the assigned subtask 40A.

[0112] In some embodiments, the nth-level tagging module 52 may include a third-level tagging module, which may be configured to perform approval 520B of the annotations 500B and supervised tagging 510B and to indicate whether the annotations performed by the first-level taggers are approved, for example, by using the nth-type supplemental data elements 32A (video recordings from dash cams with corresponding location data). In some embodiments, the third-level tagging module of the nth-level tagging module 52 may be configured to perform approval 520B via a UI module 60, which may be configured to provide corresponding user interface functionality, including presentation of the nth-type supplemental data elements 32A, to the approving third-level taggers. In some embodiments, the UI module 60 may be further configured to receive, for the annotations 500B, at least one approval feedback data element 52A′ based on the nth-type supplemental data elements 32A via a third UI for each third-level tagger.

[0113] Another difference between the embodiment of Figures 4A and 4B compared to the embodiment of Figures 3A and 3B is that during at least one intermediate iteration, the system 10 may be configured to perform a multi-level quality assurance (QA) procedure on the plurality of annotations 500B based on confidence values ​​calculated by the overpass classification ML model 90 inferred for one or more respective portions of the input data elements (e.g., satellite image portion 30A).

[0114] In particular, the overpass classification ML model 90 may be further configured to perform transmission 902B of the overpass classification data 90A to the first level tagging module 50, and / or the second level tagging module 51, and / or the nth level tagging module 52, which is presented to the respective taggers via the UI module 60. Thus, in such an embodiment, the system 10 provides additional supplemental information to the taggers to support them in performing annotation 500B, supervision 510B, and approval 520B, respectively.

[0115] In some alternative embodiments, the system 10 may be configured to replace the first-level tagging module 50 with the overpass classification ML model 90 and replace the labeled satellite image portion 50A with the overpass classification data 90A once the overpass classification ML model 90 reaches a predetermined confidence threshold in the classification output.

[0116] Reference is now made to FIG. 5A, which is a block diagram illustrating a tagging system 10 according to an alternative embodiment, and FIG. 5B, which is a sequence diagram illustrating the operation of the system 10. As shown in FIG.

[0117] The embodiments depicted in Figures 5A and 5B are similar in general aspects to the embodiments depicted in Figures 3A, 3B and 4A, 4B, respectively, except as further described.

[0118] As explained further herein, the embodiment shown in Figures 5A and 5B differs from the embodiment shown in Figures 3A, 3B and 4A, 4B in that the multi-level QA procedure (second-level tagging and n-level tagging) is performed automatically rather than manually. In the illustrated embodiment, first-level tagging is still performed manually (by UI module 60), and thus the entire tagging process is semi-automatic.

[0119] Those skilled in the art will appreciate that there may be embodiments that include a fully automated tagging process without departing from the essence of the present invention.

[0120] In some embodiments, the system 10 further includes a second-level supervised ML model 91 and an n-level supervised ML model 92.

[0121] In some embodiments, the second-level supervised ML model 91 may be configured to receive the first-type supplemental data element 31A (photographs of overpasses with corresponding location data) and generate an overpass classification output according to one or more predefined classes (e.g., presence / absence of an overpass, type of overpass such as “roadway over private road,” “roadway over sidewalk,” “roadway over waterway,” etc.). In some embodiments, the second-level supervised ML model 91 may be trained by a supervised learning algorithm using a set of labeled training examples, which may include pairs of photographs of overpasses with corresponding location data and an indication of the presence / absence or type of overpass, respectively. It will be apparent to those skilled in the art that any ML model conventionally used for image recognition tasks may be used as the second-level supervised ML model 91 without departing from the essence of the present invention.

[0122] In some embodiments, the nth-level supervised ML model 92 may be configured to receive the nth type of supplementary data element 32A (video recordings from a dash cam with corresponding location data) and generate an overpass classification output according to one or more predefined classes (e.g., presence / absence of an overpass, type of overpass such as “roadway over driveway,” “roadway over sidewalk,” “roadway over waterway,” etc.). In some embodiments, the nth-level supervised ML model 92 may be trained by a supervised learning algorithm using a set of labeled training examples, which may include pairs of video recordings from a dash cam with corresponding location data and an indication of the presence / absence or type of overpass, respectively. It will be apparent to those skilled in the art that any ML model conventionally used for image recognition tasks may be used as the nth-level supervised ML model 92 without departing from the essence of the present invention.

[0123] In some embodiments, the second-level tagging module 51 is configured to execute instructions 910B to send 512B the first type of supplemental data elements 31A (photographs of the overpass with corresponding location data) to the second-level supervised ML model 91 and infer 912B the second-level supervised ML model 91 on the received first type of supplemental data elements 31A. In some embodiments, the second-level supervised ML model 91 may be configured to send a response 911B to the second-level tagging module 51 including an overpass classification output according to one or more predefined classes.

[0124] In some embodiments, the second-level tagging module 51 may be configured to perform supervision 510B of the annotations 500B and perform an indication of whether the annotations performed by the first-level tagger are correct or incorrect, for example, by using the flyover classification output received from the second-level supervised ML model 91. In some embodiments, supervision 510B may include comparing the annotations 500B made by the first-level tagger with the classes determined by the second-level supervised ML model 91.

[0125] In some embodiments, the nth level tagging module 52 is configured to execute instructions 920B to transmit 522B the nth type supplemental data element 32A (video recording from a dash cam with corresponding location data) to the nth level supervised ML model 92 and to infer 920B the nth level supervised ML model 92 on the received nth type supplemental data element 32A. In some embodiments, the nth level supervised ML model 92 may be configured to transmit a response 921B to the nth level tagging module 52 including an overpass classification output according to one or more predefined classes.

[0126] In some embodiments, the nth level tagging module 52 may be configured to perform validation 520B of the annotations 500B and supervised 510B, and to perform an indication of whether the annotations performed by the first level tagger are validated or not, for example, by using the flyover classification output received from the nth level supervised ML model 92. In some embodiments, validation 520B may include comparing the annotations 500B made by the first level tagger with the supervised 510B made by the second level tagging module 51 and the classes determined by the nth level supervised ML model 92.

[0127] Referring now to FIG. 6, a flow diagram illustrating a method for training a machine learning (ML) model by at least one processor is presented, according to some embodiments.

[0128] As shown in step S1005, at least one processor (e.g., processor 2 of FIG. 1) may execute receiving 400B of annotation task 10A, which represents a requirement to annotate an input data element (a satellite image of satellite image dataset 20A). Step S1005 may be executed by task management module 40 and data query partitioning module 30 (as described with reference to FIGS. 3A, 3B, 4A, 4B, 5A, and 5B).

[0129] As shown in step S1010, at least one processor (e.g., processor 2 of FIG. 1) may perform division 402B of annotation task 10A into multiple annotation subtasks 40A, each representing a requirement to annotate a respective unannotated portion of an input data element (respective satellite image portion 30A). Step S1010 may be performed by task management module 40 (as described with reference to FIGS. 3A, 3B, 4A, 4B, 5A, and 5B).

[0130] As shown in step S1015, at least one processor (e.g., processor 2 of FIG. 1) may perform assignment 404B of one or more annotation subtasks of the plurality of subtasks 40A to one or more taggers. Step S1015 may be performed by task management module 40, first-level tagging module 50, and UI module 60 (as described with reference to FIGS. 3A, 3B, 4A, 4B, 5A, and 5B).

[0131] As shown in step S1020, at least one processor (e.g., processor 2 of FIG. 1) may obtain respective annotated portions of the input data elements (e.g., labeled satellite image portions 50A) based on the plurality of annotations 500B provided by the taggers corresponding to the assigned annotation subtasks 40A. Step S1020 may be performed by first-level tagging module 50 and UI module 60 (as described with reference to FIGS. 3A, 3B, 4A, 4B, 5A, and 5B).

[0132] As shown in step S1025, at least one processor (e.g., processor 2 of FIG. 1) may form a labeled dataset (e.g., aggregated labeled satellite image dataset 70A) by aggregating the annotated portions (e.g., labeled satellite image portions 50A). Step S1025 may be performed by w-th level tagging module 52 (as described with reference to FIGS. 3A, 3B, 4A, 4B, 5A, and 5B).

[0133] Referring now to FIG. 7, a flow diagram illustrating a method for training a machine learning (ML) model by at least one processor is presented, according to another embodiment.

[0134] As shown in step S2005, at least one processor (e.g., processor 2 of FIG. 1) may execute receive 400B of annotation task 10A, which represents a requirement to annotate an input data element (e.g., a satellite image of satellite imagery dataset 20A). Step S2005 may be executed by task management module 40 and data query segmentation module 30 (as described with reference to FIGS. 3A, 3B, 4A, 4B, 5A, and 5B).

[0135] As shown in step S2010, at least one processor (e.g., processor 2 of FIG. 1) may receive input data elements (e.g., satellite images from satellite imagery dataset 20A). Step S2010 may be performed by data query partitioning module 30 (as described with reference to FIGS. 3A, 3B, 4A, 4B, 5A, and 5B).

[0136] As shown in step S2015, at least one processor (e.g., processor 2 of FIG. 1) may perform division 402B of annotation task 10A into multiple annotation subtasks 40A, each representing a requirement to annotate a respective unannotated portion of an input data element (e.g., a respective satellite image portion 30A). Step S2015 may be performed by task management module 40 (as described with reference to FIGS. 3A, 3B, 4A, 4B, 5A, and 5B).

[0137] As shown in step S2020, at least one processor (e.g., processor 2 of FIG. 1) can infer a trained ML model (e.g., overpass classification ML model 90) for one or more unannotated portions of an input data element (e.g., satellite image portion 30A) to calculate a confidence value representing the reliability that the one or more portions of the input data element (e.g., satellite image portion 30A) are appropriate for one or more predefined classes (e.g., presence / absence of a multi-level overpass, type of multi-level overpass such as "roadway over driveway," "roadway over sidewalk," "roadway over waterway," etc.). Step S2020 may be performed by task management module 40 and ML model training module 80 (as described with reference to FIGS. 3A, 3B, 4A, 4B, 5A, and 5B).

[0138] As shown in step S2025, at least one processor (e.g., processor 2 of FIG. 1) may perform assignment 404B of at least one annotation subtask of the plurality of subtasks 40A to at least one tagger based on the calculated confidence value. Step S2025 may be performed by task management module 40, first-level tagging module 50, and UI module 60 (as described with reference to FIGS. 3A, 3B, 5A, and 5B).

[0139] As shown in step S2030, at least one processor (e.g., processor 2 of FIG. 1) may obtain respective annotated portions of the input data elements (e.g., labeled satellite image portions 50A) based on the plurality of annotations 500B provided by the taggers corresponding to the assigned annotation subtasks 40A. Step S2030 may be performed by first-level tagging module 50 and UI module 60 (as described with reference to FIGS. 3A, 3B, 4A, 4B, 5A, and 5B).

[0140] As shown in step S2035, at least one processor (e.g., processor 2 of FIG. 1) may form an intermediate version of the labeled dataset (e.g., aggregated labeled satellite image dataset 70A) by aggregating the annotated portions (e.g., labeled satellite image portions 50A). Step S2035 may be performed by nth-level tagging module 52 (as described with reference to FIGS. 3A, 3B, 4A, 4B, 5A, and 5B).

[0141] As shown in step S2040, at least one processor (e.g., processor 2 of FIG. 1) may perform training 800B of an ML model (e.g., overpass classification ML model 90) using the intermediate version of the labeled dataset as supervised data to calculate a confidence value representing the reliability that one or more portions of the input data elements (satellite image portion 30A) fit into one or more predefined classes (e.g., presence / absence of a multi-level overpass, type of multi-level overpass such as "roadway over driveway," "roadway over sidewalk," "roadway over waterway," etc.). Step S2040 may be performed by ML model training module 80 (as described with reference to FIGS. 3A, 3B, 4A, 4B, 5A, and 5B).

[0142] As can be seen, the present invention in the embodiment described with reference to FIG. 7 provides the following improved technical effect: it facilitates the specification of a specific amount of training data sufficient to achieve reliable training results. In such an embodiment, the process of creating a labeled dataset for training a machine learning (ML) model is inherently interconnected with the process of training the model itself, helping to optimize both processes. According to such an embodiment of the method, an intermediate version of the labeled dataset (aggregated labeled satellite image dataset 70A) can first be formed by manual tagging of some portions of input data elements (e.g., satellite image portions 30A). Then, an ML model (overpass classification ML model 90) can be trained based on the intermediate version of the labeled dataset. At this stage, because the amount of supervised data is still insufficient, the overpass classification ML model 90 may have little confidence in both positive and negative classification outputs (e.g., the presence / absence of a multi-level overpass). The entire process can then be iteratively repeated, with new unannotated portions of input data elements (e.g., satellite image portions 30A) tagged during each iteration. During each iteration, an ML model is additionally trained based on each updated intermediate version of the labeled dataset (aggregated labeled satellite imagery dataset 70A). Also during each iteration, a version of the ML model trained during the previous iteration is inferred against new, unannotated portions of the input data elements to generate respective overpass classification data 90A. Task management module 40 further uses these overpass classification data 90A to define which types of overpass data elements are poorly or incorrectly classified by the ML model. This definition may be made by evaluating cases where the ML model has a low confidence value. Thus, task management module 40 may execute only annotation subtask assignment 404B, which represents a requirement to annotate portions of the input data elements for which the ML model showed a low confidence value.

[0143] As a result, this helps to focus the tagging process only on problematic parts, and does not apply manual tagging to parts for which the ML model has already demonstrated sufficient confidence. Thus, during each iteration, the ML model becomes more and more confident about the reliability of its classification output, and as a result, the amount of annotation subtasks 40A assigned by the task management module 40 decreases. Once the ML model obtains the required confidence value, the entire process can be terminated. Thus, the training dataset has its optimal size and quality to adequately train the ML model.

[0144] As can be understood from the provided specification, the claimed invention represents a technical means for effectively detecting and correcting inaccurately tagged data, assisting taggers in reducing the number of tagging errors. The present invention represents a method for facilitating tagging of data for machine learning purposes, thereby improving the quality of training datasets and, consequently, the reliability of the prediction or classification output of ML models.

[0145] Unless explicitly stated, the method embodiments described herein are not limited to a particular order or sequence. Also, all formulas described herein are intended as examples only, and other or different formulas may be used. Furthermore, some of the described method embodiments or elements thereof may occur or be performed at the same time.

[0146] While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes, and equivalents will occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.

[0147] Various embodiments are illustrated, each of which may, of course, include features of the other illustrated embodiments, and embodiments not described in detail may include various features described herein.

Claims

1. 1. A method of creating, by at least one processor, a labeled dataset for training a machine learning (ML) model, comprising: receiving an annotation task representing a requirement to annotate an input data element; dividing the annotation task into a plurality of annotation subtasks, each subtask representing a requirement to annotate a respective non-annotated portion of the input data element; assigning one or more annotation subtasks of said plurality of subtasks to one or more taggers; obtaining an annotated portion of each of the input data elements based on a plurality of annotations provided by the taggers corresponding to the assigned annotation subtask; forming the labeled dataset by aggregating the annotated portions.

2. The method of claim 1 , wherein assigning one or more of the annotation subtasks and obtaining the respective annotated portions is performed as an iterative process comprising a sequence of at least two iterations.

3. The sequence of at least two of said repeats comprises: forming an intermediate version of the labeled dataset by aggregating the annotated portions; training the ML model using the intermediate version of the labeled dataset as supervised data to calculate a confidence value representing the reliability that one or more of the portions of the input data elements are appropriate to one or more of the predefined classes; and inferring the trained ML model on one or more of the unannotated portions of the input data elements to calculate respective confidence values; and assigning at least one annotation subtask of the plurality of subtasks to one or more tagging modules based on the calculated confidence value.

4. The sequence of at least two of said repeats comprises: at least one initial iteration, further comprising performing a multi-level quality assurance (QA) procedure on a plurality of said annotations to obtain a respective plurality of QA scores; and at least one intermediate iteration further comprising assigning one or more of the annotation subtasks of the plurality of subtasks to a particular tagger based on the QA score.

5. At least one of the initial iterations comprises: forming an intermediate version of the labeled dataset by aggregating the annotated portions; training the ML model using the intermediate version of the labeled dataset as supervised data to calculate a confidence value representing a confidence that one or more of the portions of the input data elements are appropriate to one or more of the predefined classes; At least one intermediate iteration inferring the trained ML model on one or more of the unannotated portions of the input data elements to calculate respective confidence values; The method of claim 4 , further comprising: performing the multi-level quality assurance (QA) procedure on the plurality of annotations based on the calculated confidence values.

6. receiving at least one supplemental data element related to one or more portions of said input data element by at least one common characterizing feature; The method of claim 4 or 5, further comprising: performing the multi-level quality assurance (QA) procedure on a plurality of the annotations based on at least one respective supplementary data element.

7. The step of performing a multi-level QA procedure includes: receiving annotations via a first user interface (UI) for each first-level tagger for at least one annotation subtask; receiving at least one supervised feedback data element for the annotation via a second UI related to a second-level tagger; and calculating the QA score for the annotation based on the supervised feedback data elements.

8. receiving at least one approval feedback data element for the annotation via a third UI related to a third level tagger; The method of claim 7 , further comprising: calculating the QA score for the annotation further based on the approval feedback data elements.

9. 9. The method of claim 1, wherein the input data elements correspond to a particular geographical region and comprise a plurality of overpass data elements, each portion of the input data elements corresponding to a sub-region of the geographical region and comprising a subset of the plurality of overpass data elements.

10. The method of claim 9 , wherein the annotation includes an indication of at least one overpass data element as representing a multi-level overpass or a single-level overpass.

11. 11. The method of claim 9 or 10, wherein the one or more predefined classes are selected from a first class representing the presence of a multi-level overpass in the portion and a second class representing the absence of a multi-level overpass in the portion.

12. 1. A method of creating, by at least one processor, a labeled dataset for training a machine learning (ML) model, comprising: receiving an annotation task representing a requirement to annotate an input data element; receiving the input data elements; dividing the annotation task into a plurality of annotation subtasks, each subtask representing a requirement to annotate a respective non-annotated portion of the input data element; and performing an iterative process, each iteration comprising: inferring a pre-trained ML-based model on one or more unannotated portions of the input data elements to calculate a confidence value representing the confidence that the one or more unannotated portions of the input data elements are appropriate for one or more of the pre-defined classes; assigning at least one annotation subtask of the plurality of subtasks to at least one tagger based on the calculated confidence value; obtaining an annotated portion of each of the input data elements based on a plurality of annotations provided by at least one of the taggers corresponding to at least one of the assigned annotation subtasks; forming an intermediate version of the labeled dataset by aggregating the annotated portions; and supplementally training the pre-trained ML-based model using the intermediate version of the labeled dataset as supervised data to recalculate the confidence value.

13. 1. A system for creating a labeled dataset, comprising: a non-transitory memory device having stored thereon a module of instruction code; and at least one processor associated with said memory device and configured to execute said module of instruction code, wherein upon execution of said module of instruction code, said at least one processor: receiving an annotation task representing a requirement to annotate an input data element; dividing the annotation task into a plurality of annotation subtasks, each subtask representing a requirement to annotate a respective non-annotated portion of the input data element; assigning one or more annotation subtasks of the plurality of subtasks to one or more taggers; obtaining an annotated portion of each of the input data elements based on a plurality of annotations provided by the taggers corresponding to the assigned annotation subtasks; A system configured to form the labeled dataset by aggregating the annotated portions.

14. 14. The system of claim 13, wherein at least one of the processors is further configured to assign one or more of the annotation subtasks and obtain respective annotated portions in an iterative process comprising a sequence of at least two iterations.

15. The sequence of at least two of said repeats comprises: forming an intermediate version of the labeled dataset by aggregating the annotated portions; training the ML model using the intermediate version of the labeled dataset as supervised data to calculate a confidence value representing the reliability that one or more of the portions of the input data elements are appropriate to one or more of the predefined classes; and inferring the trained ML model on one or more of the unannotated portions of the input data elements to calculate respective confidence values; and assigning at least one annotation subtask of the plurality of subtasks to one or more tagging modules based on the calculated confidence value.

16. The sequence of at least two of said repeats comprises: at least one initial iteration, further comprising performing a multi-level quality assurance (QA) procedure on a plurality of said annotations to obtain a respective plurality of QA scores; and at least one intermediate iteration further comprising assigning one or more of the annotation subtasks of the plurality of subtasks to a particular tagger based on the QA score.

17. At least one of the initial iterations comprises: forming an intermediate version of the labeled dataset by aggregating the annotated portions; training the ML model using the intermediate version of the labeled dataset as supervised data to calculate a confidence value representing a confidence that one or more of the portions of the input data elements are appropriate to one or more of the predefined classes; At least one intermediate iteration inferring the trained ML model on one or more of the unannotated portions of the input data elements to calculate respective confidence values; 17. The system of claim 16, further comprising: performing the multi-level quality assurance (QA) procedure on a plurality of the annotations based on the calculated confidence values.

18. At least one of the processors receiving at least one supplemental data element related to one or more portions of said input data element by at least one common characterizing feature; 18. The system of claim 16 or 17, further configured to perform the multi-level quality assurance (QA) procedure on a plurality of the annotations based on at least one respective supplementary data element.

19. At least one of the processors further comprises: receiving, for at least one annotation subtask, annotations via a first user interface (UI) for each first-level tagger; receiving at least one supervised feedback data element for the annotation via a second UI related to a second-level tagger; 19. The system of claim 16, further configured to perform the multi-level quality assurance (QA) procedure by calculating the QA score for the annotation based on the supervised feedback data elements.

20. At least one of the processors further comprises: receiving at least one approval feedback data element for the annotation via a third UI related to a third-level tagger; 20. The system of claim 19, further configured to perform the multi-level quality assurance (QA) procedure by calculating the QA score for the annotation further based on the approval feedback data elements.

21. 21. The system of claim 13, wherein the input data elements correspond to a particular geographic region and include a plurality of overpass data elements, each portion of the input data elements corresponding to a sub-region of the geographic region and including a subset of the plurality of overpass data elements.

22. 22. The system of claim 21, wherein the annotation includes an indication of at least one overpass data element as representing a multi-level overpass or a single-level overpass.

23. 23. The system of claim 21 or 22, wherein the one or more predefined classes are selected from a first class representing the presence of a multi-level overpass in the portion and a second class representing the absence of a multi-level overpass in the portion.

Images